opendal-0.52.0/.cargo_vcs_info.json0000644000000001420000000000100125370ustar { "git": { "sha1": "45e754b17b90e4045d8422912312ca4f88ea5143" }, "path_in_vcs": "core" }opendal-0.52.0/CHANGELOG.md000064400000000000000000010245621046102023000131550ustar 00000000000000# Change Log All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/) and this project adheres to [Semantic Versioning](https://semver.org/). ## [v0.52.0] - 2025-02-19 ### Added * feat(services/s3): Added crc64nvme for s3 by @geetanshjuneja in https://github.com/apache/opendal/pull/5580 * feat(services-fs): Support write-if-not-exists in fs backend by @SergeiPatiakin in https://github.com/apache/opendal/pull/5605 * feat(services/gcs): Impl content-encoding support for GCS stat, write and presign by @wlinna in https://github.com/apache/opendal/pull/5610 * feat(bindings/ruby): add lister by @erickguan in https://github.com/apache/opendal/pull/5600 * feat(services/swift): Added user metadata support for swift service by @zhaohaidao in https://github.com/apache/opendal/pull/5601 * feat: Implement github actions cache service v2 support by @Xuanwo in https://github.com/apache/opendal/pull/5633 * feat(core)!: implement write returns metadata by @meteorgan in https://github.com/apache/opendal/pull/5562 * feat(bindings/python): let path can be PathLike by @asukaminato0721 in https://github.com/apache/opendal/pull/5636 * feat(bindings/python): add exists by @asukaminato0721 in https://github.com/apache/opendal/pull/5637 ### Changed * refactor: Remove dead services libsql by @Xuanwo in https://github.com/apache/opendal/pull/5616 ### Fixed * fix(services/gcs): Fix content encoding can't be used alone by @Xuanwo in https://github.com/apache/opendal/pull/5614 * fix: ghac doesn't support delete anymore by @Xuanwo in https://github.com/apache/opendal/pull/5628 * fix(services/gdrive): skip the trailing slash when creating and querying the directory by @meteorgan in https://github.com/apache/opendal/pull/5631 ### Docs * docs(bindings/ruby): add documentation for Ruby binding by @erickguan in https://github.com/apache/opendal/pull/5629 * docs: Add upgrade docs for upcoming 0.52 by @Xuanwo in https://github.com/apache/opendal/pull/5634 ### CI * ci: Fix bad corepack cannot find matching keyid by @Xuanwo in https://github.com/apache/opendal/pull/5603 * ci(website): avoid including rc when calculate the latest version by @tisonkun in https://github.com/apache/opendal/pull/5608 * build: upgrade opentelemetry dependencies to 0.28.0 by @tisonkun in https://github.com/apache/opendal/pull/5625 ### Chore * chore(deps): bump uuid from 1.11.0 to 1.12.1 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/5589 * chore(deps): bump uuid from 1.11.0 to 1.12.1 in /core by @dependabot in https://github.com/apache/opendal/pull/5588 * chore(deps): bump log from 0.4.22 to 0.4.25 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/5590 * chore(deps): bump tempfile from 3.15.0 to 3.16.0 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/5586 * chore(deps): update libtest-mimic requirement from 0.7.3 to 0.8.1 in /integrations/object_store by @dependabot in https://github.com/apache/opendal/pull/5587 * chore(layers/prometheus-client): upgrade prometheus-client dependency to v0.23.1 by @koushiro in https://github.com/apache/opendal/pull/5576 * chore(ci): remove benchmark report by @dqhl76 in https://github.com/apache/opendal/pull/5626 ## [v0.51.2] - 2025-02-02 ### Added * feat(core): implement if_modified_since and if_unmodified_since for stat_with by @meteorgan in https://github.com/apache/opendal/pull/5528 * feat(layer/otelmetrics): add OtelMetricsLayer by @andylokandy in https://github.com/apache/opendal/pull/5524 * feat(integrations/object_store): implement put_opts and get_opts by @meteorgan in https://github.com/apache/opendal/pull/5513 * feat: Conditional reader for azblob, gcs, oss by @geetanshjuneja in https://github.com/apache/opendal/pull/5531 * feat(core): Add correctness check for read with if_xxx headers by @Xuanwo in https://github.com/apache/opendal/pull/5538 * feat(services/cos): Added user metadata support for cos service by @geetanshjuneja in https://github.com/apache/opendal/pull/5510 * feat(core): Implement list with deleted and versions for oss by @hoslo in https://github.com/apache/opendal/pull/5527 * feat(layer/otelmetrics): take meter when register by @andylokandy in https://github.com/apache/opendal/pull/5547 * feat(gcs): Convert TOO_MANY_REQUESTS to retryable Ratelimited by @Xuanwo in https://github.com/apache/opendal/pull/5551 * feat(services/webdfs): Add user.name support for webhdfs by @Xuanwo in https://github.com/apache/opendal/pull/5567 * feat: disable backtrace for NotFound error by @xxchan in https://github.com/apache/opendal/pull/5577 ### Changed * refactor: refactor some unnecessary clone and use next_back to make clippy happy by @yihong0618 in https://github.com/apache/opendal/pull/5554 * refactor: refactor all body.copy_to_bytes(body.remaining()) by @yihong0618 in https://github.com/apache/opendal/pull/5561 ### Fixed * fix(integrations/object_store) `object_store_opendal` now compiles on wasm32-unknown-unknown by @XiangpengHao in https://github.com/apache/opendal/pull/5530 * fix(serivces/gcs): Gcs doesn't support read with if_(un)modified_since by @Xuanwo in https://github.com/apache/opendal/pull/5537 * fix(logging): remove additional space by @xxchan in https://github.com/apache/opendal/pull/5568 ### Docs * docs: Fix opendal rust core's README not align with new vision by @Xuanwo in https://github.com/apache/opendal/pull/5541 * docs(integration/object_store): add example for datafusion by @meteorgan in https://github.com/apache/opendal/pull/5543 * docs: Add docs on how to pronounce opendal by @Xuanwo in https://github.com/apache/opendal/pull/5552 * docs(bindings/java): better javadoc by @tisonkun in https://github.com/apache/opendal/pull/5572 ### CI * ci(integration/object_store): add integration tests for object_store_opendal by @meteorgan in https://github.com/apache/opendal/pull/5536 * ci: Pin the nightly version to rust 1.84 for fuzz by @Xuanwo in https://github.com/apache/opendal/pull/5546 * ci: skip running behavior tests when adding or modifying documentation by @meteorgan in https://github.com/apache/opendal/pull/5558 * build: fix Cargo.lock and pass --locked in CI by @xxchan in https://github.com/apache/opendal/pull/5565 * build: implement release process in odev by @tisonkun in https://github.com/apache/opendal/pull/5592 ### Chore * chore: Update CODEOWNERS by @Xuanwo in https://github.com/apache/opendal/pull/5542 * chore(layer/otelmetrics): take meter by reference by @andylokandy in https://github.com/apache/opendal/pull/5553 * chore(core): Avoid using mongodb 3.2.0 by @Xuanwo in https://github.com/apache/opendal/pull/5560 * chore: add oli/oay/ofs to rust-analyzer.linkedProjects by @xxchan in https://github.com/apache/opendal/pull/5564 * chore: try use logforth by @tisonkun in https://github.com/apache/opendal/pull/5573 * chore: bump version 0.51.2 by @tisonkun in https://github.com/apache/opendal/pull/5595 ## [v0.51.1] - 2025-01-08 ### Added * feat(bin/oli): implement oli bench by @tisonkun in https://github.com/apache/opendal/pull/5443 * feat(dev): Add config parse and generate support by @Xuanwo in https://github.com/apache/opendal/pull/5454 * feat(bindings/python): generate python operator constructor types by @trim21 in https://github.com/apache/opendal/pull/5457 * feat(dev): Parse comments from config by @Xuanwo in https://github.com/apache/opendal/pull/5467 * feat(services/core): Implement stat_has_* and list_has_* correctly for services by @geetanshjuneja in https://github.com/apache/opendal/pull/5472 * feat: Add if-match & if-none-match support for reader by @XmchxUp in https://github.com/apache/opendal/pull/5492 * feat(core): Add is_current to metadata by @Wenbin1002 in https://github.com/apache/opendal/pull/5493 * feat(core): Implement list with deleted for s3 service by @Xuanwo in https://github.com/apache/opendal/pull/5498 * feat: generate java configs by @tisonkun in https://github.com/apache/opendal/pull/5503 * feat: Return hinted error for S3 wildcard if-none-match by @gruuya in https://github.com/apache/opendal/pull/5506 * feat(core): implement if_modified_since and if_unmodified_since for read_with and reader_with by @meteorgan in https://github.com/apache/opendal/pull/5500 * feat(core): Implement list with deleted and versions for cos by @hoslo in https://github.com/apache/opendal/pull/5514 ### Changed * refactor: tidy up oli build by @tisonkun in https://github.com/apache/opendal/pull/5438 * refactor(core): Deprecate OpList::version and add versions instead by @geetanshjuneja in https://github.com/apache/opendal/pull/5481 * refactor(dev): use minijinja by @tisonkun in https://github.com/apache/opendal/pull/5494 ### Fixed * fix: exception name in python by @trim21 in https://github.com/apache/opendal/pull/5453 * fix rust warning in python binding by @trim21 in https://github.com/apache/opendal/pull/5459 * fix: python binding kwargs parsing by @trim21 in https://github.com/apache/opendal/pull/5458 * fix(bindings/python): add py.typed marker file by @trim21 in https://github.com/apache/opendal/pull/5464 * fix(services/ghac): Fix stat_with_if_none_match been set in wrong by @Xuanwo in https://github.com/apache/opendal/pull/5477 * fix(ci): Correctly upgrade upload-artifact to v4 by @Xuanwo in https://github.com/apache/opendal/pull/5484 * fix(integration/object_store): object_store requires metadata in list by @Xuanwo in https://github.com/apache/opendal/pull/5501 * fix(services/s3): List with deleted should contain latest by @Xuanwo in https://github.com/apache/opendal/pull/5518 ### Docs * docs: Fix links to vision by @Xuanwo in https://github.com/apache/opendal/pull/5466 * docs(golang): remove unused pkg by @fyqtian in https://github.com/apache/opendal/pull/5473 * docs(core): Polish API docs for `Metadata` by @Xuanwo in https://github.com/apache/opendal/pull/5497 * docs: Polish docs for Operator, Reader and Writer by @Xuanwo in https://github.com/apache/opendal/pull/5516 * docs: Reorganize docs for xxx_with for better reading by @Xuanwo in https://github.com/apache/opendal/pull/5517 ### CI * ci: disable windows free-thread build by @trim21 in https://github.com/apache/opendal/pull/5449 * ci: Upgrade and fix typos by @Xuanwo in https://github.com/apache/opendal/pull/5468 ### Chore * chore(dev): Try just instead of xtasks methods by @Xuanwo in https://github.com/apache/opendal/pull/5461 * chore: pretty gen javadoc by @tisonkun in https://github.com/apache/opendal/pull/5508 * chore(ci): upgrade to manylinux_2_28 for aarch64 Python wheels by @messense in https://github.com/apache/opendal/pull/5522 ## [v0.51.0] - 2024-12-14 ### Added * feat(adapter/kv): support async iterating on scan results by @PragmaTwice in https://github.com/apache/opendal/pull/5208 * feat(bindings/ruby): Add simple operators to Ruby binding by @erickguan in https://github.com/apache/opendal/pull/5246 * feat(core/services-gcs): support user defined metadata by @jorgehermo9 in https://github.com/apache/opendal/pull/5276 * feat(core): add `if_not_exist` in `OpWrite` by @kemingy in https://github.com/apache/opendal/pull/5305 * feat: Add {stat,list}_has_* to carry the metadata that backend returns by @Xuanwo in https://github.com/apache/opendal/pull/5318 * feat(core): Implement write if not exists for azblob,azdls,gcs,oss,cos by @Xuanwo in https://github.com/apache/opendal/pull/5321 * feat(core): add new cap shared by @TennyZhuang in https://github.com/apache/opendal/pull/5328 * feat(bindings/python): support pickle [de]serialization for Operator by @TennyZhuang in https://github.com/apache/opendal/pull/5324 * feat(bindings/cpp): init the async support of C++ binding by @PragmaTwice in https://github.com/apache/opendal/pull/5195 * feat(bindings/go): support darwin by @yuchanns in https://github.com/apache/opendal/pull/5334 * feat(services/gdrive): List shows modified timestamp gdrive by @erickguan in https://github.com/apache/opendal/pull/5226 * feat(service/s3): support delete with version by @Frank-III in https://github.com/apache/opendal/pull/5349 * feat: upgrade pyo3 to 0.23 by @XmchxUp in https://github.com/apache/opendal/pull/5368 * feat: publish python3.13t free-threaded wheel by @XmchxUp in https://github.com/apache/opendal/pull/5387 * feat: add progress bar for oli cp command by @waynexia in https://github.com/apache/opendal/pull/5369 * feat(types/buffer): skip copying in `to_bytes` when `NonContiguous` contains a single `Bytes` by @ever0de in https://github.com/apache/opendal/pull/5388 * feat(bin/oli): support command mv by @meteorgan in https://github.com/apache/opendal/pull/5370 * feat(core): add if-match to `OpWrite` by @Frank-III in https://github.com/apache/opendal/pull/5360 * feat(core/layers): add correctness_check and capability_check layer to verify whether the operation and arguments is supported by @meteorgan in https://github.com/apache/opendal/pull/5352 * feat(bindings/ruby): Add I/O class for Ruby by @erickguan in https://github.com/apache/opendal/pull/5354 * feat(core): Add `content_encoding` to `MetaData` by @Frank-III in https://github.com/apache/opendal/pull/5400 * feat:(core): add `content encoding` to `Opwrite` by @Frank-III in https://github.com/apache/opendal/pull/5390 * feat(services/obs): support user defined metadata by @Frank-III in https://github.com/apache/opendal/pull/5405 * feat: impl configurable OperatorOutputStream maxBytes by @tisonkun in https://github.com/apache/opendal/pull/5422 ### Changed * refactor (bindings/zig): Improvements by @kassane in https://github.com/apache/opendal/pull/5247 * refactor: Remove metakey concept by @Xuanwo in https://github.com/apache/opendal/pull/5319 * refactor(core)!: Remove not used cap write_multi_align_size by @Xuanwo in https://github.com/apache/opendal/pull/5322 * refactor(core)!: Remove the range writer that has never been used by @Xuanwo in https://github.com/apache/opendal/pull/5323 * refactor(core): MaybeSend does not need to be unsafe by @drmingdrmer in https://github.com/apache/opendal/pull/5338 * refactor: Implement RFC-3911 Deleter API by @Xuanwo in https://github.com/apache/opendal/pull/5392 * refactor: Remove batch concept from opendal by @Xuanwo in https://github.com/apache/opendal/pull/5393 ### Fixed * fix(services/webdav): Fix lister failing when root contains spaces by @skrimix in https://github.com/apache/opendal/pull/5298 * fix(bindings/c): Bump min CMake version to support CMP0135 by @palash25 in https://github.com/apache/opendal/pull/5308 * fix(services/webhdfs): rename auth value by @notauserx in https://github.com/apache/opendal/pull/5342 * fix(bindings/cpp): remove the warning of CMP0135 by @PragmaTwice in https://github.com/apache/opendal/pull/5346 * build(python): fix pyproject meta file by @trim21 in https://github.com/apache/opendal/pull/5348 * fix(services/unftp): add `/` when not presented by @Frank-III in https://github.com/apache/opendal/pull/5382 * fix: update document against target format check and add hints by @waynexia in https://github.com/apache/opendal/pull/5361 * fix: oli clippy and CI file by @waynexia in https://github.com/apache/opendal/pull/5389 * fix(services/obs): support huawei.com by @FayeSpica in https://github.com/apache/opendal/pull/5399 * fix(integrations/cloud_filter): use explicit `stat` instead of `Entry::metadata` in `fetch_placeholders` by @ho-229 in https://github.com/apache/opendal/pull/5416 * fix(core): S3 multipart uploads does not set file metadata by @catcatmu in https://github.com/apache/opendal/pull/5430 * fix: always contains path label if configured by @waynexia in https://github.com/apache/opendal/pull/5433 ### Docs * docs: Enable force_orphan to reduce clone size by @Xuanwo in https://github.com/apache/opendal/pull/5289 * docs: Establish VISION for "One Layer, All Storage" by @Xuanwo in https://github.com/apache/opendal/pull/5309 * docs: Polish docs for write with if not exists by @Xuanwo in https://github.com/apache/opendal/pull/5320 * docs(core): add the description of version parameter for operator by @meteorgan in https://github.com/apache/opendal/pull/5144 * docs(core): Add upgrade to v0.51 by @Xuanwo in https://github.com/apache/opendal/pull/5406 * docs: Update release.md by @tisonkun in https://github.com/apache/opendal/pull/5431 ### CI * ci: Remove the token of codspeed by @Xuanwo in https://github.com/apache/opendal/pull/5283 * ci: Allow force push for `gh-pages` by @Xuanwo in https://github.com/apache/opendal/pull/5290 * build(bindings/java): fix lombok process by @tisonkun in https://github.com/apache/opendal/pull/5297 * build(bindings/python): add python 3.10 back and remove pypy by @trim21 in https://github.com/apache/opendal/pull/5347 ### Chore * chore(core/layers): align `info` method of `trait Access` and `trait LayeredAccess` by @koushiro in https://github.com/apache/opendal/pull/5258 * chore(core/raw): align `info` method of `kv::Adapter` and `typed_kv::Adapter` by @koushiro in https://github.com/apache/opendal/pull/5259 * chore(layers/oteltrace): adjust tracer span name of info method by @koushiro in https://github.com/apache/opendal/pull/5285 * chore(services/s3): remove versioning check for s3 by @meteorgan in https://github.com/apache/opendal/pull/5300 * chore: Polish the debug output of capability by @Xuanwo in https://github.com/apache/opendal/pull/5315 * chore: Update maturity.md by @tisonkun in https://github.com/apache/opendal/pull/5340 * chore: remove flagset in cargo.lock by @meteorgan in https://github.com/apache/opendal/pull/5355 * chore: add setup action for hadoop to avoid build failures by @meteorgan in https://github.com/apache/opendal/pull/5353 * chore: fix cargo clippy by @meteorgan in https://github.com/apache/opendal/pull/5379 * chore: fix cargo clippy by @meteorgan in https://github.com/apache/opendal/pull/5384 * chore: fix Bindings OCaml CI by @meteorgan in https://github.com/apache/opendal/pull/5386 * chore: Add default vscode config for more friendly developer experience by @Zheaoli in https://github.com/apache/opendal/pull/5331 * chore(website): remove outdated description by @meteorgan in https://github.com/apache/opendal/pull/5411 * chore(deps): bump clap from 4.5.20 to 4.5.21 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/5372 * chore(deps): bump anyhow from 1.0.90 to 1.0.93 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/5375 * chore(deps): bump serde from 1.0.210 to 1.0.215 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/5376 * chore(deps): bump openssh-sftp-client from 0.15.1 to 0.15.2 in /core by @dependabot in https://github.com/apache/opendal/pull/5377 * chore(ci): fix invalid Behavior Test Integration Cloud Filter trigger by @Zheaoli in https://github.com/apache/opendal/pull/5414 ## [v0.50.2] - 2024-11-04 ### Added * feat(services/ftp): List dir shows last modified timestamp by @erickguan in https://github.com/apache/opendal/pull/5213 * feat(bindings/d): add D bindings support by @kassane in https://github.com/apache/opendal/pull/5181 * feat(bindings/python): add sync `File.readline` by @TennyZhuang in https://github.com/apache/opendal/pull/5271 * feat(core/services-azblob): support user defined metadata by @jorgehermo9 in https://github.com/apache/opendal/pull/5274 * feat(core/services-s3): try load endpoint from config by @TennyZhuang in https://github.com/apache/opendal/pull/5279 ### Changed * refactor(bin/oli): use `clap_derive` to reduce boilerplate code by @koushiro in https://github.com/apache/opendal/pull/5233 ### Fixed * fix: add all-features flag for opendal_compat doc build by @XmchxUp in https://github.com/apache/opendal/pull/5234 * fix(integrations/compat): Capability has different fields by @Xuanwo in https://github.com/apache/opendal/pull/5236 * fix(integration/compat): Fix opendal 0.50 OpList has new field by @Xuanwo in https://github.com/apache/opendal/pull/5238 * fix(integrations/compat): Fix dead loop happened during list by @Xuanwo in https://github.com/apache/opendal/pull/5240 ### Docs * docs: Move our release process to github discussions by @Xuanwo in https://github.com/apache/opendal/pull/5217 * docs: change "Github" to "GitHub" by @MohammadLotfiA in https://github.com/apache/opendal/pull/5250 ### CI * ci(asf): Don't add `[DISCUSS]` prefix for discussion by @Xuanwo in https://github.com/apache/opendal/pull/5210 * build: enable services-mysql for Java and Python bindings by @tisonkun in https://github.com/apache/opendal/pull/5222 * build(binding/python): Support Python 3.13 by @Zheaoli in https://github.com/apache/opendal/pull/5248 ### Chore * chore(bin/*): remove useless deps by @koushiro in https://github.com/apache/opendal/pull/5212 * chore: tidy up c binding build and docs by @tisonkun in https://github.com/apache/opendal/pull/5243 * chore(core/layers): adjust await point to simplify combinator code by @koushiro in https://github.com/apache/opendal/pull/5255 * chore(core/blocking_operator): deduplicate deprecated `is_exist` logic by @simonsan in https://github.com/apache/opendal/pull/5261 * chore(deps): bump actions/cache from 3 to 4 by @dependabot in https://github.com/apache/opendal/pull/5262 * chore: run object_store tests in CI by @jorgehermo9 in https://github.com/apache/opendal/pull/5268 ## [v0.50.1] - 2024-10-20 ### Added * feat(core/redis): Replace client requests with connection pool by @jackyyyyyssss in https://github.com/apache/opendal/pull/5117 * feat: add copy api for lakefs service. by @liugddx in https://github.com/apache/opendal/pull/5114 * feat(core): add version(bool) for List operation to include version d… by @meteorgan in https://github.com/apache/opendal/pull/5106 * feat(bindings/python): export ConcurrentLimitLayer by @TennyZhuang in https://github.com/apache/opendal/pull/5140 * feat(bindings/c): add writer operation for Bindings C and Go by @yuchanns in https://github.com/apache/opendal/pull/5141 * feat(ofs): introduce ofs macos support by @oowl in https://github.com/apache/opendal/pull/5136 * feat: Reduce stat operation if we are reading all by @Xuanwo in https://github.com/apache/opendal/pull/5146 * feat: add NebulaGraph config by @GG2002 in https://github.com/apache/opendal/pull/5147 * feat(integrations/spring): add spring serialize method by @shoothzj in https://github.com/apache/opendal/pull/5154 * feat: support write,read,delete with template by @shoothzj in https://github.com/apache/opendal/pull/5156 * feat(bindings/java): support ConcurrentLimitLayer by @tisonkun in https://github.com/apache/opendal/pull/5168 * feat: Add if_none_match for write by @ForestLH in https://github.com/apache/opendal/pull/5129 * feat: Add OpenDAL Compat by @Xuanwo in https://github.com/apache/opendal/pull/5185 * feat(core): abstract HttpFetch trait for raw http client by @everpcpc in https://github.com/apache/opendal/pull/5184 * feat: Support NebulaGraph by @GG2002 in https://github.com/apache/opendal/pull/5116 * feat(bindings/cpp): rename is_exist to exists as core did by @PragmaTwice in https://github.com/apache/opendal/pull/5198 * feat(bindings/c): add opendal_operator_exists and mark is_exist deprecated by @PragmaTwice in https://github.com/apache/opendal/pull/5199 * feat(binding/java): prefix thread name with opendal-tokio-worker by @tisonkun in https://github.com/apache/opendal/pull/5197 ### Changed * refactor(services/cloudflare-kv): remove unneeded async and result on parse_error by @tsfotis in https://github.com/apache/opendal/pull/5128 * refactor(*): remove unneeded async and result on parse_error by @tsfotis in https://github.com/apache/opendal/pull/5131 * refactor: align C binding pattern by @tisonkun in https://github.com/apache/opendal/pull/5160 * refactor: more consistent C binding pattern by @tisonkun in https://github.com/apache/opendal/pull/5162 * refactor(integration/parquet): Use ParquetMetaDataReader instead by @Xuanwo in https://github.com/apache/opendal/pull/5170 * refactor: resolve c pointers const by @tisonkun in https://github.com/apache/opendal/pull/5171 * refactor(types/operator): rename is_exist to exists by @photino in https://github.com/apache/opendal/pull/5193 ### Fixed * fix(services/huggingface): Align with latest HuggingFace API by @morristai in https://github.com/apache/opendal/pull/5123 * fix(bindings/c): use `ManuallyDrop` instead of `forget` to make sure pointer is valid by @ethe in https://github.com/apache/opendal/pull/5166 * fix(services/s3): Mark xml deserialize error as temporary during list by @Xuanwo in https://github.com/apache/opendal/pull/5178 ### Docs * docs: add spring integration configuration doc by @shoothzj in https://github.com/apache/opendal/pull/5053 * docs: improve Node.js binding's test doc by @tisonkun in https://github.com/apache/opendal/pull/5159 * docs(bindings/c): update docs for CMake replacing by @PragmaTwice in https://github.com/apache/opendal/pull/5186 ### CI * ci(bindings/nodejs): Fix diff introduced by napi by @Xuanwo in https://github.com/apache/opendal/pull/5121 * ci: Disable aliyun drive test until #5163 addressed by @Xuanwo in https://github.com/apache/opendal/pull/5164 * ci: add package cache for build-haskell-doc by @XmchxUp in https://github.com/apache/opendal/pull/5173 * ci: add cache action for ci_bindings_ocaml & build-ocaml-doc by @XmchxUp in https://github.com/apache/opendal/pull/5174 * ci: Fix failing CI on ocaml and python by @Xuanwo in https://github.com/apache/opendal/pull/5177 * build(bindings/c): replace the build system with CMake by @PragmaTwice in https://github.com/apache/opendal/pull/5182 * build(bindings/cpp): fetch and build dependencies instead of finding system libs by @PragmaTwice in https://github.com/apache/opendal/pull/5188 * ci: Remove not needed --break-system-packages by @Xuanwo in https://github.com/apache/opendal/pull/5196 * ci: Send discussions to dev@o.a.o by @Xuanwo in https://github.com/apache/opendal/pull/5201 ### Chore * chore(bindings/python): deprecate via_map method by @TennyZhuang in https://github.com/apache/opendal/pull/5134 * chore: update binding java artifact name in README by @tisonkun in https://github.com/apache/opendal/pull/5137 * chore(fixtures/s3): Upgrade MinIO version by @ForestLH in https://github.com/apache/opendal/pull/5142 * chore(deps): bump clap from 4.5.17 to 4.5.18 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/5149 * chore(deps): bump crate-ci/typos from 1.24.3 to 1.24.6 by @dependabot in https://github.com/apache/opendal/pull/5150 * chore(deps): bump anyhow from 1.0.87 to 1.0.89 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/5151 * chore(deps): bump anyhow from 1.0.87 to 1.0.89 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/5152 * chore: fix typos in tokio_executor.rs by @tisonkun in https://github.com/apache/opendal/pull/5157 * chore: hint when java tests are skipped by @tisonkun in https://github.com/apache/opendal/pull/5158 * chore: Include license in the packaged crate by @davide125 in https://github.com/apache/opendal/pull/5176 ## [v0.50.0] - 2024-09-11 ### Added * feat(core)!: make list return path itself by @meteorgan in https://github.com/apache/opendal/pull/4959 * feat(services/oss): support role_arn and oidc_provider_arn by @tisonkun in https://github.com/apache/opendal/pull/5063 * feat(services): add lakefs support by @liugddx in https://github.com/apache/opendal/pull/5086 * feat: add list api for lakefs service. by @liugddx in https://github.com/apache/opendal/pull/5092 * feat: add write api for lakefs service. by @liugddx in https://github.com/apache/opendal/pull/5100 * feat: add delete api for lakefs service. by @liugddx in https://github.com/apache/opendal/pull/5107 ### Changed * refactor: use sqlx for sql services by @tisonkun in https://github.com/apache/opendal/pull/5040 * refactor(core)!: Add observe layer as building block by @Xuanwo in https://github.com/apache/opendal/pull/5064 * refactor(layers/prometheus): rewrite prometheus layer based on observe mod by @koushiro in https://github.com/apache/opendal/pull/5069 * refactor(bindings/java): replace `num_cpus` with `std::thread::available_parallelism` by @miroim in https://github.com/apache/opendal/pull/5080 * refactor(layers/prometheus): provide builder APIs by @koushiro in https://github.com/apache/opendal/pull/5072 * refactor(layers/prometheus-client): provide builder APIs by @koushiro in https://github.com/apache/opendal/pull/5073 * refactor(layers/metrics): rewrite metrics layer using observe layer by @koushiro in https://github.com/apache/opendal/pull/5098 ### Fixed * fix(core): TimeoutLayer now needs enable tokio time by @Xuanwo in https://github.com/apache/opendal/pull/5057 * fix(core): Fix failed list related tests by @Xuanwo in https://github.com/apache/opendal/pull/5058 * fix(services/memory): blocking_scan right range by @meteorgan in https://github.com/apache/opendal/pull/5062 * fix(core/services/mysql): Fix mysql Capability by @jackyyyyyssss in https://github.com/apache/opendal/pull/5067 * fix: fix rust 1.76 error due to temporary value being dropped by @aawsome in https://github.com/apache/opendal/pull/5071 * fix(service/fs): error due to temporary value being dropped by @miroim in https://github.com/apache/opendal/pull/5079 * fix(core/services/hdfs): Fix the HDFS write failure when atomic_write_dir is set by @meteorgan in https://github.com/apache/opendal/pull/5039 * fix(services/icloud): adjust error handling code to avoid having to write out result type explicitly by @koushiro in https://github.com/apache/opendal/pull/5091 * fix(services/monoiofs): handle async cancel during file open by @NKID00 in https://github.com/apache/opendal/pull/5094 ### Docs * docs: Update binding-java.md by @tisonkun in https://github.com/apache/opendal/pull/5087 ### CI * ci(bindings/go): add golangci-lint by @yuchanns in https://github.com/apache/opendal/pull/5060 * ci(bindings/zig): Fix build and test of zig on 0.13 by @Xuanwo in https://github.com/apache/opendal/pull/5102 * ci: Don't publish with all features by @Xuanwo in https://github.com/apache/opendal/pull/5108 * ci: Fix upload-artifacts doesn't include hidden files by @Xuanwo in https://github.com/apache/opendal/pull/5112 ### Chore * chore(bindings/go): bump ffi and sys version by @shoothzj in https://github.com/apache/opendal/pull/5055 * chore: Bump backon to 1.0.0 by @Xuanwo in https://github.com/apache/opendal/pull/5056 * chore(services/rocksdb): fix misuse rocksdb prefix iterator by @meteorgan in https://github.com/apache/opendal/pull/5059 * chore(README): add Go binding badge by @yuchanns in https://github.com/apache/opendal/pull/5074 * chore(deps): bump crate-ci/typos from 1.23.6 to 1.24.3 by @dependabot in https://github.com/apache/opendal/pull/5085 * chore(layers/prometheus-client): export `PrometheusClientLayerBuilder` type by @koushiro in https://github.com/apache/opendal/pull/5093 * chore(layers): check the examples when running tests by @koushiro in https://github.com/apache/opendal/pull/5104 * chore(integrations/parquet): Bump parquet to 53 by @Xuanwo in https://github.com/apache/opendal/pull/5109 * chore: Bump OpenDAL to 0.50.0 by @Xuanwo in https://github.com/apache/opendal/pull/5110 ## [v0.49.2] - 2024-08-26 ### Added * feat(ovfs): support read and write by @zjregee in https://github.com/apache/opendal/pull/5016 * feat(bin/ofs): introduce `integrations/cloudfilter` for ofs by @ho-229 in https://github.com/apache/opendal/pull/4935 * feat(integrations/spring): add AutoConfiguration class for Spring Mvc and Webflux by @shoothzj in https://github.com/apache/opendal/pull/5019 * feat(services/monoiofs): impl read and write, add behavior test by @NKID00 in https://github.com/apache/opendal/pull/4944 * feat(core/services-s3): support user defined metadata by @haoqixu in https://github.com/apache/opendal/pull/5030 * feat: align `fn root` semantics; fix missing root for some services; rm duplicated normalize ops by @yjhmelody in https://github.com/apache/opendal/pull/5035 * feat(core): expose configs always by @tisonkun in https://github.com/apache/opendal/pull/5034 * feat(services/monoiofs): append, create_dir, copy and rename by @NKID00 in https://github.com/apache/opendal/pull/5041 ### Changed * refactor(core): new type to print context and reduce allocations by @evenyag in https://github.com/apache/opendal/pull/5021 * refactor(layers/prometheus-client): remove useless `scheme` field from `PrometheusAccessor` and `PrometheusMetricWrapper` type by @koushiro in https://github.com/apache/opendal/pull/5026 * refactor(layers/prometheus-client): avoid multiple clone of labels by @koushiro in https://github.com/apache/opendal/pull/5028 * refactor(core/services-oss): remove the `starts_with` by @haoqixu in https://github.com/apache/opendal/pull/5036 ### Fixed * fix(layers/prometheus-client): remove duplicated `increment_request_total` of write operation by @koushiro in https://github.com/apache/opendal/pull/5023 * fix(services/monoiofs): drop JoinHandle in worker thread by @NKID00 in https://github.com/apache/opendal/pull/5031 ### CI * ci: Add contents write permission for build-website by @Xuanwo in https://github.com/apache/opendal/pull/5017 * ci: Fix test for service ghac by @Xuanwo in https://github.com/apache/opendal/pull/5018 * ci(integrations/spring): add spring boot bean load test by @shoothzj in https://github.com/apache/opendal/pull/5032 ### Chore * chore: fix path typo in release docs by @tisonkun in https://github.com/apache/opendal/pull/5038 * chore: align the `token` method semantics by @yjhmelody in https://github.com/apache/opendal/pull/5045 ## [v0.49.1] - 2024-08-15 ### Added * feat(ovfs): add lookup and unit tests by @zjregee in https://github.com/apache/opendal/pull/4997 * feat(gcs): allow setting a token directly by @jdockerty in https://github.com/apache/opendal/pull/4978 * feat(integrations/cloudfilter): introduce behavior tests by @ho-229 in https://github.com/apache/opendal/pull/4973 * feat(integrations/spring): add spring project module by @shoothzj in https://github.com/apache/opendal/pull/4988 * feat(fs): expose the metadata for fs services by @Aitozi in https://github.com/apache/opendal/pull/5005 * feat(ovfs): add file creation and deletion by @zjregee in https://github.com/apache/opendal/pull/5009 ### Fixed * fix(integrations/spring): correct parent artifactId by @shoothzj in https://github.com/apache/opendal/pull/5007 * fix(bindings/python): Make sure read until EOF by @Bicheka in https://github.com/apache/opendal/pull/4995 ### Docs * docs: Fix version detect in website by @Xuanwo in https://github.com/apache/opendal/pull/5003 * docs: add branding, license and trademarks to integrations by @PsiACE in https://github.com/apache/opendal/pull/5006 * docs(integrations/cloudfilter): improve docs and examples by @ho-229 in https://github.com/apache/opendal/pull/5010 ### CI * ci(bindings/python): Fix aws-lc-rs build on arm platforms by @Xuanwo in https://github.com/apache/opendal/pull/5004 ### Chore * chore(deps): bump fastrace to 0.7.1 by @andylokandy in https://github.com/apache/opendal/pull/5008 * chore(bindings): Disable mysql service for java and python by @Xuanwo in https://github.com/apache/opendal/pull/5013 ## [v0.49.0] - 2024-08-09 ### Added * feat(o): Add cargo-o layout by @Xuanwo in https://github.com/apache/opendal/pull/4934 * feat: impl `put_multipart` in `object_store` by @Rachelint in https://github.com/apache/opendal/pull/4793 * feat: introduce opendal `AsyncWriter` for parquet integrations by @WenyXu in https://github.com/apache/opendal/pull/4958 * feat(services/http): implement presigned request for backends without authorization by @NickCao in https://github.com/apache/opendal/pull/4970 * feat(bindings/python): strip the library for minimum file size by @NickCao in https://github.com/apache/opendal/pull/4971 * feat(gcs): allow unauthenticated requests by @jdockerty in https://github.com/apache/opendal/pull/4965 * feat: introduce opendal `AsyncReader` for parquet integrations by @WenyXu in https://github.com/apache/opendal/pull/4972 * feat(services/s3): add role_session_name in assume roles by @nerdroychan in https://github.com/apache/opendal/pull/4981 * feat: support root path for moka and mini-moka by @meteorgan in https://github.com/apache/opendal/pull/4984 * feat(ovfs): export VirtioFs struct by @zjregee in https://github.com/apache/opendal/pull/4983 * feat(core)!: implement an interceptor for the logging layer by @evenyag in https://github.com/apache/opendal/pull/4961 * feat(ovfs): support getattr and setattr by @zjregee in https://github.com/apache/opendal/pull/4987 ### Changed * refactor(java)!: Rename artifacts id opendal-java to opendal by @Xuanwo in https://github.com/apache/opendal/pull/4957 * refactor(core)!: Return associated builder instead by @Xuanwo in https://github.com/apache/opendal/pull/4968 * refactor(raw): Merge all operations into one enum by @Xuanwo in https://github.com/apache/opendal/pull/4977 * refactor(core): Use kv based context to avoid allocations by @Xuanwo in https://github.com/apache/opendal/pull/4986 ### Fixed * fix(services/memory): MemoryConfig implement Debug by @0x676e67 in https://github.com/apache/opendal/pull/4942 * fix(layers/promethues-client): doc link by @koushiro in https://github.com/apache/opendal/pull/4951 * fix(gcs): do not skip signing with `allow_anonymous` by @jdockerty in https://github.com/apache/opendal/pull/4979 ### Docs * docs: nominate-committer add announcement template by @tisonkun in https://github.com/apache/opendal/pull/4954 ### CI * ci: Bump nextest to 0.9.72 by @Xuanwo in https://github.com/apache/opendal/pull/4932 * ci: setup cloudfilter by @ho-229 in https://github.com/apache/opendal/pull/4936 * ci: Try fix opendal-lua build by @Xuanwo in https://github.com/apache/opendal/pull/4952 ### Chore * chore(deps): bump crate-ci/typos from 1.22.9 to 1.23.6 by @dependabot in https://github.com/apache/opendal/pull/4948 * chore(deps): bump tokio from 1.39.1 to 1.39.2 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/4949 * chore(deps): bump bytes from 1.6.1 to 1.7.0 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/4947 * chore(deps): bump tokio from 1.39.1 to 1.39.2 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/4946 * chore(core): fix nightly lints by @xxchan in https://github.com/apache/opendal/pull/4953 * chore(integrations/parquet): add README by @WenyXu in https://github.com/apache/opendal/pull/4980 * chore(core): Bump redis version by @Xuanwo in https://github.com/apache/opendal/pull/4985 * chore: Bump package versions by @Xuanwo in https://github.com/apache/opendal/pull/4989 ## [v0.48.0] - 2024-07-26 ### Added * feat(services/fs): Support fs config by @meteorgan in https://github.com/apache/opendal/pull/4853 * feat(services): init monoiofs by @NKID00 in https://github.com/apache/opendal/pull/4855 * feat(core/types): avoid a copy in `Buffer::to_bytes()` by cloning contiguous bytes by @LDeakin in https://github.com/apache/opendal/pull/4858 * feat(core): Add object versioning for OSS by @Lzzzzzt in https://github.com/apache/opendal/pull/4870 * feat: fs add concurrent write by @hoslo in https://github.com/apache/opendal/pull/4817 * feat(services/s3): Add object versioning for S3 by @Lzzzzzt in https://github.com/apache/opendal/pull/4873 * feat(integrations/cloudfilter): read only cloud filter by @ho-229 in https://github.com/apache/opendal/pull/4856 * feat(bindings/go): Add full native support from C to Go. by @yuchanns in https://github.com/apache/opendal/pull/4886 * feat(bindings/go): add benchmark. by @yuchanns in https://github.com/apache/opendal/pull/4893 * feat(core): support user defined metadata for oss by @meteorgan in https://github.com/apache/opendal/pull/4881 * feat(service/fastrace): rename minitrace to fastrace by @andylokandy in https://github.com/apache/opendal/pull/4906 * feat(prometheus-client): add metric label about `root` on using PrometheusClientLayer by @flaneur2020 in https://github.com/apache/opendal/pull/4907 * feat(services/monoiofs): monoio wrapper by @NKID00 in https://github.com/apache/opendal/pull/4885 * feat(layers/mime-guess): add a layer that can automatically set `Content-Type` based on the extension in the path. by @czy-29 in https://github.com/apache/opendal/pull/4912 * feat(core)!: Make config data object by @tisonkun in https://github.com/apache/opendal/pull/4915 * feat(core)!: from_map is now fallible by @tisonkun in https://github.com/apache/opendal/pull/4917 * ci(bindings/go): always test against the latest core by @yuchanns in https://github.com/apache/opendal/pull/4913 * feat(!): Allow users to build operator from config by @Xuanwo in https://github.com/apache/opendal/pull/4919 * feat: Add from_iter and via_iter for operator by @Xuanwo in https://github.com/apache/opendal/pull/4921 ### Changed * refactor(services/s3)!: renamed security_token to session_token by @Zyyeric in https://github.com/apache/opendal/pull/4875 * refactor(core)!: Make oio::Write always write all given buffer by @Xuanwo in https://github.com/apache/opendal/pull/4880 * refactor(core)!: Return `Arc` for metadata by @Lzzzzzt in https://github.com/apache/opendal/pull/4883 * refactor(core!): Make service builder takes ownership by @Xuanwo in https://github.com/apache/opendal/pull/4922 * refactor(integrations/cloudfilter): implement Filter instead of SyncFilter by @ho-229 in https://github.com/apache/opendal/pull/4920 ### Fixed * fix(services/s3): NoSuchBucket is a ConfigInvalid for OpenDAL by @tisonkun in https://github.com/apache/opendal/pull/4895 * fix: oss will not use the port by @Lzzzzzt in https://github.com/apache/opendal/pull/4899 ### Docs * docs(core): update README to add `MimeGuessLayer`. by @czy-29 in https://github.com/apache/opendal/pull/4916 * docs(core): Add upgrade docs for 0.48 by @Xuanwo in https://github.com/apache/opendal/pull/4924 * docs: fix spelling by @jbampton in https://github.com/apache/opendal/pull/4925 * docs(core): Fix comment for into_futures_async_write by @Xuanwo in https://github.com/apache/opendal/pull/4928 ### CI * ci: Add issue template and pr template for opendal by @Xuanwo in https://github.com/apache/opendal/pull/4884 * ci: Remove CI reviewer since it doesn't work by @Xuanwo in https://github.com/apache/opendal/pull/4891 ### Chore * chore!: fix typo customed should be customized by @tisonkun in https://github.com/apache/opendal/pull/4847 * chore: Fix spelling by @jbampton in https://github.com/apache/opendal/pull/4864 * chore: remove unneeded duplicate word by @jbampton in https://github.com/apache/opendal/pull/4865 * chore: fix spelling by @jbampton in https://github.com/apache/opendal/pull/4866 * chore: fix spelling by @NKID00 in https://github.com/apache/opendal/pull/4869 * chore: Make compfs able to test by @Xuanwo in https://github.com/apache/opendal/pull/4878 * chore(services/compfs): remove allow(dead_code) by @George-Miao in https://github.com/apache/opendal/pull/4879 * chore: Make rust 1.80 clippy happy by @Xuanwo in https://github.com/apache/opendal/pull/4927 * chore: Bump crates versions by @Xuanwo in https://github.com/apache/opendal/pull/4929 ## [v0.47.3] - 2024-07-03 ### Changed * refactor: Move ChunkedWrite logic into WriteContext by @Xuanwo in https://github.com/apache/opendal/pull/4826 * refactor(services/aliyun-drive): directly implement `oio::Write`. by @yuchanns in https://github.com/apache/opendal/pull/4821 ### Fixed * fix(integration/object_store): Avoid calling API inside debug by @Xuanwo in https://github.com/apache/opendal/pull/4846 * fix(integration/object_store): Fix metakey requested is incomplete by @Xuanwo in https://github.com/apache/opendal/pull/4844 ### Docs * docs(integration/unftp-sbe): Polish docs for unftp-sbe by @Xuanwo in https://github.com/apache/opendal/pull/4838 * docs(bin): Polish README for all bin by @Xuanwo in https://github.com/apache/opendal/pull/4839 ### Chore * chore(deps): bump crate-ci/typos from 1.22.7 to 1.22.9 by @dependabot in https://github.com/apache/opendal/pull/4836 * chore(deps): bump quick-xml from 0.32.0 to 0.35.0 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/4835 * chore(deps): bump nix from 0.28.0 to 0.29.0 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/4833 * chore(deps): bump metrics from 0.20.1 to 0.23.0 in /core by @TennyZhuang in https://github.com/apache/opendal/pull/4843 ## [v0.47.2] - 2024-06-30 ### Added * feat(services/compfs): basic `Access` impl by @George-Miao in https://github.com/apache/opendal/pull/4693 * feat(unftp-sbe): impl `OpendalStorage` by @George-Miao in https://github.com/apache/opendal/pull/4765 * feat(services/compfs): implement auxiliary functions by @George-Miao in https://github.com/apache/opendal/pull/4778 * feat: make AwaitTreeLayer covers oio::Read and oio::Write by @PsiACE in https://github.com/apache/opendal/pull/4787 * feat: Nodejs add devbox by @bxb100 in https://github.com/apache/opendal/pull/4791 * feat: make AsyncBacktraceLayer covers oio::Read and oio::Write by @PsiACE in https://github.com/apache/opendal/pull/4789 * feat(nodejs): add `WriteOptions` for write methods by @bxb100 in https://github.com/apache/opendal/pull/4785 * feat: setup cloud filter integration by @ho-229 in https://github.com/apache/opendal/pull/4779 * feat: add position write by @hoslo in https://github.com/apache/opendal/pull/4795 * fix(core): write concurrent doesn't set correctly by @hoslo in https://github.com/apache/opendal/pull/4816 * feat(ovfs): add filesystem to handle message by @zjregee in https://github.com/apache/opendal/pull/4720 * feat(unftp-sbe): add derives for `OpendalMetadata` by @George-Miao in https://github.com/apache/opendal/pull/4819 * feat(core/gcs): Add concurrent write for gcs back by @Xuanwo in https://github.com/apache/opendal/pull/4820 ### Changed * refactor(nodejs)!: Remove append api by @bxb100 in https://github.com/apache/opendal/pull/4796 * refactor(core): Remove unused layer `MadsimLayer` by @zzzk1 in https://github.com/apache/opendal/pull/4788 ### Fixed * fix(services/aliyun-drive): list dir without trailing slash by @yuchanns in https://github.com/apache/opendal/pull/4766 * fix(unftp-sbe): remove buffer for get by @George-Miao in https://github.com/apache/opendal/pull/4775 * fix(services/aliyun-drive): write op cannot overwrite existing files by @yuchanns in https://github.com/apache/opendal/pull/4781 * fix(core/services/onedrive): remove @odata.count for onedrive list op by @imWildCat in https://github.com/apache/opendal/pull/4803 * fix(core): Gcs's RangeWrite doesn't support concurrent write by @Xuanwo in https://github.com/apache/opendal/pull/4806 * fix(tests/behavior): skip test of write_with_overwrite for ghac by @yuchanns in https://github.com/apache/opendal/pull/4823 * fix(docs): some typos in website and nodejs binding docs by @suyanhanx in https://github.com/apache/opendal/pull/4814 * fix(core/aliyun_drive): Fix write_multi_max_size might overflow by @Xuanwo in https://github.com/apache/opendal/pull/4830 ### Docs * doc(unftp-sbe): adds example and readme by @George-Miao in https://github.com/apache/opendal/pull/4777 * doc(nodejs): update upgrade.md by @bxb100 in https://github.com/apache/opendal/pull/4799 * docs: Add README and rustdoc for fuse3_opendal by @Xuanwo in https://github.com/apache/opendal/pull/4813 * docs: use version variable in gradle, same to maven by @shoothzj in https://github.com/apache/opendal/pull/4824 ### CI * ci: set behavior test ci for aliyun drive by @suyanhanx in https://github.com/apache/opendal/pull/4657 * ci: Fix lib-darwin-x64 no released by @Xuanwo in https://github.com/apache/opendal/pull/4798 * ci(unftp-sbe): init by @George-Miao in https://github.com/apache/opendal/pull/4809 * ci: Build docs for all integrations by @Xuanwo in https://github.com/apache/opendal/pull/4811 * ci(scripts): Add a script to generate version list by @Xuanwo in https://github.com/apache/opendal/pull/4827 ### Chore * chore(ci): disable aliyun_drive for bindings test by @suyanhanx in https://github.com/apache/opendal/pull/4770 * chore(unftp-sbe): remove Cargo.lock by @George-Miao in https://github.com/apache/opendal/pull/4805 ## [v0.47.1] - 2024-06-18 ### Added * feat(core): sets default chunk_size and sends buffer > chunk_size directly by @evenyag in https://github.com/apache/opendal/pull/4710 * feat(services): add optional access_token for AliyunDrive by @yuchanns in https://github.com/apache/opendal/pull/4740 * feat(unftp-sbe): Add integration for unftp-sbe by @George-Miao in https://github.com/apache/opendal/pull/4753 ### Changed * refactor(ofs): Split fuse3 impl into fuse3_opendal by @Xuanwo in https://github.com/apache/opendal/pull/4721 * refactor(ovfs): Split ovfs impl into virtiofs_opendal by @zjregee in https://github.com/apache/opendal/pull/4723 * refactor(*): tiny refactor to the Error type by @waynexia in https://github.com/apache/opendal/pull/4737 * refactor(aliyun-drive): rewrite writer part by @yuchanns in https://github.com/apache/opendal/pull/4744 * refactor(object_store): Polish implementation details of object_store by @Xuanwo in https://github.com/apache/opendal/pull/4749 * refactor(dav-server): Polish dav-server integration details by @Xuanwo in https://github.com/apache/opendal/pull/4751 * refactor(core): Remove unused `size` for `RangeWrite`. by @reswqa in https://github.com/apache/opendal/pull/4755 ### Fixed * fix(s3): parse MultipartUploadResponse to check error in body by @waynexia in https://github.com/apache/opendal/pull/4735 * fix(services/aliyun-drive): unable to list `/` by @yuchanns in https://github.com/apache/opendal/pull/4754 ### Docs * docs: keep docs updated and tidy by @tisonkun in https://github.com/apache/opendal/pull/4709 * docs: fixup broken links by @tisonkun in https://github.com/apache/opendal/pull/4711 * docs(website): update release/verify docs by @suyanhanx in https://github.com/apache/opendal/pull/4714 * docs: Update release.md link correspondingly by @tisonkun in https://github.com/apache/opendal/pull/4717 * docs: update readme for fuse3_opendal & virtiofs_opendal by @zjregee in https://github.com/apache/opendal/pull/4730 * docs: Polish README and links to docs by @Xuanwo in https://github.com/apache/opendal/pull/4741 * docs: Enhance maintainability of the service section by @Xuanwo in https://github.com/apache/opendal/pull/4742 * docs: Polish opendal rust core README by @Xuanwo in https://github.com/apache/opendal/pull/4745 * docs: Refactor rust core examples by @Xuanwo in https://github.com/apache/opendal/pull/4757 ### CI * ci: verify build website on site content changes by @tisonkun in https://github.com/apache/opendal/pull/4712 * ci: Fix cert for redis and add docs for key maintenance by @Xuanwo in https://github.com/apache/opendal/pull/4718 * ci(nodejs): Disable services-all on windows by @Xuanwo in https://github.com/apache/opendal/pull/4762 ### Chore * chore: use more portable binutils by @tisonkun in https://github.com/apache/opendal/pull/4713 * chore(deps): bump clap from 4.5.6 to 4.5.7 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/4728 * chore(deps): bump url from 2.5.0 to 2.5.1 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/4729 * chore(binding/python): Upgrade pyo3 to 0.21 by @reswqa in https://github.com/apache/opendal/pull/4734 * chore: Make 1.79 clippy happy by @Xuanwo in https://github.com/apache/opendal/pull/4731 * chore(docs): Add new line in lone services by @Xuanwo in https://github.com/apache/opendal/pull/4743 * chore: Bump versions to prepare v0.47.1 release by @Xuanwo in https://github.com/apache/opendal/pull/4759 ## [v0.47.0] - 2024-06-07 ### Added * feat(core/types): change oio::BlockingReader to `Arc` by @hoslo in https://github.com/apache/opendal/pull/4577 * fix: format_object_meta should not require metakeys that don't exist by @rebasedming in https://github.com/apache/opendal/pull/4582 * feat: add checksums to MultiPartComplete by @JWackerbauer in https://github.com/apache/opendal/pull/4580 * feat(doc): update object_store_opendal README by @hanxuanliang in https://github.com/apache/opendal/pull/4606 * feat(services/aliyun-drive): support AliyunDrive by @yuchanns in https://github.com/apache/opendal/pull/4585 * feat(bindings/python): Update type annotations by @3ok in https://github.com/apache/opendal/pull/4630 * feat: implement OperatorInputStream and OperatorOutputStream by @tisonkun in https://github.com/apache/opendal/pull/4626 * feat(bench): add buffer benchmark by @zjregee in https://github.com/apache/opendal/pull/4603 * feat: Add Executor struct and Execute trait by @Xuanwo in https://github.com/apache/opendal/pull/4648 * feat: Add executor in OpXxx and Operator by @Xuanwo in https://github.com/apache/opendal/pull/4649 * feat: Implement and refactor concurrent tasks for multipart write by @Xuanwo in https://github.com/apache/opendal/pull/4653 * feat(core/types): blocking remove_all for object storage based services by @TennyZhuang in https://github.com/apache/opendal/pull/4665 * feat(core): Streaming reading while chunk is not set by @Xuanwo in https://github.com/apache/opendal/pull/4658 * feat(core): Add more context in error context by @Xuanwo in https://github.com/apache/opendal/pull/4673 * feat: init ovfs by @zjregee in https://github.com/apache/opendal/pull/4652 * feat: Implement retry for streaming based read by @Xuanwo in https://github.com/apache/opendal/pull/4683 * feat(core): Implement TimeoutLayer for concurrent tasks by @Xuanwo in https://github.com/apache/opendal/pull/4688 * feat(core): Add reader size check in complete reader by @Xuanwo in https://github.com/apache/opendal/pull/4690 * feat(core): Azblob supports azure workload identity by @Xuanwo in https://github.com/apache/opendal/pull/4705 ### Changed * refactor(core): Align naming for `AccessorDyn` by @morristai in https://github.com/apache/opendal/pull/4574 * refactor(core): core doesn't expose invalid input error anymore by @Xuanwo in https://github.com/apache/opendal/pull/4632 * refactor(core): Return unexpected error while content incomplete happen by @Xuanwo in https://github.com/apache/opendal/pull/4633 * refactor(core): Change Read's behavior to ensure it reads the exact size of data by @Xuanwo in https://github.com/apache/opendal/pull/4634 * refactor(bin/ofs): Fuse API by @ho-229 in https://github.com/apache/opendal/pull/4637 * refactor(binding/java)!: rename blocking and async operator by @tisonkun in https://github.com/apache/opendal/pull/4641 * refactor(core): Use concurrent tasks to refactor block write by @Xuanwo in https://github.com/apache/opendal/pull/4692 * refactor(core): Migrate RangeWrite to ConcurrentTasks by @Xuanwo in https://github.com/apache/opendal/pull/4696 ### Fixed * fix(devcontainer/post_create.sh): change pnpm@stable to pnpm@latest by @GG2002 in https://github.com/apache/opendal/pull/4584 * fix(bin/ofs): privileged mount crashes when external umount by @ho-229 in https://github.com/apache/opendal/pull/4586 * fix(bin/ofs): ofs read only mount by @ho-229 in https://github.com/apache/opendal/pull/4602 * fix(raw): Allow retrying request while decoding response failed by @Xuanwo in https://github.com/apache/opendal/pull/4612 * fix(core): return None if metadata unavailable by @NKID00 in https://github.com/apache/opendal/pull/4613 * fix(bindings/python): Use abi3 and increase MSPV to 3.11 by @Xuanwo in https://github.com/apache/opendal/pull/4623 * fix: Fetch the content length while end_bound is unknown by @Xuanwo in https://github.com/apache/opendal/pull/4631 * fix: ofs write behavior by @ho-229 in https://github.com/apache/opendal/pull/4617 * fix(core/types): remove_all not work under object-store backend by @TennyZhuang in https://github.com/apache/opendal/pull/4659 * fix(ofs): Close file during flush by @Xuanwo in https://github.com/apache/opendal/pull/4680 * fix(core): RetryLayer could panic when other threads raises panic by @Xuanwo in https://github.com/apache/opendal/pull/4685 * fix(core/prometheus): Fix metrics from prometheus not correct for reader by @Xuanwo in https://github.com/apache/opendal/pull/4691 * fix(core/oio): Make ConcurrentTasks cancel safe by only pop after ready by @Xuanwo in https://github.com/apache/opendal/pull/4707 ### Docs * docs: fix Operator::writer doc comment by @mnpw in https://github.com/apache/opendal/pull/4605 * doc: explain GCS authentication options by @jokester in https://github.com/apache/opendal/pull/4671 * docs: Fix all broken links by @Xuanwo in https://github.com/apache/opendal/pull/4694 * docs: Add upgrade note for v0.47 by @Xuanwo in https://github.com/apache/opendal/pull/4698 * docs: Add panics declare for TimeoutLayer and RetryLayer by @Xuanwo in https://github.com/apache/opendal/pull/4702 ### CI * ci: upgrade typos to 1.21.0 and ignore changelog by @hezhizhen in https://github.com/apache/opendal/pull/4601 * ci: Disable jfrog webdav tests for it's keeping failed by @Xuanwo in https://github.com/apache/opendal/pull/4607 * ci: use official typos github action by @shoothzj in https://github.com/apache/opendal/pull/4635 * build(deps): upgrade crc32c to 0.6.6 for nightly toolchain by @tisonkun in https://github.com/apache/opendal/pull/4650 ### Chore * chore: fixup release docs and scripts by @tisonkun in https://github.com/apache/opendal/pull/4571 * chore: Make rust 1.78 happy by @Xuanwo in https://github.com/apache/opendal/pull/4572 * chore: fixup items identified in releases by @tisonkun in https://github.com/apache/opendal/pull/4578 * chore(deps): bump peaceiris/actions-gh-pages from 3.9.2 to 4.0.0 by @dependabot in https://github.com/apache/opendal/pull/4561 * chore: Contribute ParadeDB by @philippemnoel in https://github.com/apache/opendal/pull/4587 * chore(deps): bump rusqlite from 0.29.0 to 0.31.0 in /core by @dependabot in https://github.com/apache/opendal/pull/4556 * chore(deps): Bump object_store to 0.10 by @TCeason in https://github.com/apache/opendal/pull/4590 * chore: remove outdated scan op in all docs.md by @GG2002 in https://github.com/apache/opendal/pull/4600 * chore: tidy services in project file by @suyanhanx in https://github.com/apache/opendal/pull/4621 * chore(deps): make crc32c optional under services-s3 by @xxchan in https://github.com/apache/opendal/pull/4643 * chore(core): Fix unit tests by @Xuanwo in https://github.com/apache/opendal/pull/4684 * chore(core): Add unit and bench tests for concurrent tasks by @Xuanwo in https://github.com/apache/opendal/pull/4695 * chore: bump version to 0.47.0 by @tisonkun in https://github.com/apache/opendal/pull/4701 * chore: Update changelogs for v0.47 by @Xuanwo in https://github.com/apache/opendal/pull/4706 ## [v0.46.0] - 2024-05-02 ### Added * feat(services/github): add github contents support by @hoslo in https://github.com/apache/opendal/pull/4281 * feat: Allow selecting webpki for reqwest by @arlyon in https://github.com/apache/opendal/pull/4303 * feat(services/swift): add support for storage_url configuration in swift service by @zjregee in https://github.com/apache/opendal/pull/4302 * feat(services/swift): add ceph test setup for swift by @zjregee in https://github.com/apache/opendal/pull/4307 * docs(website): add local content search based on lunr plugin by @m1911star in https://github.com/apache/opendal/pull/4348 * feat(services/sled): add SledConfig by @yufan022 in https://github.com/apache/opendal/pull/4351 * feat : Implementing config for part of services by @AnuRage-git in https://github.com/apache/opendal/pull/4344 * feat(bindings/java): explicit async runtime by @tisonkun in https://github.com/apache/opendal/pull/4376 * feat(services/surrealdb): support surrealdb service by @yufan022 in https://github.com/apache/opendal/pull/4375 * feat(bindings/java): avoid double dispose NativeObject by @tisonkun in https://github.com/apache/opendal/pull/4377 * feat : Implement config for services/foundationdb by @AnuRage-git in https://github.com/apache/opendal/pull/4355 * feat: add ofs ctrl-c exit unmount hook by @oowl in https://github.com/apache/opendal/pull/4393 * feat: Implement RFC-4382 Range Based Read by @Xuanwo in https://github.com/apache/opendal/pull/4381 * feat(ofs): rename2 lseek copy_file_range getattr API support by @oowl in https://github.com/apache/opendal/pull/4395 * feat(services/github): make access_token optional by @hoslo in https://github.com/apache/opendal/pull/4404 * feat(core/oio): Add readable buf by @Xuanwo in https://github.com/apache/opendal/pull/4407 * feat(ofs): add freebsd OS support by @oowl in https://github.com/apache/opendal/pull/4403 * feat(core/raw/oio): Add Writable Buf by @Xuanwo in https://github.com/apache/opendal/pull/4410 * feat(bin/ofs): Add behavior test for ofs by @ho-229 in https://github.com/apache/opendal/pull/4373 * feat(core/raw/buf): Reduce one allocation by `Arc::from_iter` by @Xuanwo in https://github.com/apache/opendal/pull/4440 * feat: ?Send async trait for HttpBackend when the target is wasm32 by @waynexia in https://github.com/apache/opendal/pull/4444 * feat: add `HttpClient::with()` constructor by @waynexia in https://github.com/apache/opendal/pull/4447 * feat: Move Buffer as public API by @Xuanwo in https://github.com/apache/opendal/pull/4450 * feat: Optimize buffer implementation and add stream support by @Xuanwo in https://github.com/apache/opendal/pull/4458 * feat(core): Implement FromIterator for Buffer by @Xuanwo in https://github.com/apache/opendal/pull/4459 * feat(services/ftp): Support multiple write by @xxxuuu in https://github.com/apache/opendal/pull/4425 * feat(raw/oio): block write change to buffer by @hoslo in https://github.com/apache/opendal/pull/4466 * feat(core): Implement read and read_into for Reader by @Xuanwo in https://github.com/apache/opendal/pull/4467 * feat(core): Implement into_stream for Reader by @Xuanwo in https://github.com/apache/opendal/pull/4473 * feat(core): Tune buffer operations based on benchmark results by @Xuanwo in https://github.com/apache/opendal/pull/4468 * feat(raw/oio): Use `Buffer` as cache in `RangeWrite` by @reswqa in https://github.com/apache/opendal/pull/4476 * feat(raw/oio): Use `Buffer` as cache in `OneshotWrite` by @reswqa in https://github.com/apache/opendal/pull/4477 * feat: add list/copy/rename for dropbox by @zjregee in https://github.com/apache/opendal/pull/4424 * feat(core): Implement write and write_from for Writer by @zjregee in https://github.com/apache/opendal/pull/4482 * feat(core): Add auto ranged read and concurrent read support by @Xuanwo in https://github.com/apache/opendal/pull/4486 * feat(core): Implement fetch for Reader by @Xuanwo in https://github.com/apache/opendal/pull/4488 * feat(core): Allow concurrent reading on bytes stream by @Xuanwo in https://github.com/apache/opendal/pull/4499 * feat: provide send-wrapper to contidionally implement Send for operators by @waynexia in https://github.com/apache/opendal/pull/4443 * feat(bin/ofs): privileged mount by @ho-229 in https://github.com/apache/opendal/pull/4507 * feat(services/compfs): init compfs by @George-Miao in https://github.com/apache/opendal/pull/4519 * feat(raw/oio): Add PooledBuf to allow reuse buffer by @Xuanwo in https://github.com/apache/opendal/pull/4522 * feat(services/fs): Add PooledBuf in fs to allow reusing memory by @Xuanwo in https://github.com/apache/opendal/pull/4525 * feat(core): add access methods for `Buffer` by @George-Miao in https://github.com/apache/opendal/pull/4530 * feat(core): implement `IoBuf` for `Buffer` by @George-Miao in https://github.com/apache/opendal/pull/4532 * feat(services/compfs): compio runtime and compfs structure by @George-Miao in https://github.com/apache/opendal/pull/4534 * feat(core): change Result to default error by @George-Miao in https://github.com/apache/opendal/pull/4535 * feat(services/github): support list recursive by @hoslo in https://github.com/apache/opendal/pull/4423 * feat: Add crc32c checksums to S3 Service by @JWackerbauer in https://github.com/apache/opendal/pull/4533 * feat: Add into_bytes_sink for Writer by @Xuanwo in https://github.com/apache/opendal/pull/4541 ### Changed * refactor(core/raw): Migrate `oio::Read` to async in trait by @Xuanwo in https://github.com/apache/opendal/pull/4336 * refactor(core/raw): Align oio::BlockingRead API with oio::Read by @Xuanwo in https://github.com/apache/opendal/pull/4349 * refactor(core/oio): Migrate `oio::List` to async fn in trait by @Xuanwo in https://github.com/apache/opendal/pull/4352 * refactor(core/raw): Migrate oio::Write from WriteBuf to Bytes by @Xuanwo in https://github.com/apache/opendal/pull/4356 * refactor(core/raw): Migrate `oio::Write` to async in trait by @Xuanwo in https://github.com/apache/opendal/pull/4358 * refactor(bindings/python): Change the return type of `File::read` to `PyResult<&PyBytes>` by @reswqa in https://github.com/apache/opendal/pull/4360 * refactor(core/raw): Cleanup not used `oio::WriteBuf` and `oio::ChunkedBytes` after refactor by @Xuanwo in https://github.com/apache/opendal/pull/4361 * refactor: Remove reqwest related features by @Xuanwo in https://github.com/apache/opendal/pull/4365 * refactor(bin/ofs): make `ofs` API public by @ho-229 in https://github.com/apache/opendal/pull/4387 * refactor: Impl into_futures_io_async_write for Writer by @Xuanwo in https://github.com/apache/opendal/pull/4406 * refactor(core/oio): Use ReadableBuf to remove extra clone during write by @Xuanwo in https://github.com/apache/opendal/pull/4408 * refactor(core/raw/oio): Mark oio::Write::write as an unsafe function by @Xuanwo in https://github.com/apache/opendal/pull/4413 * refactor(core/raw/oio): Use oio buffer in write by @Xuanwo in https://github.com/apache/opendal/pull/4436 * refactor(core/raw): oio::Write is safe operation now by @Xuanwo in https://github.com/apache/opendal/pull/4438 * refactor(core): Use Buffer in http request by @Xuanwo in https://github.com/apache/opendal/pull/4460 * refactor(core): Polish FuturesBytesStream by avoiding extra copy by @Xuanwo in https://github.com/apache/opendal/pull/4474 * refactor: Use Buffer as cache in MultipartWrite by @tisonkun in https://github.com/apache/opendal/pull/4493 * refactor: kv::adapter should use Buffer (read part) by @tisonkun in https://github.com/apache/opendal/pull/4494 * refactor: typed_kv::adapter should use Buffer by @tisonkun in https://github.com/apache/opendal/pull/4497 * refactor: kv::adapter should use Buffer (write part) by @tisonkun in https://github.com/apache/opendal/pull/4496 * refactor: Migrate FuturesAsyncReader to stream based by @Xuanwo in https://github.com/apache/opendal/pull/4508 * refactor(services/fs): Extract FsCore from FsBackend by @Xuanwo in https://github.com/apache/opendal/pull/4523 * refactor(core): Migrate `Accessor` to async fn in trait by @George-Miao in https://github.com/apache/opendal/pull/4562 * refactor(core): Align naming for `Accessor` by @George-Miao in https://github.com/apache/opendal/pull/4564 ### Fixed * fix(bin/ofs): crashes when fh=None by @ho-229 in https://github.com/apache/opendal/pull/4297 * fix(services/hdfs): fix poll_close when retry by @hoslo in https://github.com/apache/opendal/pull/4309 * fix: fix main CI by @xxchan in https://github.com/apache/opendal/pull/4319 * fix: fix ghac CI by @xxchan in https://github.com/apache/opendal/pull/4320 * fix(services/dropbox): fix dropbox batch test panic in ci by @zjregee in https://github.com/apache/opendal/pull/4329 * fix(services/hdfs-native): remove unsupported capabilities by @jihuayu in https://github.com/apache/opendal/pull/4333 * fix(bin/ofs): build failed when target_os != linux by @ho-229 in https://github.com/apache/opendal/pull/4334 * fix(bin/ofs): only memory backend available by @ho-229 in https://github.com/apache/opendal/pull/4353 * fix(bindings/python): Fix the semantic of `size` argument for python `File::read` by @reswqa in https://github.com/apache/opendal/pull/4359 * fix(bindings/python): `File::write` should return the written bytes by @reswqa in https://github.com/apache/opendal/pull/4367 * fix(services/s3): omit default ports by @yufan022 in https://github.com/apache/opendal/pull/4379 * fix(core/services/gdrive): Fix gdrive test failed for refresh token by @Xuanwo in https://github.com/apache/opendal/pull/4435 * fix(core/services/cos): Don't allow empty credential by @Xuanwo in https://github.com/apache/opendal/pull/4457 * fix(oay): support `WebdavFile` continuous reading and writing by @G-XD in https://github.com/apache/opendal/pull/4374 * fix(integrations/webdav): Fix read file API changes by @Xuanwo in https://github.com/apache/opendal/pull/4462 * fix(s3): don't delete bucket by @sameer in https://github.com/apache/opendal/pull/4430 * fix(core): Buffer offset misuse by @George-Miao in https://github.com/apache/opendal/pull/4481 * fix(core): Read chunk should respect users input by @Xuanwo in https://github.com/apache/opendal/pull/4487 * fix(services/cos): fix mdx by @hoslo in https://github.com/apache/opendal/pull/4510 * fix: minor doc issue by @George-Miao in https://github.com/apache/opendal/pull/4517 * fix(website): community sync calendar iframe by @suyanhanx in https://github.com/apache/opendal/pull/4549 * fix: community sync calendar iframe load failed by @suyanhanx in https://github.com/apache/opendal/pull/4550 ### Docs * docs: Add gsoc proposal guide by @Xuanwo in https://github.com/apache/opendal/pull/4287 * docs: Add blog for apache opendal participates in gsoc 2024 by @Xuanwo in https://github.com/apache/opendal/pull/4288 * docs: Fix and improve docs for presign operations by @Xuanwo in https://github.com/apache/opendal/pull/4294 * docs: Polish the docs for writer by @Xuanwo in https://github.com/apache/opendal/pull/4296 * docs: Add redirect for discord and maillist by @Xuanwo in https://github.com/apache/opendal/pull/4312 * docs: update swift docs by @zjregee in https://github.com/apache/opendal/pull/4327 * docs: publish docs for object-store-opendal by @zjregee in https://github.com/apache/opendal/pull/4328 * docs: fix redirect error in object-store-opendal by @zjregee in https://github.com/apache/opendal/pull/4332 * docs(website): fix grammar and spelling by @jbampton in https://github.com/apache/opendal/pull/4340 * docs: fix nodejs binding docs footer copyright by @m1911star in https://github.com/apache/opendal/pull/4346 * docs(bin/ofs): update README by @ho-229 in https://github.com/apache/opendal/pull/4354 * docs(services/gcs): update gcs credentials description by @zjregee in https://github.com/apache/opendal/pull/4362 * docs(bindings/python): ipynb examples for polars and pandas by @reswqa in https://github.com/apache/opendal/pull/4368 * docs(core): correct the doc for icloud and memcached by @Kilerd in https://github.com/apache/opendal/pull/4422 * docs: polish release doc for vote result by @suyanhanx in https://github.com/apache/opendal/pull/4429 * docs: Update links to o.a.o/discord by @Xuanwo in https://github.com/apache/opendal/pull/4442 * docs: add layers example by @zjregee in https://github.com/apache/opendal/pull/4449 * docs: publish docs for dav-server-opendalfs by @zjregee in https://github.com/apache/opendal/pull/4503 * docs: Add docs for read_with and reader_with by @Xuanwo in https://github.com/apache/opendal/pull/4516 * docs: Add upgrade doc for rust core 0.46 by @Xuanwo in https://github.com/apache/opendal/pull/4543 * docs: update the outdated download link by @suyanhanx in https://github.com/apache/opendal/pull/4546 ### CI * ci(binding/java): Don't create too many files in CI by @Xuanwo in https://github.com/apache/opendal/pull/4289 * ci: Address Node.js 16 actions are deprecated by @Xuanwo in https://github.com/apache/opendal/pull/4293 * ci: Disable vsftpd test for it's keeping failure by @Xuanwo in https://github.com/apache/opendal/pull/4295 * build: remove `service-*` from default features by @xxchan in https://github.com/apache/opendal/pull/4311 * ci: upgrade typos in ci by @Young-Flash in https://github.com/apache/opendal/pull/4322 * ci: Disable R2 until we figure what happened by @Xuanwo in https://github.com/apache/opendal/pull/4323 * ci(s3/minio): Disable IMDSv2 for mini anonymous tests by @Xuanwo in https://github.com/apache/opendal/pull/4326 * ci: Fix unit tests missing protoc by @Xuanwo in https://github.com/apache/opendal/pull/4369 * ci: Fix foundationdb not setup for unit test by @Xuanwo in https://github.com/apache/opendal/pull/4370 * ci: bump license header formatter by @tisonkun in https://github.com/apache/opendal/pull/4390 * ci: fix dragonflydb docker-compose healthcheck broken by @oowl in https://github.com/apache/opendal/pull/4431 * ci: fix planner for bin ofs by @ho-229 in https://github.com/apache/opendal/pull/4448 * ci: Revert oay changes to fix CI by @Xuanwo in https://github.com/apache/opendal/pull/4463 * ci: Fix CI for all bindings by @Xuanwo in https://github.com/apache/opendal/pull/4469 * ci: Disable dropbox test until #4484 fixed by @Xuanwo in https://github.com/apache/opendal/pull/4485 * ci: Fix build of python binding for chunk write changes by @Xuanwo in https://github.com/apache/opendal/pull/4529 * build(core): bump compio version to v0.10.0 by @George-Miao in https://github.com/apache/opendal/pull/4531 * ci: Disable tikv for pingcap's mirror is unstable by @Xuanwo in https://github.com/apache/opendal/pull/4538 * build: staging website by @tisonkun in https://github.com/apache/opendal/pull/4565 * build: fixup follow asfyaml rules by @tisonkun in https://github.com/apache/opendal/pull/4566 ### Chore * chore(deps): bump tempfile from 3.9.0 to 3.10.1 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/4298 * chore(deps): bump wasm-bindgen-test from 0.3.40 to 0.3.41 in /core by @dependabot in https://github.com/apache/opendal/pull/4299 * chore(deps): bump log from 0.4.20 to 0.4.21 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/4301 * chore(services/redis): Fix docs & comments typos by @AJIOB in https://github.com/apache/opendal/pull/4306 * chore(editorconfig): add `yaml` file type by @jbampton in https://github.com/apache/opendal/pull/4339 * chore(editorconfig): add `rdf` file type as `indent_size = 2` by @jbampton in https://github.com/apache/opendal/pull/4341 * chore(editorconfig): order entries and add `indent_style = tab` for Go by @jbampton in https://github.com/apache/opendal/pull/4342 * chore(labeler): order the GitHub labeler labels by @jbampton in https://github.com/apache/opendal/pull/4343 * chore: Cleanup of oio::Read, docs, comments, naming by @Xuanwo in https://github.com/apache/opendal/pull/4345 * chore: Remove not exist read operations by @Xuanwo in https://github.com/apache/opendal/pull/4412 * chore(deps): bump toml from 0.8.10 to 0.8.12 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/4418 * chore(deps): bump toml from 0.8.10 to 0.8.12 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/4420 * chore(deps): bump tokio from 1.36.0 to 1.37.0 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/4419 * chore(deps): bump prometheus-client from 0.22.1 to 0.22.2 in /core by @dependabot in https://github.com/apache/opendal/pull/4417 * chore: update copyright year to 2024 in NOTICE by @shoothzj in https://github.com/apache/opendal/pull/4433 * chore: Bump bytes to 1.6 to avoid compileing error by @Xuanwo in https://github.com/apache/opendal/pull/4472 * chore: Add output types in OperatorFutures by @Xuanwo in https://github.com/apache/opendal/pull/4475 * chore(core): Use reader's chunk size instead by @Xuanwo in https://github.com/apache/opendal/pull/4489 * chore: sort tomls by taplo by @tisonkun in https://github.com/apache/opendal/pull/4491 * chore(core): Align Reader and Writer's API design by @Xuanwo in https://github.com/apache/opendal/pull/4498 * chore: Add docs and tests for reader related types by @Xuanwo in https://github.com/apache/opendal/pull/4513 * chore: Reorganize the blocking reader layout by @Xuanwo in https://github.com/apache/opendal/pull/4514 * chore: Align with chunk instead of confusing buffer by @Xuanwo in https://github.com/apache/opendal/pull/4528 * chore: Refactor Write and BlockingWrite API by @Xuanwo in https://github.com/apache/opendal/pull/4540 * chore(core): Change std result to opendal result in core by @tannal in https://github.com/apache/opendal/pull/4542 * chore: fixup download links by @tisonkun in https://github.com/apache/opendal/pull/4547 * chore(deps): bump clap from 4.5.1 to 4.5.4 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/4557 * chore(deps): bump anyhow from 1.0.80 to 1.0.82 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/4559 * chore(deps): bump libc from 0.2.153 to 0.2.154 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/4558 ## [v0.45.1] - 2024-02-22 ### Added * feat(services/vercel_blob): support vercel blob by @hoslo in https://github.com/apache/opendal/pull/4103 * feat(bindings/python): add ruff as linter by @asukaminato0721 in https://github.com/apache/opendal/pull/4135 * feat(services/hdfs-native): Add capabilities for hdfs-native service by @jihuayu in https://github.com/apache/opendal/pull/4174 * feat(services/sqlite): Add list capability supported for sqlite by @jihuayu in https://github.com/apache/opendal/pull/4180 * feat(services/azblob): support multi write for azblob by @wcy-fdu in https://github.com/apache/opendal/pull/4181 * feat(release): Implement releasing OpenDAL components separately by @Xuanwo in https://github.com/apache/opendal/pull/4196 * feat: object store adapter based on v0.9 by @waynexia in https://github.com/apache/opendal/pull/4233 * feat(bin/ofs): implement fuse for linux by @ho-229 in https://github.com/apache/opendal/pull/4179 * feat(services/memcached): change to binary protocol by @hoslo in https://github.com/apache/opendal/pull/4252 * feat(services/memcached): setup auth test for memcached by @hoslo in https://github.com/apache/opendal/pull/4259 * feat(services/yandex_disk): setup test for yandex disk by @hoslo in https://github.com/apache/opendal/pull/4264 * feat: add ci support for ceph_rados by @ZhengLin-Li in https://github.com/apache/opendal/pull/4191 * feat: Implement Config for part of services by @Xuanwo in https://github.com/apache/opendal/pull/4277 * feat: add jfrog test setup for webdav by @zjregee in https://github.com/apache/opendal/pull/4265 ### Changed * refactor(bindings/python): simplify async writer aexit by @suyanhanx in https://github.com/apache/opendal/pull/4128 * refactor(service/d1): Add D1Config by @jihuayu in https://github.com/apache/opendal/pull/4129 * refactor: Rewrite webdav to improve code quality by @Xuanwo in https://github.com/apache/opendal/pull/4280 ### Fixed * fix: Azdls returns 403 while continuation contains `=` by @Xuanwo in https://github.com/apache/opendal/pull/4105 * fix(bindings/python): missed to call close for the file internally by @zzl221000 in https://github.com/apache/opendal/pull/4122 * fix(bindings/python): sync writer exit close raise error by @suyanhanx in https://github.com/apache/opendal/pull/4127 * fix(services/chainsafe): fix 423 http status by @hoslo in https://github.com/apache/opendal/pull/4148 * fix(services/webdav): Add possibility to answer without response if file isn't exist by @AJIOB in https://github.com/apache/opendal/pull/4170 * fix(services/webdav): Recreate root directory if need by @AJIOB in https://github.com/apache/opendal/pull/4173 * fix(services/webdav): remove base_dir component by @hoslo in https://github.com/apache/opendal/pull/4231 * fix(core): Poll TimeoutLayer::sleep once to make sure timer registered by @Xuanwo in https://github.com/apache/opendal/pull/4230 * fix(services/webdav): Fix endpoint suffix not handled by @Xuanwo in https://github.com/apache/opendal/pull/4257 * fix(services/webdav): Fix possible error with value loosing from config by @AJIOB in https://github.com/apache/opendal/pull/4262 ### Docs * docs: add request for add secrets of services by @suyanhanx in https://github.com/apache/opendal/pull/4104 * docs(website): announce release v0.45.0 to news by @morristai in https://github.com/apache/opendal/pull/4152 * docs(services/gdrive): Update Google Drive capabilities list docs by @jihuayu in https://github.com/apache/opendal/pull/4158 * docs: Fix docs build by @Xuanwo in https://github.com/apache/opendal/pull/4162 * docs: add docs for Ceph Rados Gateway S3 by @ZhengLin-Li in https://github.com/apache/opendal/pull/4190 * docs: Fix typo in `core/src/services/http/docs.md` by @jbampton in https://github.com/apache/opendal/pull/4226 * docs: Fix spelling in Rust files by @jbampton in https://github.com/apache/opendal/pull/4227 * docs: fix typo in `website/README.md` by @jbampton in https://github.com/apache/opendal/pull/4228 * docs: fix spelling by @jbampton in https://github.com/apache/opendal/pull/4229 * docs: fix spelling; change `github` to `GitHub` by @jbampton in https://github.com/apache/opendal/pull/4232 * docs: fix typo by @jbampton in https://github.com/apache/opendal/pull/4234 * docs: fix typo in `bindings/c/CONTRIBUTING.md` by @jbampton in https://github.com/apache/opendal/pull/4235 * docs: fix spelling in code comments by @jbampton in https://github.com/apache/opendal/pull/4236 * docs: fix spelling in `CONTRIBUTING` by @jbampton in https://github.com/apache/opendal/pull/4237 * docs: fix Markdown link in `bindings/README.md` by @jbampton in https://github.com/apache/opendal/pull/4238 * docs: fix links and spelling in Markdown by @jbampton in https://github.com/apache/opendal/pull/4239 * docs: fix grammar and spelling in Markdown in `examples/rust` by @jbampton in https://github.com/apache/opendal/pull/4241 * docs: remove unneeded duplicate words from Rust files by @jbampton in https://github.com/apache/opendal/pull/4243 * docs: fix grammar and spelling in Markdown by @jbampton in https://github.com/apache/opendal/pull/4245 * docs: Add architectural.png to website by @Xuanwo in https://github.com/apache/opendal/pull/4261 * docs: Re-org project README by @Xuanwo in https://github.com/apache/opendal/pull/4260 * docs: order the README `Who is using OpenDAL` list by @jbampton in https://github.com/apache/opendal/pull/4263 ### CI * ci: Use old version of seafile mc instead by @Xuanwo in https://github.com/apache/opendal/pull/4107 * ci: Refactor workflows layout by @Xuanwo in https://github.com/apache/opendal/pull/4139 * ci: Migrate hdfs default setup by @Xuanwo in https://github.com/apache/opendal/pull/4140 * ci: Refactor check.sh into check.py to get ready for multi components release by @Xuanwo in https://github.com/apache/opendal/pull/4159 * ci: Add test case for hdfs over gcs bucket by @ArmandoZ in https://github.com/apache/opendal/pull/4145 * ci: Add hdfs test case for s3 by @ArmandoZ in https://github.com/apache/opendal/pull/4184 * ci: Add hdfs test case for azurite by @ArmandoZ in https://github.com/apache/opendal/pull/4185 * ci: Add support for releasing all rust packages by @Xuanwo in https://github.com/apache/opendal/pull/4200 * ci: Fix dependabot not update by @Xuanwo in https://github.com/apache/opendal/pull/4202 * ci: reduce the open pull request limits to 1 by @jbampton in https://github.com/apache/opendal/pull/4225 * ci: Remove version suffix from package versions by @Xuanwo in https://github.com/apache/opendal/pull/4254 * ci: Fix fuzz test for minio s3 name change by @Xuanwo in https://github.com/apache/opendal/pull/4266 * ci: Mark python 3.13 is not supported by @Xuanwo in https://github.com/apache/opendal/pull/4269 * ci: Disable yandex disk test for too slow by @Xuanwo in https://github.com/apache/opendal/pull/4274 * ci: Split python CI into release and checks by @Xuanwo in https://github.com/apache/opendal/pull/4275 * ci(release): Make sure LICENSE and NOTICE files are included by @Xuanwo in https://github.com/apache/opendal/pull/4283 * ci(release): Refactor and merge the check.py into verify.py by @Xuanwo in https://github.com/apache/opendal/pull/4284 ### Chore * chore(deps): bump actions/cache from 3 to 4 by @dependabot in https://github.com/apache/opendal/pull/4118 * chore(deps): bump toml from 0.8.8 to 0.8.9 by @dependabot in https://github.com/apache/opendal/pull/4109 * chore(deps): bump dav-server from 0.5.7 to 0.5.8 by @dependabot in https://github.com/apache/opendal/pull/4111 * chore(deps): bump assert_cmd from 2.0.12 to 2.0.13 by @dependabot in https://github.com/apache/opendal/pull/4112 * chore(deps): bump actions/setup-dotnet from 3 to 4 by @dependabot in https://github.com/apache/opendal/pull/4115 * chore(deps): bump mongodb from 2.7.1 to 2.8.0 by @dependabot in https://github.com/apache/opendal/pull/4110 * chore(deps): bump quick-xml from 0.30.0 to 0.31.0 by @dependabot in https://github.com/apache/opendal/pull/4113 * chore: Make every components separate, remove workspace by @Xuanwo in https://github.com/apache/opendal/pull/4134 * chore: Fix build of core by @Xuanwo in https://github.com/apache/opendal/pull/4137 * chore: Fix workflow links for opendal by @Xuanwo in https://github.com/apache/opendal/pull/4147 * chore(website): Bump download link for 0.45.0 release by @morristai in https://github.com/apache/opendal/pull/4149 * chore: Fix name DtraceLayerWrapper by @jayvdb in https://github.com/apache/opendal/pull/4165 * chore: Align core version by @Xuanwo in https://github.com/apache/opendal/pull/4197 * chore: update benchmark doc by @wcy-fdu in https://github.com/apache/opendal/pull/4201 * chore(deps): bump clap from 4.4.18 to 4.5.1 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/4221 * chore(deps): bump serde from 1.0.196 to 1.0.197 in /bin/oay by @dependabot in https://github.com/apache/opendal/pull/4214 * chore(deps): bump anyhow from 1.0.79 to 1.0.80 in /bin/ofs by @dependabot in https://github.com/apache/opendal/pull/4209 * chore(deps): bump anyhow from 1.0.79 to 1.0.80 in /bin/oli by @dependabot in https://github.com/apache/opendal/pull/4216 * chore(deps): bump cacache from 12.0.0 to 13.0.0 in /core by @dependabot in https://github.com/apache/opendal/pull/4215 ## [v0.45.0] - 2024-01-29 ### Added * feat(ofs): introduce ofs execute bin by @oowl in https://github.com/apache/opendal/pull/4033 * feat: add a missing news by @WenyXu in https://github.com/apache/opendal/pull/4056 * feat(services/koofr): set test for koofr by @suyanhanx in https://github.com/apache/opendal/pull/4050 * feat(layers/dtrace): Support User Statically-Defined Tracing(aka USDT) on Linux by @Zheaoli in https://github.com/apache/opendal/pull/4053 * feat(website): add missing news and organize the onboarding guide by @morristai in https://github.com/apache/opendal/pull/4072 ### Changed * refactor!: avoid hard dep to tokio rt by @tisonkun in https://github.com/apache/opendal/pull/4061 ### Fixed * fix(examples/cpp): display the results to standard output. by @SYaoJun in https://github.com/apache/opendal/pull/4040 * fix(service/icloud):Missing 'X-APPLE-WEBAUTH-USER cookie' and URL initialized failed by @bokket in https://github.com/apache/opendal/pull/4029 * fix: Implement timeout layer correctly by using timeout by @Xuanwo in https://github.com/apache/opendal/pull/4059 * fix(koofr): create_dir when exist by @hoslo in https://github.com/apache/opendal/pull/4062 * fix(seafile): test_list_dir_with_metakey by @hoslo in https://github.com/apache/opendal/pull/4063 * fix: list path recursive should not return path itself by @youngsofun in https://github.com/apache/opendal/pull/4067 ### Docs * docs: Remove not needed actions in release guide by @Xuanwo in https://github.com/apache/opendal/pull/4037 * docs: fix spelling errors in README.md by @SYaoJun in https://github.com/apache/opendal/pull/4039 * docs: New PMC member Liuqing Yue by @Xuanwo in https://github.com/apache/opendal/pull/4047 * docs: New Committer Yang Shuai by @Xuanwo in https://github.com/apache/opendal/pull/4054 * docs(services/sftp): add more explanation for endpoint config by @silver-ymz in https://github.com/apache/opendal/pull/4055 ### CI * ci(services/s3): Use minio/minio image instead by @Xuanwo in https://github.com/apache/opendal/pull/4070 * ci: Fix CI after moving out of workspacs by @Xuanwo in https://github.com/apache/opendal/pull/4081 ### Chore * chore: Delete bindings/ruby/cucumber.yml by @tisonkun in https://github.com/apache/opendal/pull/4030 * chore(website): Bump download link for 0.44.2 release by @Zheaoli in https://github.com/apache/opendal/pull/4034 * chore(website): Update the release tips by @Zheaoli in https://github.com/apache/opendal/pull/4036 * chore: add doap file by @tisonkun in https://github.com/apache/opendal/pull/4038 * chore(website): Add extra artifacts check process in release document by @Zheaoli in https://github.com/apache/opendal/pull/4041 * chore(bindings/dotnet): update cargo.lock and set up ci by @suyanhanx in https://github.com/apache/opendal/pull/4084 * chore(bindings/dotnet): build os detect by @suyanhanx in https://github.com/apache/opendal/pull/4085 * chore(bindings/ocaml): pinning OCaml binding opendal version for release by @Ranxy in https://github.com/apache/opendal/pull/4086 ## [v0.44.2] - 2023-01-19 ### Added * feat: add behavior tests for blocking buffer reader by @WenyXu in https://github.com/apache/opendal/pull/3872 * feat(services): add pcloud support by @hoslo in https://github.com/apache/opendal/pull/3892 * feat(services/hdfs): Atomic write for hdfs by @shbhmrzd in https://github.com/apache/opendal/pull/3875 * feat(services/hdfs): add atomic_write_dir to hdfsconfig debug by @shbhmrzd in https://github.com/apache/opendal/pull/3902 * feat: add MongodbConfig by @zjregee in https://github.com/apache/opendal/pull/3906 * RFC-3898: Concurrent Writer by @WenyXu in https://github.com/apache/opendal/pull/3898 * feat(services): add yandex disk support by @hoslo in https://github.com/apache/opendal/pull/3918 * feat: implement concurrent `MultipartUploadWriter` by @WenyXu in https://github.com/apache/opendal/pull/3915 * feat: add concurrent writer behavior tests by @WenyXu in https://github.com/apache/opendal/pull/3920 * feat: implement concurrent `RangeWriter` by @WenyXu in https://github.com/apache/opendal/pull/3923 * feat: add `concurrent` and `buffer` parameters into FuzzInput by @WenyXu in https://github.com/apache/opendal/pull/3921 * feat(fuzz): add azblob as test service by @suyanhanx in https://github.com/apache/opendal/pull/3931 * feat(services/webhdfs): Implement write with append by @hoslo in https://github.com/apache/opendal/pull/3937 * feat(core/bench): Add benchmark for concurrent write by @Xuanwo in https://github.com/apache/opendal/pull/3942 * feat(oio): add block_write support by @hoslo in https://github.com/apache/opendal/pull/3945 * feat(services/webhdfs): Implement multi write via CONCAT by @hoslo in https://github.com/apache/opendal/pull/3939 * feat(core): Allow retry in concurrent write operations by @Xuanwo in https://github.com/apache/opendal/pull/3958 * feat(services/ghac): Add workaround for AWS S3 based GHES by @Xuanwo in https://github.com/apache/opendal/pull/3985 * feat: Implement path cache and refactor gdrive by @Xuanwo in https://github.com/apache/opendal/pull/3975 * feat(services): add hdfs native layout by @shbhmrzd in https://github.com/apache/opendal/pull/3933 * feat(services/s3): Return error if credential is empty after loaded by @Xuanwo in https://github.com/apache/opendal/pull/4000 * feat(services/gdrive): Use trash instead of permanently deletes by @Xuanwo in https://github.com/apache/opendal/pull/4002 * feat(services): add koofr support by @hoslo in https://github.com/apache/opendal/pull/3981 * feat(icloud): Add basic Apple iCloud Drive support by @bokket in https://github.com/apache/opendal/pull/3980 ### Changed * refactor: Merge compose_{read,write} into enum_utils by @Xuanwo in https://github.com/apache/opendal/pull/3871 * refactor(services/ftp): Impl parse_error instead of `From` by @bokket in https://github.com/apache/opendal/pull/3891 * docs: very minor English wording fix in error message by @gabrielgrant in https://github.com/apache/opendal/pull/3900 * refactor(services/rocksdb): Impl parse_error instead of `From` by @suyanhanx in https://github.com/apache/opendal/pull/3903 * refactor: Re-organize the layout of tests by @Xuanwo in https://github.com/apache/opendal/pull/3904 * refactor(services/etcd): Impl parse_error instead of `From` by @suyanhanx in https://github.com/apache/opendal/pull/3910 * refactor(services/sftp): Impl parse_error instead of `From` by @G-XD in https://github.com/apache/opendal/pull/3914 * refactor!: Bump MSRV to 1.75 by @Xuanwo in https://github.com/apache/opendal/pull/3851 * refactor(services/redis): Impl parse_error instead of `From` by @suyanhanx in https://github.com/apache/opendal/pull/3938 * refactor!: Revert the bump of MSRV to 1.75 by @Xuanwo in https://github.com/apache/opendal/pull/3952 * refactor(services/onedrive): Add OnedriveConfig to implement ConfigDeserializer by @Borber in https://github.com/apache/opendal/pull/3954 * refactor(service/dropbox): Add DropboxConfig by @howiieyu in https://github.com/apache/opendal/pull/3961 * refactor: Polish internal types and remove not needed deps by @Xuanwo in https://github.com/apache/opendal/pull/3964 * refactor: Add concurrent error test for BlockWrite by @Xuanwo in https://github.com/apache/opendal/pull/3968 * refactor: Remove not needed types in icloud by @Xuanwo in https://github.com/apache/opendal/pull/4021 ### Fixed * fix: Bump pyo3 to fix false positive of unnecessary_fallible_conversions by @Xuanwo in https://github.com/apache/opendal/pull/3873 * fix(core): Handling content encoding correctly by @Xuanwo in https://github.com/apache/opendal/pull/3907 * fix: fix RangeWriter incorrect `next_offset` by @WenyXu in https://github.com/apache/opendal/pull/3927 * fix(oio::BlockWrite): fix write_once case by @hoslo in https://github.com/apache/opendal/pull/3953 * fix: Don't retry close if concurrent > 1 to avoid content lost by @Xuanwo in https://github.com/apache/opendal/pull/3957 * fix(doc): fix rfc typos by @howiieyu in https://github.com/apache/opendal/pull/3971 * fix: Don't call wake_by_ref in OperatorFuture by @Xuanwo in https://github.com/apache/opendal/pull/4003 * fix: async fn resumed after initiate part failed by @Xuanwo in https://github.com/apache/opendal/pull/4013 * fix(pcloud,seafile): use get_basename and get_parent by @hoslo in https://github.com/apache/opendal/pull/4020 * fix(ci): remove pr author from review candidates by @dqhl76 in https://github.com/apache/opendal/pull/4023 ### Docs * docs(bindings/python): drop unnecessary patchelf by @tisonkun in https://github.com/apache/opendal/pull/3889 * docs: Polish core's quick start by @Xuanwo in https://github.com/apache/opendal/pull/3896 * docs(gcs): correct the description of credential by @WenyXu in https://github.com/apache/opendal/pull/3928 * docs: Add 0.44.1 download link by @Xuanwo in https://github.com/apache/opendal/pull/3929 * docs(release): how to clean up old releases by @tisonkun in https://github.com/apache/opendal/pull/3934 * docs(website): polish download page by @suyanhanx in https://github.com/apache/opendal/pull/3932 * docs: improve user verify words by @tisonkun in https://github.com/apache/opendal/pull/3941 * docs: fix incorrect word used by @zegevlier in https://github.com/apache/opendal/pull/3944 * docs: improve wording for community pages by @tisonkun in https://github.com/apache/opendal/pull/3978 * docs(bindings/nodejs): copyright in footer by @suyanhanx in https://github.com/apache/opendal/pull/3986 * docs(bindings/nodejs): build docs locally doc by @suyanhanx in https://github.com/apache/opendal/pull/3987 * docs: Fix missing word in download by @Xuanwo in https://github.com/apache/opendal/pull/3993 * docs(bindings/java): copyright in footer by @G-XD in https://github.com/apache/opendal/pull/3996 * docs(website): update footer by @suyanhanx in https://github.com/apache/opendal/pull/4008 * docs: add trademark information to every possible published readme by @PsiACE in https://github.com/apache/opendal/pull/4014 * docs(website): replace podling to project in website by @morristai in https://github.com/apache/opendal/pull/4015 * docs: Update release guide to adapt as a new TLP by @Xuanwo in https://github.com/apache/opendal/pull/4011 * docs: Add WebHDFS version compatibility details by @shbhmrzd in https://github.com/apache/opendal/pull/4024 ### CI * build(deps): bump actions/download-artifact from 3 to 4 by @dependabot in https://github.com/apache/opendal/pull/3885 * build(deps): bump once_cell from 1.18.0 to 1.19.0 by @dependabot in https://github.com/apache/opendal/pull/3880 * build(deps): bump napi-derive from 2.14.2 to 2.14.6 by @dependabot in https://github.com/apache/opendal/pull/3879 * build(deps): bump url from 2.4.1 to 2.5.0 by @dependabot in https://github.com/apache/opendal/pull/3876 * build(deps): bump mlua from 0.8.10 to 0.9.2 by @oowl in https://github.com/apache/opendal/pull/3890 * ci: Disable supabase tests for our test org has been paused by @Xuanwo in https://github.com/apache/opendal/pull/3908 * ci: Downgrade artifact actions until regression addressed by @Xuanwo in https://github.com/apache/opendal/pull/3935 * ci: Refactor fuzz to integrate with test planner by @Xuanwo in https://github.com/apache/opendal/pull/3936 * ci: Pick random reviewers from committer list by @Xuanwo in https://github.com/apache/opendal/pull/4001 ### Chore * chore: update release related docs and script by @dqhl76 in https://github.com/apache/opendal/pull/3870 * chore(NOTICE): update copyright to year 2024 by @suyanhanx in https://github.com/apache/opendal/pull/3894 * chore: Format code to make readers happy by @Xuanwo in https://github.com/apache/opendal/pull/3912 * chore: Remove unused dep async-compat by @Xuanwo in https://github.com/apache/opendal/pull/3947 * chore: display project logo on Rust docs by @tisonkun in https://github.com/apache/opendal/pull/3983 * chore: improve trademarks in bindings docs by @tisonkun in https://github.com/apache/opendal/pull/3984 * chore: use Apache OpenDAL™ in the first and most prominent mention by @tisonkun in https://github.com/apache/opendal/pull/3988 * chore: add more information for javadocs by @tisonkun in https://github.com/apache/opendal/pull/3989 * chore: precise footer by @tisonkun in https://github.com/apache/opendal/pull/3997 * chore: use full form name when necessary by @tisonkun in https://github.com/apache/opendal/pull/3998 * chore(bindings/python): Enable sftp service by default for unix platform by @Zheaoli in https://github.com/apache/opendal/pull/4006 * chore: remove disclaimer by @suyanhanx in https://github.com/apache/opendal/pull/4009 * chore: Remove incubating from releases by @Xuanwo in https://github.com/apache/opendal/pull/4010 * chore: trim incubator prefix everywhere by @tisonkun in https://github.com/apache/opendal/pull/4016 * chore: fixup doc link in release_java.yml by @tisonkun in https://github.com/apache/opendal/pull/4019 * chore: simplify reviewer candidates logic by @tisonkun in https://github.com/apache/opendal/pull/4017 ## [v0.44.1] - 2023-12-31 ### Added * feat(service/memcached): Add MemCachedConfig by @ankit-pn in https://github.com/apache/opendal/pull/3827 * feat(service/rocksdb): Add RocksdbConfig by @ankit-pn in https://github.com/apache/opendal/pull/3828 * feat(services): add chainsafe support by @hoslo in https://github.com/apache/opendal/pull/3834 * feat(bindings/python): Build all available services for python by @Xuanwo in https://github.com/apache/opendal/pull/3836 * feat: Adding Atomicserver config by @k-aishwarya in https://github.com/apache/opendal/pull/3845 * feat(oio::read): implement the async buffer reader by @WenyXu in https://github.com/apache/opendal/pull/3811 * feat(oio::read): implement the blocking buffer reader by @WenyXu in https://github.com/apache/opendal/pull/3860 * feat: adapt the `CompleteReader` by @WenyXu in https://github.com/apache/opendal/pull/3861 * feat: add basic behavior tests for buffer reader by @WenyXu in https://github.com/apache/opendal/pull/3862 * feat: add fuzz reader with buffer tests by @WenyXu in https://github.com/apache/opendal/pull/3866 * feat(ofs): implement ofs based on fuse3 by @Inokinoki in https://github.com/apache/opendal/pull/3857 ### Changed * refactor: simplify `bindings_python.yml` by @messense in https://github.com/apache/opendal/pull/3837 * refactor: Add edge test for aws assume role with web identity by @Xuanwo in https://github.com/apache/opendal/pull/3839 * refactor(services/webdav): Add WebdavConfig to implement ConfigDeserializer by @kwaa in https://github.com/apache/opendal/pull/3846 * refactor: use TwoWays instead of TwoWaysReader and TwoWaysWriter by @WenyXu in https://github.com/apache/opendal/pull/3863 ### Fixed * fix: Add tests for listing recursively on not supported services by @Xuanwo in https://github.com/apache/opendal/pull/3826 * fix(services/upyun): fix list api by @hoslo in https://github.com/apache/opendal/pull/3841 * fix: fix a bypass seek relative bug in `BufferReader` by @WenyXu in https://github.com/apache/opendal/pull/3864 * fix: fix the bypass read does not sync the `cur` of `BufferReader` by @WenyXu in https://github.com/apache/opendal/pull/3865 ### Docs * docs: Add Apache prefix for all bindings by @Xuanwo in https://github.com/apache/opendal/pull/3829 * docs: Add apache prefix for python docs by @Xuanwo in https://github.com/apache/opendal/pull/3830 * docs: Add branding in README by @Xuanwo in https://github.com/apache/opendal/pull/3831 * docs: Add trademark for Apache OpenDAL™ by @Xuanwo in https://github.com/apache/opendal/pull/3832 * docs: Add trademark sign for core by @Xuanwo in https://github.com/apache/opendal/pull/3833 * docs: Enable doc_auto_cfg when docs cfg has been enabled by @Xuanwo in https://github.com/apache/opendal/pull/3835 * docs: Address branding for haskell and C bindings by @Xuanwo in https://github.com/apache/opendal/pull/3840 * doc: add 0.44.0 release link to download.md by @dqhl76 in https://github.com/apache/opendal/pull/3868 ### CI * ci: Remove workflows that not running or ready by @Xuanwo in https://github.com/apache/opendal/pull/3842 * ci: Migrate ftp to test planner by @Xuanwo in https://github.com/apache/opendal/pull/3843 ### Chore * chore(bindings/java): Add name and description metadata by @tisonkun in https://github.com/apache/opendal/pull/3838 * chore(website): improve a bit trademark refs by @tisonkun in https://github.com/apache/opendal/pull/3847 * chore: Fix clippy warnings found in rust 1.75 by @Xuanwo in https://github.com/apache/opendal/pull/3849 * chore(bindings/python): improve ASF branding by @tisonkun in https://github.com/apache/opendal/pull/3850 * chore(bindings/haskell): improve ASF branding by @tisonkun in https://github.com/apache/opendal/pull/3852 * chore(bindings/c): make c binding separate workspace by @suyanhanx in https://github.com/apache/opendal/pull/3856 * chore(bindings/haskell): support co-log-0.6.0 && ghc-9.4 by @silver-ymz in https://github.com/apache/opendal/pull/3858 ## [v0.44.0] - 2023-12-26 ### Added * feat(core): service add HuggingFace file system by @morristai in https://github.com/apache/opendal/pull/3670 * feat(service/moka): bump moka from 0.10.4 to 0.12.1 by @G-XD in https://github.com/apache/opendal/pull/3711 * feat: add operator.list support for OCaml binding by @Young-Flash in https://github.com/apache/opendal/pull/3706 * feat(test): add Huggingface behavior test by @morristai in https://github.com/apache/opendal/pull/3712 * feat: Add list prefix support by @Xuanwo in https://github.com/apache/opendal/pull/3728 * feat: Implement ConcurrentFutures to remove the dependences on tokio by @Xuanwo in https://github.com/apache/opendal/pull/3746 * feat(services): add seafile support by @hoslo in https://github.com/apache/opendal/pull/3771 * RFC-3734: Buffered reader by @WenyXu in https://github.com/apache/opendal/pull/3734 * feat: Add content range support for RpRead by @Xuanwo in https://github.com/apache/opendal/pull/3777 * feat: Add presign_stat_with support by @Xuanwo in https://github.com/apache/opendal/pull/3778 * feat: Add project layout for ofs by @Xuanwo in https://github.com/apache/opendal/pull/3779 * feat(binding/nodejs): align list api by @suyanhanx in https://github.com/apache/opendal/pull/3784 * feat: Make OpenDAL available under wasm32 arch by @Xuanwo in https://github.com/apache/opendal/pull/3796 * feat(services): add upyun support by @hoslo in https://github.com/apache/opendal/pull/3790 * feat: Add edge test s3_read_on_wasm by @Xuanwo in https://github.com/apache/opendal/pull/3802 * feat(services/azblob): available under wasm32 arch by @suyanhanx in https://github.com/apache/opendal/pull/3806 * feat: make services-gdrive compile for wasm target by @Young-Flash in https://github.com/apache/opendal/pull/3808 * feat(core): Make gcs available on wasm32 arch by @Xuanwo in https://github.com/apache/opendal/pull/3816 ### Changed * refactor(service/etcd): use EtcdConfig in from_map by @G-XD in https://github.com/apache/opendal/pull/3703 * refactor(object_store): upgrade object_store to 0.7. by @youngsofun in https://github.com/apache/opendal/pull/3713 * refactor: List must support list without recursive by @Xuanwo in https://github.com/apache/opendal/pull/3721 * refactor: replace ftp tls impl as rustls by @oowl in https://github.com/apache/opendal/pull/3760 * refactor: Remove never used Stream poll_reset API by @Xuanwo in https://github.com/apache/opendal/pull/3774 * refactor: Polish operator read_with by @Xuanwo in https://github.com/apache/opendal/pull/3775 * refactor: Migrate gcs builder to config based by @Xuanwo in https://github.com/apache/opendal/pull/3786 * refactor(service/hdfs): Add HdfsConfig to implement ConfigDeserializer by @shbhmrzd in https://github.com/apache/opendal/pull/3800 * refactor(raw): add parse_header_to_str fn by @hoslo in https://github.com/apache/opendal/pull/3804 * refactor(raw): refactor APIs like parse_content_disposition by @hoslo in https://github.com/apache/opendal/pull/3815 * refactor: Polish http_util parse headers by @Xuanwo in https://github.com/apache/opendal/pull/3817 ### Fixed * fix(oli): Fix cp -r command returns invalid path error by @kebe7jun in https://github.com/apache/opendal/pull/3687 * fix(website): folder name mismatch by @suyanhanx in https://github.com/apache/opendal/pull/3707 * fix(binding/java): fix SPECIAL_DIR_NAME by @G-XD in https://github.com/apache/opendal/pull/3715 * fix(services/dropbox): Workaround for dropbox limitations for create_folder by @Xuanwo in https://github.com/apache/opendal/pull/3719 * fix(ocaml_binding): sort `actual` & `expected` to pass ci by @Young-Flash in https://github.com/apache/opendal/pull/3733 * fix(ci): Make sure merge_local_staging handles all subdir by @Xuanwo in https://github.com/apache/opendal/pull/3788 * fix(services/gdrive): fix return value of `get_file_id_by_path` by @G-XD in https://github.com/apache/opendal/pull/3801 * fix(core): List root should not return itself by @Xuanwo in https://github.com/apache/opendal/pull/3824 ### Docs * docs: add maturity model check by @suyanhanx in https://github.com/apache/opendal/pull/3680 * docs(website): show maturity model by @suyanhanx in https://github.com/apache/opendal/pull/3709 * docs(website): only VOTEs from PPMC members are binding by @G-XD in https://github.com/apache/opendal/pull/3710 * doc: add 0.43.0 release link to download.md by @G-XD in https://github.com/apache/opendal/pull/3729 * docs: Add process on nominating committers and ppmc members by @Xuanwo in https://github.com/apache/opendal/pull/3740 * docs: Deploy website to nightlies for every tags by @Xuanwo in https://github.com/apache/opendal/pull/3739 * docs: Remove not released bindings docs from top level header by @Xuanwo in https://github.com/apache/opendal/pull/3741 * docs: Add dependencies list for all packages by @Xuanwo in https://github.com/apache/opendal/pull/3743 * docs: Update maturity docs by @Xuanwo in https://github.com/apache/opendal/pull/3750 * docs: update the RFC doc by @suyanhanx in https://github.com/apache/opendal/pull/3748 * docs(website): polish deploy to nightlies by @suyanhanx in https://github.com/apache/opendal/pull/3753 * docs: add event calendar in community page by @dqhl76 in https://github.com/apache/opendal/pull/3767 * docs(community): polish events by @suyanhanx in https://github.com/apache/opendal/pull/3768 * docs(bindings/ruby): reflect test framework refactor by @tisonkun in https://github.com/apache/opendal/pull/3798 * docs(website): add service Huggingface to website by @morristai in https://github.com/apache/opendal/pull/3812 * docs: update release docs to add cargo-deny setup by @dqhl76 in https://github.com/apache/opendal/pull/3821 ### CI * build(deps): bump cacache from 11.7.1 to 12.0.0 by @dependabot in https://github.com/apache/opendal/pull/3690 * build(deps): bump prometheus-client from 0.21.2 to 0.22.0 by @dependabot in https://github.com/apache/opendal/pull/3694 * build(deps): bump github/issue-labeler from 3.2 to 3.3 by @dependabot in https://github.com/apache/opendal/pull/3698 * ci: Add behavior test for b2 by @Xuanwo in https://github.com/apache/opendal/pull/3714 * ci(cargo): Add frame pointer support in build flag by @Zheaoli in https://github.com/apache/opendal/pull/3772 * ci: Workaround ring 0.17 build issue, bring aarch64 and armv7l back by @Xuanwo in https://github.com/apache/opendal/pull/3781 * ci: Support CI test for s3_read_on_wasm by @Zheaoli in https://github.com/apache/opendal/pull/3813 ### Chore * chore: bump aws-sdk-s3 from 0.38.0 to 1.4.0 by @memoryFade in https://github.com/apache/opendal/pull/3704 * chore: Disable obs test for workaround by @Xuanwo in https://github.com/apache/opendal/pull/3717 * chore: Fix bindings CI by @Xuanwo in https://github.com/apache/opendal/pull/3722 * chore(binding/nodejs,website): Replace yarn with pnpm by @suyanhanx in https://github.com/apache/opendal/pull/3730 * chore: Bring persy CI back by @Xuanwo in https://github.com/apache/opendal/pull/3751 * chore(bindings/python): upgrade pyo3 to 0.20 by @messense in https://github.com/apache/opendal/pull/3758 * chore: remove unused binding feature file by @tisonkun in https://github.com/apache/opendal/pull/3757 * chore: Bump governor from 0.5.1 to 0.6.0 by @G-XD in https://github.com/apache/opendal/pull/3761 * chore: Split bindings/ocaml to separate workspace by @Xuanwo in https://github.com/apache/opendal/pull/3792 * chore: Split bindings/ruby to separate workspace by @ho-229 in https://github.com/apache/opendal/pull/3794 * chore(bindings/php): bump ext-php-rs to support latest php & separate workspace by @suyanhanx in https://github.com/apache/opendal/pull/3799 * chore: Address comments from hackernews by @Xuanwo in https://github.com/apache/opendal/pull/3805 * chore(bindings/ocaml): dep opendal point to core by @suyanhanx in https://github.com/apache/opendal/pull/3814 ## [v0.43.0] - 2023-11-30 ### Added * feat(bindings/C): Add opendal_operator_rename and opendal_operator_copy by @jiaoew1991 in https://github.com/apache/opendal/pull/3517 * feat(binding/python): Add new API to convert between AsyncOperator and Operator by @Zheaoli in https://github.com/apache/opendal/pull/3514 * feat: Implement RFC-3526: List Recursive by @Xuanwo in https://github.com/apache/opendal/pull/3556 * feat(service): add alluxio rest api support by @hoslo in https://github.com/apache/opendal/pull/3564 * feat(bindings/python): add OPENDAL_DISABLE_RANDOM_ROOT support by @Justin-Xiang in https://github.com/apache/opendal/pull/3550 * feat(core): add Alluxio e2e test by @hoslo in https://github.com/apache/opendal/pull/3573 * feat(service): alluxio support write by @hoslo in https://github.com/apache/opendal/pull/3566 * feat(bindings/nodejs): add retry layer by @suyanhanx in https://github.com/apache/opendal/pull/3484 * RFC: Concurrent Stat in List by @morristai in https://github.com/apache/opendal/pull/3574 * feat(service/hdfs): enable rename in hdfs service by @qingwen220 in https://github.com/apache/opendal/pull/3592 * feat: Improve the read_to_end perf and add benchmark vs_fs by @Xuanwo in https://github.com/apache/opendal/pull/3617 * feat: Add benchmark vs aws sdk s3 by @Xuanwo in https://github.com/apache/opendal/pull/3620 * feat: Improve the performance of s3 services by @Xuanwo in https://github.com/apache/opendal/pull/3622 * feat(service): support b2 by @hoslo in https://github.com/apache/opendal/pull/3604 * feat(core): Implement RFC-3574 Concurrent Stat In List by @morristai in https://github.com/apache/opendal/pull/3599 * feat: Implement stat dir correctly based on RFC-3243 List Prefix by @Xuanwo in https://github.com/apache/opendal/pull/3651 * feat(bindings/nodejs): Add capability support by @suyanhanx in https://github.com/apache/opendal/pull/3654 * feat: disable `ftp` for python and java binding by @ZutJoe in https://github.com/apache/opendal/pull/3659 * feat(bindings/nodejs): read/write stream by @suyanhanx in https://github.com/apache/opendal/pull/3619 ### Changed * refactor(services/persy): migrate tot test planner by @G-XD in https://github.com/apache/opendal/pull/3476 * refactor(service/etcd): Add EtcdConfig to implement ConfigDeserializer by @Xuxiaotuan in https://github.com/apache/opendal/pull/3543 * refactor(services/azblob): add AzblobConfig by @acehinnnqru in https://github.com/apache/opendal/pull/3553 * refactor(services/cacache): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3568 * refactor(services/sled): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3569 * refactor(services/webhdfs): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3578 * refactor(core): Rename all `Page` to `List` by @Xuanwo in https://github.com/apache/opendal/pull/3589 * refactor: Change List API into poll based and return one entry instead by @Xuanwo in https://github.com/apache/opendal/pull/3593 * refactor(services/tikv): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3587 * refactor(service/redis): Migrate task to new task planner by @sunheyi6 in https://github.com/apache/opendal/pull/3374 * refactor(oio): Polish IncomingAsyncBody::bytes by @Xuanwo in https://github.com/apache/opendal/pull/3621 * refactor(services/rocksdb): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3636 * refactor(services/azfile): Check if dir exists before create by @ZutJoe in https://github.com/apache/opendal/pull/3652 * refactor: Polish concurrent list by @Xuanwo in https://github.com/apache/opendal/pull/3658 ### Fixed * fix(bindings/python): Fix the test command in doc by @Justin-Xiang in https://github.com/apache/opendal/pull/3541 * fix(ci): try enable corepack before setup-node action by @suyanhanx in https://github.com/apache/opendal/pull/3609 * fix(service/hdfs): enable hdfs append support by @qingwen220 in https://github.com/apache/opendal/pull/3600 * fix(ci): fix setup node by @suyanhanx in https://github.com/apache/opendal/pull/3611 * fix(core): Path in remove not normalized by @Xuanwo in https://github.com/apache/opendal/pull/3671 * fix(core): Build with redis features and Rust < 1.72 by @vincentdephily in https://github.com/apache/opendal/pull/3683 ### Docs * docs: Add questdb in users list by @caicancai in https://github.com/apache/opendal/pull/3532 * docs: Add macos specific doc updates for hdfs by @shbhmrzd in https://github.com/apache/opendal/pull/3559 * docs(bindings/python): Fix the test command in doc by @Justin-Xiang in https://github.com/apache/opendal/pull/3561 * docs(bindings/java): add basic usage in README by @caicancai in https://github.com/apache/opendal/pull/3534 * doc: add 0.42.0 release link to download.md by @silver-ymz in https://github.com/apache/opendal/pull/3598 ### CI * ci(services/libsql): add rust test threads limit by @G-XD in https://github.com/apache/opendal/pull/3540 * ci(services/redb): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3518 * ci: Disable libsql behavior test until we or upstream address them by @Xuanwo in https://github.com/apache/opendal/pull/3552 * ci: Add new Python binding reviewer by @Zheaoli in https://github.com/apache/opendal/pull/3560 * ci(bindings/nodejs): add aarch64 build support by @suyanhanx in https://github.com/apache/opendal/pull/3567 * ci(planner): Polish the workflow planner code by @Zheaoli in https://github.com/apache/opendal/pull/3570 * ci(core): Add dry run for rc tags by @Xuanwo in https://github.com/apache/opendal/pull/3624 * ci: Disable persy until it has been fixed by @Xuanwo in https://github.com/apache/opendal/pull/3631 * ci: Calling cargo to make sure rust has been setup by @Xuanwo in https://github.com/apache/opendal/pull/3633 * ci: Fix etcd with tls and auth failed to start by @Xuanwo in https://github.com/apache/opendal/pull/3637 * ci(services/etcd): Use ALLOW_NONE_AUTHENTICATION as workaround by @Xuanwo in https://github.com/apache/opendal/pull/3638 * ci: dry run publish on rc tags for python binding by @everpcpc in https://github.com/apache/opendal/pull/3645 * ci: Add java linux arm64 build by @Xuanwo in https://github.com/apache/opendal/pull/3660 * ci(java/binding): Use zigbuild for glibc 2.17 support by @Xuanwo in https://github.com/apache/opendal/pull/3664 * ci(bindings/python): remove aarch support by @G-XD in https://github.com/apache/opendal/pull/3674 ### Chore * chore(servies/sftp): Upgrade openssh-sftp-client to 0.14 by @sd44 in https://github.com/apache/opendal/pull/3538 * chore(service/tikv): rename Backend to TikvBackend by @caicancai in https://github.com/apache/opendal/pull/3545 * chore(docs): add cpp binding in README by @cjj2010 in https://github.com/apache/opendal/pull/3546 * chore(service/sqlite): fix typo on sqlite by @caicancai in https://github.com/apache/opendal/pull/3549 * chore(bindings/C): resolve doxygen warnings by @sd44 in https://github.com/apache/opendal/pull/3572 * chore: removed dotenv in bindings/nodejs/index.js by @AlexVCS in https://github.com/apache/opendal/pull/3579 * chore: update opentelemetry to v0.21.x by @jtescher in https://github.com/apache/opendal/pull/3580 * chore: Add cpp binding Google style clang-format && format the code by @JackDrogon in https://github.com/apache/opendal/pull/3581 * chore: bump suppaftp version to 5.2 by @oowl in https://github.com/apache/opendal/pull/3590 * chore(ci): fix artifacts path for publish pypi by @everpcpc in https://github.com/apache/opendal/pull/3597 * chore: Code cleanup to make rust 1.74 happy by @Xuanwo in https://github.com/apache/opendal/pull/3602 * chore: Fix `raw::tests` been excluded unexpectedly by @Xuanwo in https://github.com/apache/opendal/pull/3623 * chore: Bump dpes and remove native-tls in mysql-async by @Xuanwo in https://github.com/apache/opendal/pull/3627 * chore(core): Have mysql_async use rustls instead of native-tls by @amunra in https://github.com/apache/opendal/pull/3634 * chore: Polish docs for Capability by @Xuanwo in https://github.com/apache/opendal/pull/3635 * chore: Bump reqsign to 0.14.4 for jsonwebtoken by @Xuanwo in https://github.com/apache/opendal/pull/3644 * chore(ci): nodejs binding publish dry run by @suyanhanx in https://github.com/apache/opendal/pull/3632 * chore: Polish comments for `stat` and `stat_with` by @Xuanwo in https://github.com/apache/opendal/pull/3657 * chore: clearer doc for python binding by @wcy-fdu in https://github.com/apache/opendal/pull/3667 * chore: Bump to v0.43.0 to start release process by @G-XD in https://github.com/apache/opendal/pull/3672 * chore: Bump to v0.43.0 to start release process (Round 2) by @G-XD in https://github.com/apache/opendal/pull/3676 * chore: add license.workspace to help cargo deny reports by @tisonkun in https://github.com/apache/opendal/pull/3679 * chore: clearer doc for list metakey by @wcy-fdu in https://github.com/apache/opendal/pull/3666 ## [v0.42.0] - 2023-11-07 ### Added * feat(binding/java): add `rename` support by @G-XD in https://github.com/apache/opendal/pull/3238 * feat(prometheus): add bytes metrics as counter by @flaneur2020 in https://github.com/apache/opendal/pull/3246 * feat(binding/python): new behavior testing for python by @laipz8200 in https://github.com/apache/opendal/pull/3245 * feat(binding/python): Support AsyncOperator tests. by @laipz8200 in https://github.com/apache/opendal/pull/3254 * feat(service/libsql): support libsql by @G-XD in https://github.com/apache/opendal/pull/3233 * feat(binding/python): allow setting append/buffer/more in write() call by @jokester in https://github.com/apache/opendal/pull/3256 * feat(services/persy): change blocking_x in async_x call to tokio::task::blocking_spawn by @Zheaoli in https://github.com/apache/opendal/pull/3221 * feat: Add edge test cases for OpenDAL Core by @Xuanwo in https://github.com/apache/opendal/pull/3274 * feat(service/d1): Support d1 for opendal by @realtaobo in https://github.com/apache/opendal/pull/3248 * feat(services/redb): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in https://github.com/apache/opendal/pull/3276 * feat: Add blocking layer for C bindings by @jiaoew1991 in https://github.com/apache/opendal/pull/3278 * feat(binding/c): Add blocking_reader for C binding by @jiaoew1991 in https://github.com/apache/opendal/pull/3259 * feat(services/sled): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in https://github.com/apache/opendal/pull/3280 * feat(services/rocksdb): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in https://github.com/apache/opendal/pull/3279 * feat(binding/java): make `Metadata` a POJO by @G-XD in https://github.com/apache/opendal/pull/3277 * feat(bindings/java): convey backtrace on exception by @tisonkun in https://github.com/apache/opendal/pull/3286 * feat(layer/prometheus): Support custom metric bucket for Histogram by @Zheaoli in https://github.com/apache/opendal/pull/3275 * feat(bindings/python): read APIs return `memoryview` instead of `bytes` to avoid copy by @messense in https://github.com/apache/opendal/pull/3310 * feat(service/azfile): add azure file service support by @dqhl76 in https://github.com/apache/opendal/pull/3312 * feat(services/oss): Add allow anonymous support by @Xuanwo in https://github.com/apache/opendal/pull/3321 * feat(bindings/python): build and publish aarch64 and armv7l wheels by @messense in https://github.com/apache/opendal/pull/3325 * feat(bindings/java): support duplicate operator by @tisonkun in https://github.com/apache/opendal/pull/3330 * feat(core): Add enabled for Scheme by @Xuanwo in https://github.com/apache/opendal/pull/3331 * feat(bindings/java): support get enabled services by @tisonkun in https://github.com/apache/opendal/pull/3336 * feat(bindings/java): Migrate behavior tests to new Workflow Planner by @Xuanwo in https://github.com/apache/opendal/pull/3341 * feat(layer/prometheus): Support output path as a metric label by @Zheaoli in https://github.com/apache/opendal/pull/3335 * feat(service/mongodb): Support mongodb service by @Zheaoli in https://github.com/apache/opendal/pull/3355 * feat: Make PrometheusClientLayer Clonable by @flaneur2020 in https://github.com/apache/opendal/pull/3352 * feat(service/cloudflare_kv): support cloudflare KV by @my-vegetable-has-exploded in https://github.com/apache/opendal/pull/3348 * feat(core): exposing `Metadata::metakey()` api by @G-XD in https://github.com/apache/opendal/pull/3373 * feat(binding/java): add list & remove_all support by @G-XD in https://github.com/apache/opendal/pull/3333 * feat: Add write_total_max_size in Capability by @realtaobo in https://github.com/apache/opendal/pull/3309 * feat(core): service add DBFS API 2.0 support by @morristai in https://github.com/apache/opendal/pull/3334 * feat(bindings/java): use random root for behavior tests by @tisonkun in https://github.com/apache/opendal/pull/3408 * feat(services/oss): Add start-after support for oss list by @wcy-fdu in https://github.com/apache/opendal/pull/3410 * feat(binding/python): Export full_capability API for Python binding by @Zheaoli in https://github.com/apache/opendal/pull/3402 * feat(test): Enable new test workflow planner for python binding by @Zheaoli in https://github.com/apache/opendal/pull/3397 * feat: Implement Lazy Reader by @Xuanwo in https://github.com/apache/opendal/pull/3395 * feat(binding/nodejs): upgrade test behavior and infra by @eryue0220 in https://github.com/apache/opendal/pull/3297 * feat(binding/python): Support Copy operation for Python binding by @Zheaoli in https://github.com/apache/opendal/pull/3454 * feat(bindings/python): Add layer API for operator by @Xuanwo in https://github.com/apache/opendal/pull/3464 * feat(bindings/java): add layers onto ops by @tisonkun in https://github.com/apache/opendal/pull/3392 * feat(binding/python): Support rename API for Python binding by @Zheaoli in https://github.com/apache/opendal/pull/3467 * feat(binding/python): Support remove_all API for Python binding by @Zheaoli in https://github.com/apache/opendal/pull/3469 * feat(core): fix token leak in OneDrive by @morristai in https://github.com/apache/opendal/pull/3470 * feat(core): service add OpenStack Swift support by @morristai in https://github.com/apache/opendal/pull/3461 * feat(bindings/python)!: Implement File and AsyncFile to replace Reader by @Xuanwo in https://github.com/apache/opendal/pull/3474 * feat(services): Implement ConfigDeserializer and add S3Config as example by @Xuanwo in https://github.com/apache/opendal/pull/3490 * feat(core): add OpenStack Swift e2e test by @morristai in https://github.com/apache/opendal/pull/3493 * feat(doc): add OpenStack Swift document for the website by @morristai in https://github.com/apache/opendal/pull/3494 * feat(services/sqlite): add SqliteConfig by @hoslo in https://github.com/apache/opendal/pull/3497 * feat(bindings/C): implement capability by @Ji-Xinyou in https://github.com/apache/opendal/pull/3479 * feat: add mongodb gridfs service support by @realtaobo in https://github.com/apache/opendal/pull/3491 * feat(services): add RedisConfig by @hoslo in https://github.com/apache/opendal/pull/3498 * feat: Add opendal_metadata_last_modified and opendal_operator_create_dir by @jiaoew1991 in https://github.com/apache/opendal/pull/3515 ### Changed * refactor(services/sqlite): Polish sqlite via adding connection pool by @Xuanwo in https://github.com/apache/opendal/pull/3249 * refactor: Remove cucumber based test in python by @laipz8200 in https://github.com/apache/opendal/pull/3253 * refactor: Introduce OpenDAL Workflow Planner by @Xuanwo in https://github.com/apache/opendal/pull/3258 * refactor(bindings/C): Implement error with error message by @Ji-Xinyou in https://github.com/apache/opendal/pull/3250 * refactor(oay): import dav-server-opendalfs by @Young-Flash in https://github.com/apache/opendal/pull/3285 * refactor(bindings/java): explicit error handling by @tisonkun in https://github.com/apache/opendal/pull/3288 * refactor(services/gdrive): Extract folder search logic by @Xuanwo in https://github.com/apache/opendal/pull/3234 * refactor(core): use `list_with` in `Operator::list` by @G-XD in https://github.com/apache/opendal/pull/3305 * refactor(!): Bump and update MSRV to 1.67 by @Xuanwo in https://github.com/apache/opendal/pull/3316 * refactor(tests): Apply OPENDAL_TEST for behavior test by @Xuanwo in https://github.com/apache/opendal/pull/3322 * refactor(bindings/java): align test idiom with OPENDAL_TEST by @tisonkun in https://github.com/apache/opendal/pull/3328 * refactor(bindings/java): split behavior tests by @tisonkun in https://github.com/apache/opendal/pull/3332 * refactor(ci/behavior_test): Migrate to 1password instead by @Xuanwo in https://github.com/apache/opendal/pull/3338 * refactor(core/{fuzz,benches}): Migrate to OPENDANL_TEST by @Xuanwo in https://github.com/apache/opendal/pull/3343 * refactor(bindings/C): Alter naming convention for consistency by @Ji-Xinyou in https://github.com/apache/opendal/pull/3282 * refactor(service/mysql): Migrate to new task planner by @Zheaoli in https://github.com/apache/opendal/pull/3357 * refactor(service/postgresql): Migrate task to new task planner by @Zheaoli in https://github.com/apache/opendal/pull/3358 * refactor(services/etcd): Migrate etcd task to new behavior test planner by @Zheaoli in https://github.com/apache/opendal/pull/3360 * refactor(services/http): Migrate http task to new behavior test planner by @Zheaoli in https://github.com/apache/opendal/pull/3362 * refactor(services/sqlite): Migrate sqlite task to new behavior test planner by @Zheaoli in https://github.com/apache/opendal/pull/3365 * refactor(services/gdrive): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3368 * refactor(services/redis): migrate to test planner for kvrocks,dragonfly by @suyanhanx in https://github.com/apache/opendal/pull/3369 * refactor(services/azblob): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3370 * refactor(services/cos,obs): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3371 * refactor(services/oss): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3375 * refactor(services/memcached): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3377 * refactor(services/gcs): migrate tot test planner by @suyanhanx in https://github.com/apache/opendal/pull/3391 * refactor(services/moka): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3394 * refactor(services/dashmap): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3396 * refactor(services/memory): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3390 * refactor(services/azdls): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3405 * refactor(services/mini_moka): migrate to test planner by @dqhl76 in https://github.com/apache/opendal/pull/3416 * refactor(core/fuzz): Fix some bugs inside fuzzer by @Xuanwo in https://github.com/apache/opendal/pull/3418 * refactor(tests): Extract tests related logic into raw::tests for reuse by @Xuanwo in https://github.com/apache/opendal/pull/3420 * refactor(service/dropbox): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3381 * refactor(services/supabase): migrate to test planner by @G-XD in https://github.com/apache/opendal/pull/3406 * refactor(services/sftp): migrate to test planner by @suyanhanx in https://github.com/apache/opendal/pull/3412 * refactor(services/wasabi)!: Remove native support for wasabi services by @Xuanwo in https://github.com/apache/opendal/pull/3455 * refactor(ci): Polish the test planner code by @Zheaoli in https://github.com/apache/opendal/pull/3457 * refactor(services/webdav): migrate to test planner for webdav by @shauvet in https://github.com/apache/opendal/pull/3379 * refactor(services/redis): Enable rustls support by default for redis by @Xuanwo in https://github.com/apache/opendal/pull/3471 * refactor(bindings/python): Refactor layout for python bindings by @Xuanwo in https://github.com/apache/opendal/pull/3473 * refactor(services/libsql): Migrate libsql task to new behavior test planner by @Zheaoli in https://github.com/apache/opendal/pull/3363 * refactor(service/postgresql): Add PostgresqlConfig to implement ConfigDeserializer by @sd44 in https://github.com/apache/opendal/pull/3495 * refactor(binding/python): Add multiple custom exception for each of error code in Rust Core by @Zheaoli in https://github.com/apache/opendal/pull/3492 * refactor(service/libsql): Add LibsqlConfig to implement ConfigDeserializer by @sd44 in https://github.com/apache/opendal/pull/3501 * refactor(service/http): Add HttpConfig to implement ConfigDeserializer by @sd44 in https://github.com/apache/opendal/pull/3507 * refactor(service/ftp): Add FtpConfig to implement ConfigDeserializer by @sd44 in https://github.com/apache/opendal/pull/3510 * refactor(service/sftp): Add SftpConfig to implement ConfigDeserializer by @sd44 in https://github.com/apache/opendal/pull/3511 * refactor(service/tikv): Add TikvConfig to implement ConfigDeserializer by @caicancai in https://github.com/apache/opendal/pull/3512 ### Fixed * fix: Fix read result not full by @jiaoew1991 in https://github.com/apache/opendal/pull/3350 * fix(services/cos): fix prefix param by @G-XD in https://github.com/apache/opendal/pull/3384 * fix(services/ghac)!: Remove enable_create_simulation support for ghac by @Xuanwo in https://github.com/apache/opendal/pull/3423 * fix: ASF event URL by @tisonkun in https://github.com/apache/opendal/pull/3431 * fix(binding/java): fix return value of presign-related method by @G-XD in https://github.com/apache/opendal/pull/3433 * fix(mongo/backend): remove redundant code by @bestgopher in https://github.com/apache/opendal/pull/3439 * fix: nodejs test adapt `OPENDAL_DISABLE_RANDOM_ROOT` by @suyanhanx in https://github.com/apache/opendal/pull/3456 * fix(services/s3): Accept List responses without ETag by @amunra in https://github.com/apache/opendal/pull/3478 * fix(bindings/python): fix type annotations and improve docs by @messense in https://github.com/apache/opendal/pull/3483 * fix(services/dropbox): Check if folder exists before calling create dir by @leenstx in https://github.com/apache/opendal/pull/3513 ### Docs * docs: Add docs in website for sqlite/mysql/postgresql services by @Zheaoli in https://github.com/apache/opendal/pull/3290 * docs: add docs in website for atomicserver by @Zheaoli in https://github.com/apache/opendal/pull/3293 * docs: Add docs on website for GHAC service by @Zheaoli in https://github.com/apache/opendal/pull/3296 * docs: Add docs on website for cacache services by @Zheaoli in https://github.com/apache/opendal/pull/3294 * docs: Add docs on website for libsql services by @Zheaoli in https://github.com/apache/opendal/pull/3299 * docs: download link for v0.41.0 by @suyanhanx in https://github.com/apache/opendal/pull/3298 * docs: Add docs on website for persy service by @Zheaoli in https://github.com/apache/opendal/pull/3300 * docs: Add docs on website for d1 services by @Zheaoli in https://github.com/apache/opendal/pull/3295 * docs: Add docs on website for redb service by @Zheaoli in https://github.com/apache/opendal/pull/3301 * docs: Add docs on website for tikv service by @Zheaoli in https://github.com/apache/opendal/pull/3302 * docs: Add docs on website for Vercel Artifacts service by @Zheaoli in https://github.com/apache/opendal/pull/3303 * docs: update release doc by @suyanhanx in https://github.com/apache/opendal/pull/3306 * docs(bindings): bindings README and binding release status by @suyanhanx in https://github.com/apache/opendal/pull/3340 * docs(bindings/java): update how to run behavior test by @tisonkun in https://github.com/apache/opendal/pull/3342 * docs: fix something in docs by @my-vegetable-has-exploded in https://github.com/apache/opendal/pull/3353 * docs: Update mysql `connection_string` config description in doc by @xring in https://github.com/apache/opendal/pull/3388 * doc: apply `range_reader` change in upgrade doc by @wcy-fdu in https://github.com/apache/opendal/pull/3401 * docs(readme): Fix capitalization about the ABFS service in README.md by @caicancai in https://github.com/apache/opendal/pull/3485 * docs: Add Milvus as C binding's user by @Xuanwo in https://github.com/apache/opendal/pull/3523 ### CI * ci: Add bindings_go workflow by @jiaoew1991 in https://github.com/apache/opendal/pull/3260 * ci: Only fetch origin while in pull request by @Xuanwo in https://github.com/apache/opendal/pull/3268 * ci: add a new test case for the disk is full by @sunheyi6 in https://github.com/apache/opendal/pull/3079 * ci: Passing GITHUB_TOKEN to avoid rate limit by @Xuanwo in https://github.com/apache/opendal/pull/3272 * ci(services/hdfs): Use dlcdn.apache.org instead by @Xuanwo in https://github.com/apache/opendal/pull/3308 * ci: Fix HDFS test by @Xuanwo in https://github.com/apache/opendal/pull/3320 * ci: Fix plan not generated correctly for PR from forked repo by @Xuanwo in https://github.com/apache/opendal/pull/3327 * ci(services/azfile): add azfile integration test by @dqhl76 in https://github.com/apache/opendal/pull/3409 * ci: Fix behavior tests been ignored by @Xuanwo in https://github.com/apache/opendal/pull/3422 * ci(binding/java): remove `testWriteFileWithNonAsciiName` behavior test by @G-XD in https://github.com/apache/opendal/pull/3424 * ci(bindings/python): Remove not passing test cases until we addressed by @Xuanwo in https://github.com/apache/opendal/pull/3432 * ci(services/sftp): Move setup logic into docker-compose by @Xuanwo in https://github.com/apache/opendal/pull/3430 * ci(test): Add health check for WebDAV docker compose config by @Zheaoli in https://github.com/apache/opendal/pull/3448 * ci: Switch to 1password connect to avoid rate limit by @Xuanwo in https://github.com/apache/opendal/pull/3447 * ci: Use cargo test instead of carge nextest by @Xuanwo in https://github.com/apache/opendal/pull/3505 * build(bindings/java): Allow building on `linux-aarch_64` by @amunra in https://github.com/apache/opendal/pull/3527 * ci: support behavior test for gridfs by @realtaobo in https://github.com/apache/opendal/pull/3520 ### Chore * chore(ci): publish to pypi with github OIDC credential by @everpcpc in https://github.com/apache/opendal/pull/3252 * chore(bindings/java): align mapping POJO pattern by @tisonkun in https://github.com/apache/opendal/pull/3289 * chore: do not export unreleased bindings by @suyanhanx in https://github.com/apache/opendal/pull/3339 * chore: update object_store unit tests and s3 endpoint docs by @thorseraq in https://github.com/apache/opendal/pull/3345 * chore: Fix typo in mysql doc by @lewiszlw in https://github.com/apache/opendal/pull/3351 * chore: try format yaml files by @suyanhanx in https://github.com/apache/opendal/pull/3364 * chore(bindings/java): move out convert fns by @tisonkun in https://github.com/apache/opendal/pull/3389 * chore(bindings/java): use JDK 8 time APIs by @tisonkun in https://github.com/apache/opendal/pull/3400 * chore: remove unused dependencies by @xxchan in https://github.com/apache/opendal/pull/3414 * chore(test): Compare with digest instead of whole content by @Xuanwo in https://github.com/apache/opendal/pull/3419 * chore: remove useless workflow file by @suyanhanx in https://github.com/apache/opendal/pull/3425 * chore(deps): bump minitrace from 0.5.1 to 0.6.1 by @andylokandy in https://github.com/apache/opendal/pull/3449 * chore(deps): bump korandoru/hawkeye from 3.4.0 to 3.6.0 by @dependabot in https://github.com/apache/opendal/pull/3446 * chore(deps): bump toml from 0.7.8 to 0.8.6 by @dependabot in https://github.com/apache/opendal/pull/3442 * chore(deps): bump actions/setup-node from 3 to 4 by @dependabot in https://github.com/apache/opendal/pull/3445 * chore(deps): bump etcd-client from 0.11.1 to 0.12.1 by @dependabot in https://github.com/apache/opendal/pull/3441 * chore(services/libsql): Fix typos in backend by @sd44 in https://github.com/apache/opendal/pull/3506 * chore: Bump to v0.42.0 to start release process by @silver-ymz in https://github.com/apache/opendal/pull/3509 * chore(service/vercel_artifacts): add doc in backend by @caicancai in https://github.com/apache/opendal/pull/3508 * chore: Remove not released packages while releasing by @Xuanwo in https://github.com/apache/opendal/pull/3519 * chore: Bump to v0.42.0 to start release process (Round 2) by @silver-ymz in https://github.com/apache/opendal/pull/3521 * chore: Fix typo in CHANGELOG by @caicancai in https://github.com/apache/opendal/pull/3524 * chore: add updated Cargo.toml to git archive by @silver-ymz in https://github.com/apache/opendal/pull/3525 * chore(bindings/java): improve build.py script by @tisonkun in https://github.com/apache/opendal/pull/3529 ## [v0.41.0] - 2023-10-08 ### Added * feat: allow using `prometheus-client` crate with PrometheusClientLayer by @flaneur2020 in https://github.com/apache/opendal/pull/3134 * feat(binding/java): support info ops by @G-XD in https://github.com/apache/opendal/pull/3154 * test(binding/java): add behavior test framework by @G-XD in https://github.com/apache/opendal/pull/3129 * feat: Include starting offset for GHAC upload Content-Range by @huonw in https://github.com/apache/opendal/pull/3163 * feat(bindings/cpp): make ReaderStream manage the lifetime of Reader by @silver-ymz in https://github.com/apache/opendal/pull/3164 * feat: Enable multi write for ghac by @Xuanwo in https://github.com/apache/opendal/pull/3165 * feat: Add mysql support for OpenDAL by @Zheaoli in https://github.com/apache/opendal/pull/3170 * feat(service/postgresql): support connection pool by @Zheaoli in https://github.com/apache/opendal/pull/3176 * feat(services/ghac): Allow explicitly setting ghac endpoint/token, not just env vars by @huonw in https://github.com/apache/opendal/pull/3177 * feat(service/azdls): add append support for azdls by @dqhl76 in https://github.com/apache/opendal/pull/3186 * feat(bindings/python): Enable `BlockingLayer` for non-blocking services that don't support blocking by @messense in https://github.com/apache/opendal/pull/3198 * feat: Add write_can_empty in Capability and related tests by @Xuanwo in https://github.com/apache/opendal/pull/3200 * feat: Add basic support for bindings/go using CGO by @jiaoew1991 in https://github.com/apache/opendal/pull/3204 * feat(binding/java): add `copy` test by @G-XD in https://github.com/apache/opendal/pull/3207 * feat(service/sqlite): Support sqlite for opendal by @Zheaoli in https://github.com/apache/opendal/pull/3212 * feat(services/sqlite): Support blocking_get/set/delete in sqlite service by @Zheaoli in https://github.com/apache/opendal/pull/3218 * feat(oay): port `WebdavFs` to dav-server-fs-opendal by @Young-Flash in https://github.com/apache/opendal/pull/3119 ### Changed * refactor(services/dropbox): Use OpWrite instead of passing all args as parameters by @ImSingee in https://github.com/apache/opendal/pull/3126 * refactor(binding/java): read should return bytes by @tisonkun in https://github.com/apache/opendal/pull/3153 * refactor(bindings/java)!: operator jni calls by @tisonkun in https://github.com/apache/opendal/pull/3166 * refactor(tests): reuse function to remove duplicate code by @zhao-gang in https://github.com/apache/opendal/pull/3219 ### Fixed * fix(tests): Create test files one by one instead of concurrently by @Xuanwo in https://github.com/apache/opendal/pull/3132 * chore(ci): fix web identity token path for aws s3 assume role test by @everpcpc in https://github.com/apache/opendal/pull/3141 * fix(services/s3): Detect region returned too early when header is empty by @Xuanwo in https://github.com/apache/opendal/pull/3187 * fix: making OpenDAL compilable on 32hf platforms by @ClSlaid in https://github.com/apache/opendal/pull/3188 * fix(binding/java): decode Java’s modified UTF-8 format by @G-XD in https://github.com/apache/opendal/pull/3195 ### Docs * docs(release): describe how to close the Nexus staging repo by @tisonkun in https://github.com/apache/opendal/pull/3125 * docs: update release docs for cpp and haskell bindings by @silver-ymz in https://github.com/apache/opendal/pull/3130 * docs: Polish VISION to make it more clear by @Xuanwo in https://github.com/apache/opendal/pull/3135 * docs: Add start tracking issues about the next release by @Xuanwo in https://github.com/apache/opendal/pull/3145 * docs: Add download link for 0.40.0 by @Xuanwo in https://github.com/apache/opendal/pull/3149 * docs(bindings/cpp): add more using details about cmake by @silver-ymz in https://github.com/apache/opendal/pull/3155 * docs(bindings/java): Added an example of adding dependencies using Gradle by @eastack in https://github.com/apache/opendal/pull/3158 * docs: include disclaimer in announcement template by @Venderbad in https://github.com/apache/opendal/pull/3172 * docs: Add pants as a user by @huonw in https://github.com/apache/opendal/pull/3180 * docs: Add basic readme for go binding by @Xuanwo in https://github.com/apache/opendal/pull/3206 * docs: add multilingual getting started by @tisonkun in https://github.com/apache/opendal/pull/3214 * docs: multiple improvements by @tisonkun in https://github.com/apache/opendal/pull/3215 * docs: Add verify script by @Xuanwo in https://github.com/apache/opendal/pull/3239 ### CI * ci: Align tags with semver specs by @Xuanwo in https://github.com/apache/opendal/pull/3136 * ci: Migrate obs to databend labs sponsored bucket by @Xuanwo in https://github.com/apache/opendal/pull/3137 * build(bindings/java): support develop with JDK 21 by @tisonkun in https://github.com/apache/opendal/pull/3140 * ci: Migrate GCS to Databend Labs sponsored bucket by @Xuanwo in https://github.com/apache/opendal/pull/3142 * build(bindings/java): upgrade maven wrapper version by @tisonkun in https://github.com/apache/opendal/pull/3167 * build(bindings/java): support explicit cargo build target by @tisonkun in https://github.com/apache/opendal/pull/3168 * ci: Pin Kvrocks docker image to 2.5.1 to avoid test failure by @git-hulk in https://github.com/apache/opendal/pull/3192 * ci(bindings/ocaml): add doc by @Ranxy in https://github.com/apache/opendal/pull/3208 * build(deps): bump actions/checkout from 3 to 4 by @dependabot in https://github.com/apache/opendal/pull/3222 * build(deps): bump korandoru/hawkeye from 3.3.0 to 3.4.0 by @dependabot in https://github.com/apache/opendal/pull/3223 * build(deps): bump rusqlite from 0.25.4 to 0.29.0 by @dependabot in https://github.com/apache/opendal/pull/3226 ### Chore * chore(bindings/haskell): add rpath to haskell linker option by @silver-ymz in https://github.com/apache/opendal/pull/3128 * chore(ci): add test for aws s3 assume role by @everpcpc in https://github.com/apache/opendal/pull/3139 * chore: Incorrect debug information by @OmAximani0 in https://github.com/apache/opendal/pull/3183 * chore: bump quick-xml version to 0.30 by @Venderbad in https://github.com/apache/opendal/pull/3190 * chore: Let's welcome the contributors from hacktoberfest! by @Xuanwo in https://github.com/apache/opendal/pull/3193 * chore(bindings/java): simplify library path resolution by @tisonkun in https://github.com/apache/opendal/pull/3196 * chore: Make clippy happy by @Xuanwo in https://github.com/apache/opendal/pull/3229 ## [v0.40.0] - 2023-09-18 ### Added * feat(service/etcd): support list by @G-XD in https://github.com/apache/opendal/pull/2755 * feat: setup the integrate with PHP binding by @godruoyi in https://github.com/apache/opendal/pull/2726 * feat(oay): Add `read_dir` by @Young-Flash in https://github.com/apache/opendal/pull/2736 * feat(obs): support loading credential from env by @everpcpc in https://github.com/apache/opendal/pull/2767 * feat: add async backtrace layer by @dqhl76 in https://github.com/apache/opendal/pull/2765 * feat: Add OCaml Binding by @Ranxy in https://github.com/apache/opendal/pull/2757 * feat(bindings/haskell): support logging layer by @silver-ymz in https://github.com/apache/opendal/pull/2705 * feat: Add FoundationDB Support for OpenDAL by @ArmandoZ in https://github.com/apache/opendal/pull/2751 * feat(oay): add write for oay webdav by @Young-Flash in https://github.com/apache/opendal/pull/2769 * feat: Implement RFC-2774 Lister API by @Xuanwo in https://github.com/apache/opendal/pull/2787 * feat(bindings/haskell): enhance original `OpMonad` to support custom IO monad by @silver-ymz in https://github.com/apache/opendal/pull/2789 * feat: Add into_seekable_read_by_range support for blocking read by @Xuanwo in https://github.com/apache/opendal/pull/2799 * feat(layers/blocking): add blocking layer by @yah01 in https://github.com/apache/opendal/pull/2780 * feat: Add async list with metakey support by @Xuanwo in https://github.com/apache/opendal/pull/2803 * feat(binding/php): Add basic io by @godruoyi in https://github.com/apache/opendal/pull/2782 * feat: fuzz test support read from .env by different services by @dqhl76 in https://github.com/apache/opendal/pull/2824 * feat(services/rocksdb): Add scan support by @JLerxky in https://github.com/apache/opendal/pull/2827 * feat: Add postgresql support for OpenDAL by @Xuanwo in https://github.com/apache/opendal/pull/2815 * feat: ci for php binding by @godruoyi in https://github.com/apache/opendal/pull/2830 * feat: Add create_dir, remove, copy and rename API for oay-webdav by @Young-Flash in https://github.com/apache/opendal/pull/2832 * feat(oli): oli stat should show path as specified by users by @sarutak in https://github.com/apache/opendal/pull/2842 * feat(services/moka, services/mini-moka): Add scan support by @JLerxky in https://github.com/apache/opendal/pull/2850 * feat(oay): impl some method for `WebdavMetaData` by @Young-Flash in https://github.com/apache/opendal/pull/2857 * feat: Implement list with metakey for blocking by @Xuanwo in https://github.com/apache/opendal/pull/2861 * feat(services/redis): add redis cluster support by @G-XD in https://github.com/apache/opendal/pull/2858 * feat(services/dropbox): read support range by @suyanhanx in https://github.com/apache/opendal/pull/2848 * feat(layers/logging): Allow users to control print backtrace or not by @Xuanwo in https://github.com/apache/opendal/pull/2872 * feat: add native & full capability by @yah01 in https://github.com/apache/opendal/pull/2874 * feat: Implement RFC-2758 Merge Append Into Write by @Xuanwo in https://github.com/apache/opendal/pull/2880 * feat(binding/ocaml): Add support for operator reader and metadata by @Ranxy in https://github.com/apache/opendal/pull/2881 * feat(core): replace field `_pin` with `!Unpin` as argument by @morristai in https://github.com/apache/opendal/pull/2886 * feat: Add retry for Writer::sink operation by @Xuanwo in https://github.com/apache/opendal/pull/2896 * feat: remove operator range_read and range_reader API by @oowl in https://github.com/apache/opendal/pull/2898 * feat(core): Add unit test for ChunkedCursor by @Xuanwo in https://github.com/apache/opendal/pull/2907 * feat(types): remove blocking operation range_read and range_reader API by @oowl in https://github.com/apache/opendal/pull/2912 * feat(types): add stat_with API for blocking operator by @oowl in https://github.com/apache/opendal/pull/2915 * feat(services/gdrive): credential manage by @suyanhanx in https://github.com/apache/opendal/pull/2914 * feat(core): Implement Exact Buf Writer by @Xuanwo in https://github.com/apache/opendal/pull/2917 * feat: Add benchmark for buf write by @Xuanwo in https://github.com/apache/opendal/pull/2922 * feat(core/raw): Add stream support for multipart by @Xuanwo in https://github.com/apache/opendal/pull/2923 * feat(types): synchronous blocking operator and operator's API by @oowl in https://github.com/apache/opendal/pull/2924 * feat(bindings/java): bundled services by @tisonkun in https://github.com/apache/opendal/pull/2934 * feat(core/raw): support stream body for mixedpart by @silver-ymz in https://github.com/apache/opendal/pull/2936 * feat(bindings/python): expose presign api by @silver-ymz in https://github.com/apache/opendal/pull/2950 * feat(bindings/nodejs): Implement presign test by @suyanhanx in https://github.com/apache/opendal/pull/2969 * docs(services/gdrive): update service doc by @suyanhanx in https://github.com/apache/opendal/pull/2973 * feat(bindings/cpp): init cpp binding by @silver-ymz in https://github.com/apache/opendal/pull/2980 * feat: gcs insert object support cache control by @fatelei in https://github.com/apache/opendal/pull/2974 * feat(bindings/cpp): expose all api returned by value by @silver-ymz in https://github.com/apache/opendal/pull/3001 * feat(services/gdrive): implement rename by @suyanhanx in https://github.com/apache/opendal/pull/3007 * feat(bindings/cpp): expose reader by @silver-ymz in https://github.com/apache/opendal/pull/3004 * feat(bindings/cpp): expose lister by @silver-ymz in https://github.com/apache/opendal/pull/3011 * feat(core): Avoid copy if input is larger than buffer_size by @Xuanwo in https://github.com/apache/opendal/pull/3016 * feat(service/gdrive): add gdrive list support by @Young-Flash in https://github.com/apache/opendal/pull/3025 * feat(services/etcd): Enable etcd connection pool by @Xuanwo in https://github.com/apache/opendal/pull/3041 * feat: Add buffer support for all services by @Xuanwo in https://github.com/apache/opendal/pull/3045 * feat(bindings/java): auto enable blocking layer by @tisonkun in https://github.com/apache/opendal/pull/3049 * feat(bindings/java): support presign ops by @tisonkun in https://github.com/apache/opendal/pull/3069 * feat(services/azblob): Rewrite the method signatures using OpWrite by @acehinnnqru in https://github.com/apache/opendal/pull/3068 * feat(services/cos): Rewrite the method signatures using OpWrite by @acehinnnqru in https://github.com/apache/opendal/pull/3070 * feat(services/obs): Rewrite method signatures using OpWrite by @hanxuanliang in https://github.com/apache/opendal/pull/3075 * feat(services/cos): Rewrite the methods signature using OpStat/OpRead by @acehinnnqru in https://github.com/apache/opendal/pull/3073 * feat: Add AtomicServer Support for OpenDAL by @ArmandoZ in https://github.com/apache/opendal/pull/2878 * feat(services/onedrive): Rewrite the method signatures using OpWrite by @acehinnnqru in https://github.com/apache/opendal/pull/3091 * feat(services/azblob): Rewrite azblob methods signature using OpRead/OpStat by @acehinnnqru in https://github.com/apache/opendal/pull/3072 * feat(services/obs): Rewrite methods signature in obs using OpRead/OpStat by @hanxuanliang in https://github.com/apache/opendal/pull/3094 * feat(service/gdrive): add gdrive copy by @Young-Flash in https://github.com/apache/opendal/pull/3098 * feat(services/wasabi): Rewrite the method signatures using OpRead,OpW… by @acehinnnqru in https://github.com/apache/opendal/pull/3099 ### Changed * refactor(bindings/haskell): unify ffi of creating operator by @silver-ymz in https://github.com/apache/opendal/pull/2778 * refactor: Remove optimize in into_seekable_read_by_range by @Xuanwo in https://github.com/apache/opendal/pull/2796 * refactor(bindings/ocaml): Refactor module to support documentation by @Ranxy in https://github.com/apache/opendal/pull/2794 * refactor: Implement backtrace for Error correctly by @Xuanwo in https://github.com/apache/opendal/pull/2871 * refactor: Move object_store_opendal to integrations by @Xuanwo in https://github.com/apache/opendal/pull/2888 * refactor(services/gdrive): prepare for CI by @suyanhanx in https://github.com/apache/opendal/pull/2892 * refactor(core): Split buffer logic from underlying storage operations by @Xuanwo in https://github.com/apache/opendal/pull/2903 * refactor(service/webdav): Add docker-compose file to simplify the CI by @dqhl76 in https://github.com/apache/opendal/pull/2873 * refactor(raw): Return written bytes in oio::Write by @Xuanwo in https://github.com/apache/opendal/pull/3005 * refactor: Refactor oio::Write by accepting oio::Reader instead by @Xuanwo in https://github.com/apache/opendal/pull/3008 * refactor(core): Rename confusing pipe into copy_from by @Xuanwo in https://github.com/apache/opendal/pull/3015 * refactor: Remove oio::Write::copy_from by @Xuanwo in https://github.com/apache/opendal/pull/3018 * refactor: Make oio::Write accept Buf instead by @Xuanwo in https://github.com/apache/opendal/pull/3021 * refactor: Relax bounds on Writer::{sink, copy} by @huonw in https://github.com/apache/opendal/pull/3027 * refactor: Refactor oio::Write into poll-based to create more room for optimization by @Xuanwo in https://github.com/apache/opendal/pull/3029 * refactor: Polish multipart writer to allow oneshot optimization by @Xuanwo in https://github.com/apache/opendal/pull/3031 * refactor: Polish implementation details of WriteBuf and add vector chunks support by @Xuanwo in https://github.com/apache/opendal/pull/3034 * refactor: Add ChunkedBytes to improve the exact buf write by @Xuanwo in https://github.com/apache/opendal/pull/3035 * refactor: Polish RangeWrite implementation to remove the extra buffer logic by @Xuanwo in https://github.com/apache/opendal/pull/3038 * refactor: Remove the requirement of passing `content_length` to writer by @Xuanwo in https://github.com/apache/opendal/pull/3044 * refactor(services/azblob): instead `parse_batch_delete_response` with `Multipart::parse` by @G-XD in https://github.com/apache/opendal/pull/3071 * refactor(services/webdav): Refactor `webdav_put` signatures by using `OpWrite`. by @laipz8200 in https://github.com/apache/opendal/pull/3076 * refactor(services/azdls): Use OpWrite instead of passing all args as parameters by @liul85 in https://github.com/apache/opendal/pull/3077 * refactor(services/webdav): Use OpRead in `webdav_get`. by @laipz8200 in https://github.com/apache/opendal/pull/3081 * refactor(services/oss): Refactor `oss_put_object` signatures by using OpWrite by @sysu-yunz in https://github.com/apache/opendal/pull/3080 * refactor(services/http): Rewrite `http` methods signature by using OpRead/OpStat by @miroim in https://github.com/apache/opendal/pull/3083 * refactor(services/gcs): Rewrite `gcs` methods signature by using OpXxxx by @wavty in https://github.com/apache/opendal/pull/3087 * refactor: move all `fixtures` from `core/src/services/{service}` to top-level `fixtures/{service}` by @G-XD in https://github.com/apache/opendal/pull/3088 * refactor(services/webhdfs): Rewrite `webhdfs` methods signature by using `OpXxxx` by @cxorm in https://github.com/apache/opendal/pull/3109 ### Fixed * fix(docs): KEYS broken link by @suyanhanx in https://github.com/apache/opendal/pull/2749 * fix: scheme from_str missing redb and tikv by @Ranxy in https://github.com/apache/opendal/pull/2766 * fix(ci): pin zig version to 0.11.0 by @oowl in https://github.com/apache/opendal/pull/2772 * fix: fix compile error by low version of backon in old project by @silver-ymz in https://github.com/apache/opendal/pull/2781 * fix: Bump openssh-sftp-client from 0.13.5 to 0.13.7 by @yah01 in https://github.com/apache/opendal/pull/2797 * fix: add redis for nextcloud to solve file locking problem by @dqhl76 in https://github.com/apache/opendal/pull/2805 * fix: Fix behavior tests for blocking layer by @Xuanwo in https://github.com/apache/opendal/pull/2809 * fix(services/s3): remove default region `us-east-1` for non-aws s3 by @G-XD in https://github.com/apache/opendal/pull/2812 * fix(oli): Fix a test name in ls.rs by @sarutak in https://github.com/apache/opendal/pull/2817 * fix(oli, doc): Fix examples of config.toml for oli by @sarutak in https://github.com/apache/opendal/pull/2819 * fix: Cleanup temporary files generated in tests automatically by @sarutak in https://github.com/apache/opendal/pull/2823 * fix(services/rocksdb): Make sure return key starts with input path by @Xuanwo in https://github.com/apache/opendal/pull/2828 * fix(services/sftp): bump openssh-sftp-client to 0.13.9 by @silver-ymz in https://github.com/apache/opendal/pull/2831 * fix(oli): oli commands don't work properly for files in CWD by @sarutak in https://github.com/apache/opendal/pull/2833 * fix(oli): oli commands should not accept invalid URI format by @sarutak in https://github.com/apache/opendal/pull/2845 * fix(bindings/c): Fix an example of the C binding by @sarutak in https://github.com/apache/opendal/pull/2854 * fix(doc): Update instructions for building the C binding in README.md by @sarutak in https://github.com/apache/opendal/pull/2856 * fix(oay): add some error handle by @Young-Flash in https://github.com/apache/opendal/pull/2879 * fix: Set default timeouts for HttpClient by @sarutak in https://github.com/apache/opendal/pull/2895 * fix(website): broken edit link by @suyanhanx in https://github.com/apache/opendal/pull/2913 * fix(binding/java): Overwrite default NOTICE file with correct years by @tisonkun in https://github.com/apache/opendal/pull/2918 * fix(services/gcs): migrate to new multipart impl for gcs_insert_object_request by @silver-ymz in https://github.com/apache/opendal/pull/2838 * fix(core): Invalid lister should not panic nor endless loop by @Xuanwo in https://github.com/apache/opendal/pull/2931 * fix: Enable exact_buf_write for R2 by @Xuanwo in https://github.com/apache/opendal/pull/2935 * fix(services/s3): allow 404 resp when deleting a non-existing object by @gongyisheng in https://github.com/apache/opendal/pull/2941 * fix(doc): use crate::docs::rfc to replace relative path in doc by @gongyisheng in https://github.com/apache/opendal/pull/2942 * fix: S3 copy error on non-ascii file path by @BoWuGit in https://github.com/apache/opendal/pull/2909 * fix: copy error on non-ascii file path for cos/obs/wasabi services by @BoWuGit in https://github.com/apache/opendal/pull/2948 * fix(doc): add GCS api reference and known issues to service/s3 doc by @gongyisheng in https://github.com/apache/opendal/pull/2949 * fix(oay): pass litmus copymove test by @Young-Flash in https://github.com/apache/opendal/pull/2944 * fix(core): Make sure OpenDAL works with http2 on GCS by @Xuanwo in https://github.com/apache/opendal/pull/2956 * fix(nodejs|java): Add place holder for BDD test by @Xuanwo in https://github.com/apache/opendal/pull/2962 * fix(core): Fix capability of services is not set correctly by @Xuanwo in https://github.com/apache/opendal/pull/2968 * fix(core): Fix capability of services is not set correctly by @JLerxky in https://github.com/apache/opendal/pull/2982 * fix(services/gcs): Fix handling of media and multipart insert by @Xuanwo in https://github.com/apache/opendal/pull/2997 * fix(services/webdav): decode path before set Entry by @G-XD in https://github.com/apache/opendal/pull/3020 * fix(services/oss): set content_md5 in lister by @G-XD in https://github.com/apache/opendal/pull/3043 * fix: Correct the name of azdfs to azdls by @Xuanwo in https://github.com/apache/opendal/pull/3046 * fix: Don't apply blocking layer when service support blocking by @Xuanwo in https://github.com/apache/opendal/pull/3050 * fix: call `flush` before `sync_all` by @WenyXu in https://github.com/apache/opendal/pull/3053 * fix: Metakeys are not propagated with the blocking operators by @Xuanwo in https://github.com/apache/opendal/pull/3116 ### Docs * doc: fix released doc minor error by @oowl in https://github.com/apache/opendal/pull/2737 * docs: create README.md for oli by @STRRL in https://github.com/apache/opendal/pull/2752 * docs: polish fuzz README by @dqhl76 in https://github.com/apache/opendal/pull/2777 * docs: Add an example for PostgreSQL service by @sarutak in https://github.com/apache/opendal/pull/2847 * docs: improve php binding documentation by @godruoyi in https://github.com/apache/opendal/pull/2843 * docs: Fix missing link for rust example by @sarutak in https://github.com/apache/opendal/pull/2866 * docs: Add blog on how opendal read data by @Xuanwo in https://github.com/apache/opendal/pull/2869 * docs: Fix missing link to the contribution guide for the Node.js binding by @sarutak in https://github.com/apache/opendal/pull/2876 * doc: add 0.39.0 release link to download.md by @oowl in https://github.com/apache/opendal/pull/2882 * doc: add missing release step by @oowl in https://github.com/apache/opendal/pull/2883 * docs: add new committer landing doc by @dqhl76 in https://github.com/apache/opendal/pull/2905 * docs: auto release maven artifacts by @tisonkun in https://github.com/apache/opendal/pull/2729 * doc(tests): fix test command by @G-XD in https://github.com/apache/opendal/pull/2920 * docs: add service doc for gcs by @silver-ymz in https://github.com/apache/opendal/pull/2930 * docs(services/gcs): fix rust core doc include by @suyanhanx in https://github.com/apache/opendal/pull/2932 * docs: migrate all existed service documents by @silver-ymz in https://github.com/apache/opendal/pull/2937 * docs: Fix incorrect links to rfcs by @Xuanwo in https://github.com/apache/opendal/pull/2943 * docs: Update Release Process by @Xuanwo in https://github.com/apache/opendal/pull/2964 * docs(services/sftp): update comments about windows support and password login support by @silver-ymz in https://github.com/apache/opendal/pull/2967 * docs: add service doc for etcd & dropbox & foundationdb & moka by @G-XD in https://github.com/apache/opendal/pull/2986 * docs(bindings/cpp): add CONTRIBUTING.md by @silver-ymz in https://github.com/apache/opendal/pull/2984 * docs(bindings/cpp): use doxygen to generate API docs by @silver-ymz in https://github.com/apache/opendal/pull/2988 * docs(bindings/c): add awesome-doxygen to beautify document by @silver-ymz in https://github.com/apache/opendal/pull/2999 * docs(contributing): add podling status report guide by @PsiACE in https://github.com/apache/opendal/pull/2996 * docs: fix spelling - change `Github` to `GitHub` by @jbampton in https://github.com/apache/opendal/pull/3012 * docs: fix spelling - change `MacOS` to `macOS` by @jbampton in https://github.com/apache/opendal/pull/3013 * docs: add service doc for gdrive & onedrive by @nasnoisaac in https://github.com/apache/opendal/pull/3028 * docs(services/sftp): update comments about password login by @silver-ymz in https://github.com/apache/opendal/pull/3065 * docs: Add OwO 1st by @Xuanwo in https://github.com/apache/opendal/pull/3086 * docs: Add upgrade note for v0.40 by @Xuanwo in https://github.com/apache/opendal/pull/3096 * docs: add basic example for cpp binding by @silver-ymz in https://github.com/apache/opendal/pull/3108 * docs: Add comments for blocking layer by @Xuanwo in https://github.com/apache/opendal/pull/3117 ### CI * build(deps): bump serde_json from 1.0.99 to 1.0.104 by @dependabot in https://github.com/apache/opendal/pull/2746 * build(deps): bump tracing-opentelemetry from 0.17.4 to 0.19.0 by @dependabot in https://github.com/apache/opendal/pull/2744 * build(deps): bump paste from 1.0.13 to 1.0.14 by @dependabot in https://github.com/apache/opendal/pull/2742 * build(deps): bump opentelemetry from 0.19.0 to 0.20.0 by @dependabot in https://github.com/apache/opendal/pull/2743 * build(deps): bump object_store from 0.5.6 to 0.6.1 by @dependabot in https://github.com/apache/opendal/pull/2745 * ci: use cache to speed up haskell ci by @silver-ymz in https://github.com/apache/opendal/pull/2792 * ci: Add setup for php and ocaml in dev container by @Xuanwo in https://github.com/apache/opendal/pull/2825 * ci: Trying to fix rocksdb build by @Xuanwo in https://github.com/apache/opendal/pull/2867 * ci: add reproducibility check by @tisonkun in https://github.com/apache/opendal/pull/2863 * ci(services/postgresql): add docker-compose to simplify the CI by @G-XD in https://github.com/apache/opendal/pull/2877 * ci(service/s3): Add docker-compose-minio file to simplify the CI by @gongyisheng in https://github.com/apache/opendal/pull/2887 * ci(services/hdfs): Load native lib instead by @Xuanwo in https://github.com/apache/opendal/pull/2900 * ci(services/rocksdb): Make sure rocksdb lib is loaded by @Xuanwo in https://github.com/apache/opendal/pull/2902 * build(bindings/java): bundle bare binaries in JARs with classifier by @tisonkun in https://github.com/apache/opendal/pull/2910 * ci(bindings/java): enable auto staging JARs on Apache Nexus repository by @tisonkun in https://github.com/apache/opendal/pull/2939 * ci(fix): Add PORTABLE to make sure rocksdb compiled with the same CPU feature set by @gongyisheng in https://github.com/apache/opendal/pull/2976 * ci(oay): Polish oay webdav test by @Young-Flash in https://github.com/apache/opendal/pull/2971 * build(deps): bump cbindgen from 0.24.5 to 0.25.0 by @dependabot in https://github.com/apache/opendal/pull/2992 * build(deps): bump actions/checkout from 2 to 3 by @dependabot in https://github.com/apache/opendal/pull/2995 * build(deps): bump pin-project from 1.1.2 to 1.1.3 by @dependabot in https://github.com/apache/opendal/pull/2993 * build(deps): bump chrono from 0.4.26 to 0.4.28 by @dependabot in https://github.com/apache/opendal/pull/2989 * build(deps): bump redb from 1.0.4 to 1.1.0 by @dependabot in https://github.com/apache/opendal/pull/2991 * build(deps): bump lazy-regex from 2.5.0 to 3.0.1 by @dependabot in https://github.com/apache/opendal/pull/2990 * build(deps): bump korandoru/hawkeye from 3.1.0 to 3.3.0 by @dependabot in https://github.com/apache/opendal/pull/2994 * ci(bindings/cpp): add ci for test and doc by @silver-ymz in https://github.com/apache/opendal/pull/2998 * ci(services/tikv): add tikv integration test with tls by @G-XD in https://github.com/apache/opendal/pull/3026 * ci: restrict workflow that need password by @dqhl76 in https://github.com/apache/opendal/pull/3039 * ci: Don't release while tag contains rc by @Xuanwo in https://github.com/apache/opendal/pull/3048 * ci(bindings/java): skip RedisServiceTest on macos and windows by @tisonkun in https://github.com/apache/opendal/pull/3054 * ci: Disable PHP build temporarily by @Xuanwo in https://github.com/apache/opendal/pull/3058 * ci(bindings/java): release workflow always uses bash by @tisonkun in https://github.com/apache/opendal/pull/3056 * ci(binding/java): Enable release build only when releasing by @Xuanwo in https://github.com/apache/opendal/pull/3057 * ci(binding/java): Use cargo profile instead of --release by @Xuanwo in https://github.com/apache/opendal/pull/3059 * ci: Move platform build checks from java binding to rust core by @Xuanwo in https://github.com/apache/opendal/pull/3060 * ci(bindings/haskell): add release workflow by @silver-ymz in https://github.com/apache/opendal/pull/3082 * ci: Build rc but don't publish by @Xuanwo in https://github.com/apache/opendal/pull/3089 * ci: Don't verify content for dry run by @Xuanwo in https://github.com/apache/opendal/pull/3115 ### Chore * chore(core): bump cargo.toml http version to 0.2.9 by @oowl in https://github.com/apache/opendal/pull/2740 * chore: do not export example directory by @oowl in https://github.com/apache/opendal/pull/2750 * chore: Fix build after merging of ocaml by @Xuanwo in https://github.com/apache/opendal/pull/2776 * chore: Bump bytes to 1.4 to allow the usage of spare_capacity_mut by @Xuanwo in https://github.com/apache/opendal/pull/2784 * chore: disable oldtime feature of chrono by @paolobarbolini in https://github.com/apache/opendal/pull/2793 * chore: Disable blocking layer until we make all services passed by @Xuanwo in https://github.com/apache/opendal/pull/2806 * chore(bindings/haskell): post release 0.1.0 by @silver-ymz in https://github.com/apache/opendal/pull/2814 * chore(bindings/ocaml): Add contributing document to readme by @Ranxy in https://github.com/apache/opendal/pull/2829 * chore: Make clippy happy by @Xuanwo in https://github.com/apache/opendal/pull/2851 * chore: add health check for docker-compose minio by @oowl in https://github.com/apache/opendal/pull/2899 * chore(ci): offload healthcheck logic to docker-compose config by @oowl in https://github.com/apache/opendal/pull/2901 * chore: Make clippy happy by @Xuanwo in https://github.com/apache/opendal/pull/2927 * chore: Make C Binding clippy happy by @Xuanwo in https://github.com/apache/opendal/pull/2928 * chore: Fix failed ci by @silver-ymz in https://github.com/apache/opendal/pull/2938 * chore(ci): remove unreviewable test file and add generate test file step before testing by @gongyisheng in https://github.com/apache/opendal/pull/3003 * chore(bindings/cpp): update CMakeLists.txt to prepare release by @silver-ymz in https://github.com/apache/opendal/pull/3030 * chore: fix typo of SftpWriter error message by @silver-ymz in https://github.com/apache/opendal/pull/3032 * chore: Polish some details of layers implementation by @Xuanwo in https://github.com/apache/opendal/pull/3061 * chore(bindings/haskell): make cargo build type same with cabal by @silver-ymz in https://github.com/apache/opendal/pull/3067 * chore(bindings/haskell): add PVP-compliant version bounds by @silver-ymz in https://github.com/apache/opendal/pull/3093 * chore(bindings/java): align ErrorKind with exception code by @tisonkun in https://github.com/apache/opendal/pull/3095 * chore: Bump version to v0.40 to start release process by @Xuanwo in https://github.com/apache/opendal/pull/3101 * chore(bindings/haskell): rename library name from opendal-hs to opendal by @silver-ymz in https://github.com/apache/opendal/pull/3112 ## [v0.39.0] - 2023-07-31 ### Added * feat: add a behaviour test for InvalidInput by @dqhl76 in https://github.com/apache/opendal/pull/2644 * feat(services/persy): add a basic persy service impl by @PsiACE in https://github.com/apache/opendal/pull/2648 * feat(services/vercel_artifacts): Impl `stat` by @suyanhanx in https://github.com/apache/opendal/pull/2649 * feat(test): add fuzz test for range_reader by @dqhl76 in https://github.com/apache/opendal/pull/2609 * feat(core/http_util): Remove sensitive header like Set-Cookie by @Xuanwo in https://github.com/apache/opendal/pull/2664 * feat: Add RetryInterceptor support for RetryLayer by @Xuanwo in https://github.com/apache/opendal/pull/2666 * feat: support kerberos for hdfs service by @zuston in https://github.com/apache/opendal/pull/2668 * feat: support append for hdfs by @zuston in https://github.com/apache/opendal/pull/2671 * feat(s3): Use us-east-1 while head bucket returns 403 without X-Amz-Bucket-Region by @john8628 in https://github.com/apache/opendal/pull/2677 * feat(oay): Add webdav basic read impl by @Young-Flash in https://github.com/apache/opendal/pull/2658 * feat(services/redis): enable TLS by @Stormshield-robinc in https://github.com/apache/opendal/pull/2670 * feat(services/etcd): introduce new service backend etcd by @G-XD in https://github.com/apache/opendal/pull/2672 * feat(service/obs):add multipart upload function support by @A-Stupid-Sun in https://github.com/apache/opendal/pull/2685 * feat(services/s3): Add assume role support by @Xuanwo in https://github.com/apache/opendal/pull/2687 * feat(services/tikv): introduce new service backend tikv by @oowl in https://github.com/apache/opendal/pull/2565 * feat(service/cos): add multipart upload function support by @ArmandoZ in https://github.com/apache/opendal/pull/2697 * feat(oio): Add MultipartUploadWrite to easier the work for Writer by @Xuanwo in https://github.com/apache/opendal/pull/2699 * feat(test): add fuzz target for writer by @dqhl76 in https://github.com/apache/opendal/pull/2706 * feat: cos multipart uploads write by @parkma99 in https://github.com/apache/opendal/pull/2712 * feat(layers): support await_tree instrument by @oowl in https://github.com/apache/opendal/pull/2623 * feat(tests): Extract fuzz test of #2717 by @Xuanwo in https://github.com/apache/opendal/pull/2720 * feat: oss multipart uploads write by @parkma99 in https://github.com/apache/opendal/pull/2723 * feat: add override_content_type by @G-XD in https://github.com/apache/opendal/pull/2734 ### Changed * refactor(services/redis): Polish features of redis by @Xuanwo in https://github.com/apache/opendal/pull/2681 * refactor(services/s3): Check header first for region detect by @Xuanwo in https://github.com/apache/opendal/pull/2691 * refactor(raw/oio): Reorganize to allow adding more features by @Xuanwo in https://github.com/apache/opendal/pull/2698 * refactor: Polish fuzz build time by @Xuanwo in https://github.com/apache/opendal/pull/2721 ### Fixed * fix(services/cos): fix cos service comments by @A-Stupid-Sun in https://github.com/apache/opendal/pull/2656 * fix(test): profile setting warning by @dqhl76 in https://github.com/apache/opendal/pull/2657 * fix(bindings/C): fix the memory found in valgrind. by @Ji-Xinyou in https://github.com/apache/opendal/pull/2673 * fix: owncloud test sometimes fail by @dqhl76 in https://github.com/apache/opendal/pull/2684 * fix(services/obs): remove content-length check in backend by @suyanhanx in https://github.com/apache/opendal/pull/2686 * fix: fix `HADOOP_CONF_DIR` setting in guidance document by @wcy-fdu in https://github.com/apache/opendal/pull/2713 * fix: Seek before the start of file should be invalid by @Xuanwo in https://github.com/apache/opendal/pull/2718 * fix(layer/minitrace): fix doctest by @andylokandy in https://github.com/apache/opendal/pull/2728 ### Docs * docs: add instructions to fix wrong vote mail and uploads by @ClSlaid in https://github.com/apache/opendal/pull/2682 * doc(services/tikv): add tikv service backend to readme by @oowl in https://github.com/apache/opendal/pull/2711 * docs(bindings/java): improve safety doc for get_current_env by @tisonkun in https://github.com/apache/opendal/pull/2733 ### CI * ci(services/webdav): Setup integration test for owncloud by @dqhl76 in https://github.com/apache/opendal/pull/2659 * ci: Fix unexpected error in owncloud by @Xuanwo in https://github.com/apache/opendal/pull/2663 * ci: upgrade hawkeye action by @tisonkun in https://github.com/apache/opendal/pull/2665 * ci: Make owncloud happy by reduce the concurrency by @Xuanwo in https://github.com/apache/opendal/pull/2667 * ci: Setup protoc in rust builder by @Xuanwo in https://github.com/apache/opendal/pull/2674 * ci: Fix Cargo.lock not updated by @Xuanwo in https://github.com/apache/opendal/pull/2680 * ci: Add services fuzz test for read/write/range_read by @dqhl76 in https://github.com/apache/opendal/pull/2710 ### Chore * chore: Update CODEOWNERS by @Xuanwo in https://github.com/apache/opendal/pull/2676 * chore(bindings/python): upgrade pyo3 to 0.19 by @messense in https://github.com/apache/opendal/pull/2694 * chore: upgrade quick-xml to 0.29 by @messense in https://github.com/apache/opendal/pull/2696 * chore(download): update version 0.38.1 by @suyanhanx in https://github.com/apache/opendal/pull/2714 * chore(service/minitrace): update to v0.5.0 by @andylokandy in https://github.com/apache/opendal/pull/2725 ## [v0.38.1] - 2023-07-14 ### Added - feat(binding/lua): add rename and create_dir operator function by @oowl in https://github.com/apache/opendal/pull/2564 - feat(services/azblob): support sink by @suyanhanx in https://github.com/apache/opendal/pull/2574 - feat(services/gcs): support sink by @suyanhanx in https://github.com/apache/opendal/pull/2576 - feat(services/oss): support sink by @suyanhanx in https://github.com/apache/opendal/pull/2577 - feat(services/obs): support sink by @suyanhanx in https://github.com/apache/opendal/pull/2578 - feat(services/cos): impl sink by @suyanhanx in https://github.com/apache/opendal/pull/2587 - feat(service): Support stat for Dropbox by @Zheaoli in https://github.com/apache/opendal/pull/2588 - feat(services/dropbox): impl create_dir and polish error handling by @suyanhanx in https://github.com/apache/opendal/pull/2600 - feat(services/dropbox): Implement refresh token support by @Xuanwo in https://github.com/apache/opendal/pull/2604 - feat(service/dropbox): impl batch delete by @suyanhanx in https://github.com/apache/opendal/pull/2606 - feat(CI): set Kvrocks test for service redis by @suyanhanx in https://github.com/apache/opendal/pull/2613 - feat(core): object versioning APIs by @suyanhanx in https://github.com/apache/opendal/pull/2614 - feat(oay): actually read configuration from `oay.toml` by @messense in https://github.com/apache/opendal/pull/2615 - feat(services/webdav): impl sink by @suyanhanx in https://github.com/apache/opendal/pull/2622 - feat(services/fs): impl Sink for Fs by @Ji-Xinyou in https://github.com/apache/opendal/pull/2626 - feat(core): impl `delete_with` on blocking operator by @suyanhanx in https://github.com/apache/opendal/pull/2633 - feat(bindings/C): add support for list in C binding by @Ji-Xinyou in https://github.com/apache/opendal/pull/2448 - feat(services/s3): Add detect_region support for S3Builder by @parkma99 in https://github.com/apache/opendal/pull/2634 ### Changed - refactor(core): Add ErrorKind InvalidInput to indicate users input error by @dqhl76 in https://github.com/apache/opendal/pull/2637 - refactor(services/s3): Add more detect logic for detect_region by @Xuanwo in https://github.com/apache/opendal/pull/2645 ### Fixed - fix(doc): fix codeblock rendering by @xxchan in https://github.com/apache/opendal/pull/2592 - fix(service/minitrace): should set local parent by @andylokandy in https://github.com/apache/opendal/pull/2620 - fix(service/minitrace): update doc by @andylokandy in https://github.com/apache/opendal/pull/2621 ### Docs - doc(bindings/haskell): add module document by @silver-ymz in https://github.com/apache/opendal/pull/2566 - docs: Update license related comments by @Prashanth-Chandra in https://github.com/apache/opendal/pull/2573 - docs: add hdfs namenode High Availability related troubleshoot by @wcy-fdu in https://github.com/apache/opendal/pull/2601 - docs: polish release doc by @PsiACE in https://github.com/apache/opendal/pull/2608 - docs(blog): add Apache OpenDAL(Incubating): Access Data Freely by @PsiACE in https://github.com/apache/opendal/pull/2607 - docs(RFC): Object Versioning by @suyanhanx in https://github.com/apache/opendal/pull/2602 ### CI - ci: Disable bindings/java deploy for now by @tisonkun in https://github.com/apache/opendal/pull/2560 - ci: Disable the failed stage-release job instead by @tisonkun in https://github.com/apache/opendal/pull/2561 - ci: add haddock generator for haskell binding by @silver-ymz in https://github.com/apache/opendal/pull/2569 - ci(binding/lua): add luarocks package manager support by @oowl in https://github.com/apache/opendal/pull/2558 - build(deps): bump predicates from 2.1.5 to 3.0.1 by @dependabot in https://github.com/apache/opendal/pull/2583 - build(deps): bump tower-http from 0.4.0 to 0.4.1 by @dependabot in https://github.com/apache/opendal/pull/2582 - build(deps): bump chrono from 0.4.24 to 0.4.26 by @dependabot in https://github.com/apache/opendal/pull/2581 - build(deps): bump redis from 0.22.3 to 0.23.0 by @dependabot in https://github.com/apache/opendal/pull/2580 - build(deps): bump cbindgen from 0.24.3 to 0.24.5 by @dependabot in https://github.com/apache/opendal/pull/2579 - ci: upgrade hawkeye to v3 by @tisonkun in https://github.com/apache/opendal/pull/2585 - ci(services/webdav): Setup integration test for nextcloud by @Xuanwo in https://github.com/apache/opendal/pull/2631 ### Chore - chore: add haskell binding link to website by @silver-ymz in https://github.com/apache/opendal/pull/2571 - chore: fix cargo warning for resolver by @xxchan in https://github.com/apache/opendal/pull/2590 - chore: bump log to 0.4.19 by @xxchan in https://github.com/apache/opendal/pull/2591 - chore(deps): update deps to latest version by @suyanhanx in https://github.com/apache/opendal/pull/2596 - chore: Add release 0.38.0 to download by @PsiACE in https://github.com/apache/opendal/pull/2597 - chore(service/minitrace): automatically generate span name by @andylokandy in https://github.com/apache/opendal/pull/2618 ## New Contributors - @Prashanth-Chandra made their first contribution in https://github.com/apache/opendal/pull/2573 - @andylokandy made their first contribution in https://github.com/apache/opendal/pull/2618 - @parkma99 made their first contribution in https://github.com/apache/opendal/pull/2634 ## [v0.38.0] - 2023-06-27 ### Added - feat(raw/http_util): Implement mixed multipart parser by @Xuanwo in https://github.com/apache/opendal/pull/2430 - feat(services/gcs): Add batch delete support by @wcy-fdu in https://github.com/apache/opendal/pull/2142 - feat(core): Add Write::sink API by @Xuanwo in https://github.com/apache/opendal/pull/2440 - feat(services/s3): Allow retry for unexpected 499 error by @Xuanwo in https://github.com/apache/opendal/pull/2453 - feat(layer): add throttle layer by @morristai in https://github.com/apache/opendal/pull/2444 - feat(bindings/haskell): init haskell binding by @silver-ymz in https://github.com/apache/opendal/pull/2463 - feat(core): add capability check by @unixzii in https://github.com/apache/opendal/pull/2461 - feat(bindings/haskell): add CONTRIBUTING.md by @silver-ymz in https://github.com/apache/opendal/pull/2466 - feat(bindings/haskell): add CI test for haskell binding by @silver-ymz in https://github.com/apache/opendal/pull/2468 - feat(binding/lua): introduce opendal lua binding by @oowl in https://github.com/apache/opendal/pull/2469 - feat(bindings/swift): add Swift binding by @unixzii in https://github.com/apache/opendal/pull/2470 - feat(bindings/haskell): support `is_exist` `create_dir` `copy` `rename` `delete` by @silver-ymz in https://github.com/apache/opendal/pull/2475 - feat(bindings/haskell): add `Monad` wrapper by @silver-ymz in https://github.com/apache/opendal/pull/2482 - feat(bindings/dotnet): basic structure by @tisonkun in https://github.com/apache/opendal/pull/2485 - feat(services/dropbox): Support create/read/delete for Dropbox by @Zheaoli in https://github.com/apache/opendal/pull/2264 - feat(bindings/java): support load system lib by @tisonkun in https://github.com/apache/opendal/pull/2502 - feat(blocking operator): add remove_all api by @infdahai in https://github.com/apache/opendal/pull/2449 - feat(core): adopt WebHDFS LISTSTATUS_BATCH for better performance by @morristai in https://github.com/apache/opendal/pull/2499 - feat(bindings/haskell): support stat by @silver-ymz in https://github.com/apache/opendal/pull/2504 - feat(adapters-kv): add rename and copy support to kv adapters by @oowl in https://github.com/apache/opendal/pull/2513 - feat: Implement sink for services s3 by @Xuanwo in https://github.com/apache/opendal/pull/2508 - feat(adapters-kv): add rename and copy support to non typed kv adapters by @oowl in https://github.com/apache/opendal/pull/2515 - feat: Implement test harness via libtest-mimic instead by @Xuanwo in https://github.com/apache/opendal/pull/2517 - feat(service/sled): introduce tree support by @oowl in https://github.com/apache/opendal/pull/2516 - feat(bindings/haskell): support list and scan by @silver-ymz in https://github.com/apache/opendal/pull/2527 - feat(services/redb): support redb service by @oowl in https://github.com/apache/opendal/pull/2526 - feat(core): implement service for Mini Moka by @morristai in https://github.com/apache/opendal/pull/2537 - feat(core): add Mini Moka GitHub Action workflow job by @morristai in https://github.com/apache/opendal/pull/2539 - feat(services): add cacache backend by @PsiACE in https://github.com/apache/opendal/pull/2548 - feat: Implement Writer::copy so user can copy from AsyncRead by @Xuanwo in https://github.com/apache/opendal/pull/2552 ### Changed - refactor(bindings/C): refactor c bindings to call all APIs using pointer by @Ji-Xinyou in https://github.com/apache/opendal/pull/2489 ### Fixed - fix(services/azblob): Fix azblob batch max operations by @A-Stupid-Sun in https://github.com/apache/opendal/pull/2434 - fix(services/sftp): change default root config to remote server setting by @silver-ymz in https://github.com/apache/opendal/pull/2431 - fix: Enable `std` feature for futures to allow `futures::AsyncRead` by @Xuanwo in https://github.com/apache/opendal/pull/2450 - fix(services/gcs): GCS should support create dir by @Xuanwo in https://github.com/apache/opendal/pull/2467 - fix(bindings/C): use copy_from_slice instead of from_static in opendal_bytes by @Ji-Xinyou in https://github.com/apache/opendal/pull/2473 - fix(bindings/swift): reorg the package to correct its name by @unixzii in https://github.com/apache/opendal/pull/2479 - fix: Fix the build for zig binding by @Xuanwo in https://github.com/apache/opendal/pull/2493 - fix(service/webhdfs): fix webhdfs config builder for disable_list_batch by @morristai in https://github.com/apache/opendal/pull/2509 - fix(core/types): add missing `vercel artifacts` for `FromStr` by @cijiugechu in https://github.com/apache/opendal/pull/2519 - fix(types/operator): fix operation limit error default size by @oowl in https://github.com/apache/opendal/pull/2536 ### Docs - docs: Replace `create` with `new` by @NiwakaDev in https://github.com/apache/opendal/pull/2427 - docs(services/redis): fix redis via config example by @A-Stupid-Sun in https://github.com/apache/opendal/pull/2443 - docs: add rust usage example by @Young-Flash in https://github.com/apache/opendal/pull/2447 - docs: Polish rust examples by @Xuanwo in https://github.com/apache/opendal/pull/2456 - docs: polish docs and fix typos by @suyanhanx in https://github.com/apache/opendal/pull/2458 - docs: fix a typo on the landing page by @unixzii in https://github.com/apache/opendal/pull/2460 - docs(examples/rust): Add 01-init-operator by @Xuanwo in https://github.com/apache/opendal/pull/2464 - docs: update readme.md to match the output by @rrain7 in https://github.com/apache/opendal/pull/2486 - docs: Update components for Libraries and Services by @Xuanwo in https://github.com/apache/opendal/pull/2487 - docs: Add OctoBase into our users list by @Xuanwo in https://github.com/apache/opendal/pull/2506 - docs: Fix scan not checked for sled services by @Xuanwo in https://github.com/apache/opendal/pull/2507 - doc(binding/lua): Improve readme doc for contribute and usage by @oowl in https://github.com/apache/opendal/pull/2511 - doc(services/redb): add doc for redb service backend by @oowl in https://github.com/apache/opendal/pull/2538 - doc(bindings/swift): add CONTRIBUTING.md by @unixzii in https://github.com/apache/opendal/pull/2540 - docs: Add new rust example 02-async-io by @Xuanwo in https://github.com/apache/opendal/pull/2541 - docs: Fix link for CONTRIBUTING.md by @HuSharp in https://github.com/apache/opendal/pull/2544 - doc: polish release doc by @suyanhanx in https://github.com/apache/opendal/pull/2531 - docs: Move verify to upper folder by @Xuanwo in https://github.com/apache/opendal/pull/2546 - doc(binding/lua): add ldoc generactor for lua binding by @oowl in https://github.com/apache/opendal/pull/2549 - docs: Add new architectural image for OpenDAL by @Xuanwo in https://github.com/apache/opendal/pull/2553 - docs: Polish README for core and bindings by @Xuanwo in https://github.com/apache/opendal/pull/2554 ### CI - ci: Fix append test should use copy_buf to avoid call times by @Xuanwo in https://github.com/apache/opendal/pull/2436 - build(bindings/ruby): fix compile rb-sys on Apple M1 by @tisonkun in https://github.com/apache/opendal/pull/2451 - ci: Use summary for zig test to fix build by @Xuanwo in https://github.com/apache/opendal/pull/2480 - ci(workflow): add lua binding test workflow by @oowl in https://github.com/apache/opendal/pull/2478 - build(deps): bump actions/setup-python from 3 to 4 by @dependabot in https://github.com/apache/opendal/pull/2481 - ci(bindings/swift): add CI for Swift binding by @unixzii in https://github.com/apache/opendal/pull/2492 - ci: Try to make webhdfs tests more stable by @Xuanwo in https://github.com/apache/opendal/pull/2503 - ci(bindings/java): auto release snapshot by @tisonkun in https://github.com/apache/opendal/pull/2521 - ci: Disable the stage snapshot CI by @Xuanwo in https://github.com/apache/opendal/pull/2528 - ci: fix opendal-java snapshot releases by @tisonkun in https://github.com/apache/opendal/pull/2532 - ci: Fix typo in binding java CI by @Xuanwo in https://github.com/apache/opendal/pull/2534 - ci(bindings/swift): optimize time consumption of CI pipeline by @unixzii in https://github.com/apache/opendal/pull/2545 - ci: Fix PR label not updated while edited by @Xuanwo in https://github.com/apache/opendal/pull/2547 ### Chore - chore: Add redis bench support by @Xuanwo in https://github.com/apache/opendal/pull/2438 - chore(bindings/nodejs): update index.d.ts by @suyanhanx in https://github.com/apache/opendal/pull/2459 - chore: Add release 0.37.0 to download by @suyanhanx in https://github.com/apache/opendal/pull/2472 - chore: Fix Cargo.lock not updated by @Xuanwo in https://github.com/apache/opendal/pull/2490 - chore: Polish some code details by @Xuanwo in https://github.com/apache/opendal/pull/2505 - chore(bindings/nodejs): provide more precise type for scheme by @cijiugechu in https://github.com/apache/opendal/pull/2520 ## [v0.37.0] - 2023-06-06 ### Added - feat(services/webdav): support redirection when get 302/307 response during read operation by @Yansongsongsong in https://github.com/apache/opendal/pull/2256 - feat: Add Zig Bindings Module by @kassane in https://github.com/apache/opendal/pull/2374 - feat: Implement Timeout Layer by @Xuanwo in https://github.com/apache/opendal/pull/2395 - feat(bindings/c): add opendal_operator_blocking_delete method by @jiaoew1991 in https://github.com/apache/opendal/pull/2416 - feat(services/obs): add append support by @infdahai in https://github.com/apache/opendal/pull/2422 ### Changed - refactor(bindings/zig): enable tests and more by @tisonkun in https://github.com/apache/opendal/pull/2375 - refactor(bindings/zig): add errors handler and module test by @kassane in https://github.com/apache/opendal/pull/2381 - refactor(http_util): Adopt reqwest's redirect support by @Xuanwo in https://github.com/apache/opendal/pull/2390 ### Fixed - fix(bindings/zig): reflect C interface changes by @tisonkun in https://github.com/apache/opendal/pull/2378 - fix(services/azblob): Fix batch delete doesn't work on azure by @Xuanwo in https://github.com/apache/opendal/pull/2382 - fix(services/oss): Fix oss batch max operations by @A-Stupid-Sun in https://github.com/apache/opendal/pull/2414 - fix(core): Don't wake up operator futures while not ready by @Xuanwo in https://github.com/apache/opendal/pull/2415 - fix(services/s3): Fix s3 batch max operations by @A-Stupid-Sun in https://github.com/apache/opendal/pull/2418 ### Docs - docs: service doc for s3 by @suyanhanx in https://github.com/apache/opendal/pull/2376 - docs(bindings/C): The documentation for OpenDAL C binding by @Ji-Xinyou in https://github.com/apache/opendal/pull/2373 - docs: add link for c binding by @suyanhanx in https://github.com/apache/opendal/pull/2380 - docs: docs for kv services by @suyanhanx in https://github.com/apache/opendal/pull/2396 - docs: docs for fs related services by @suyanhanx in https://github.com/apache/opendal/pull/2397 - docs(bindings/java): do not release snapshot versions anymore by @tisonkun in https://github.com/apache/opendal/pull/2398 - docs: doc for ipmfs by @suyanhanx in https://github.com/apache/opendal/pull/2408 - docs: add service doc for oss by @A-Stupid-Sun in https://github.com/apache/opendal/pull/2409 - docs: improvement of Python binding by @ideal in https://github.com/apache/opendal/pull/2411 - docs: doc for download by @suyanhanx in https://github.com/apache/opendal/pull/2424 - docs: Add release guide by @Xuanwo in https://github.com/apache/opendal/pull/2425 ### CI - ci: Enable semantic PRs by @Xuanwo in https://github.com/apache/opendal/pull/2370 - ci: improve licenserc settings by @tisonkun in https://github.com/apache/opendal/pull/2377 - build(deps): bump reqwest from 0.11.15 to 0.11.18 by @dependabot in https://github.com/apache/opendal/pull/2389 - build(deps): bump pyo3 from 0.18.2 to 0.18.3 by @dependabot in https://github.com/apache/opendal/pull/2388 - ci: Enable nextest for all behavior tests by @Xuanwo in https://github.com/apache/opendal/pull/2400 - ci: reflect ascii file rewrite by @tisonkun in https://github.com/apache/opendal/pull/2419 - ci: Remove website from git archive by @Xuanwo in https://github.com/apache/opendal/pull/2420 - ci: Add integration tests for Cloudflare R2 by @Xuanwo in https://github.com/apache/opendal/pull/2423 ### Chore - chore(bindings/python): upgrade maturin to 1.0 by @messense in https://github.com/apache/opendal/pull/2369 - chore: Fix license headers for release/labler by @Xuanwo in https://github.com/apache/opendal/pull/2371 - chore(bindings/C): add one simple read/write example into readme and code by @Ji-Xinyou in https://github.com/apache/opendal/pull/2421 ## [v0.36.0] - 2023-05-30 ### Added - feat(service/fs): add append support for fs (#2296) - feat(services/sftp): add append support for sftp (#2297) - RFC-2299: Chain based Operator API (#2299) - feat(services/azblob): add append support (#2302) - feat(bindings/nodejs): add append support (#2322) - feat(bindings/C): opendal_operator_ptr construction using kvs (#2329) - feat(services/cos): append support (#2332) - feat(bindings/java): implement Operator#delete (#2345) - feat(bindings/java): support append (#2350) - feat(bindings/java): save one jni call in the hot path (#2353) - feat: server side encryption support for azblob (#2347) ### Changed - refactor(core): Implement RFC-2299 for stat_with (#2303) - refactor(core): Implement RFC-2299 for BlockingOperator::write_with (#2305) - refactor(core): Implement RFC-2299 for appender_with (#2307) - refactor(core): Implement RFC-2299 for read_with (#2308) - refactor(core): Implement RFC-2299 for read_with (#2308) - refactor(core): Implement RFC-2299 for append_with (#2312) - refactor(core): Implement RFC-2299 for write_with (#2315) - refactor(core): Implement RFC-2299 for reader_with (#2316) - refactor(core): Implement RFC-2299 for writer_with (#2317) - refactor(core): Implement RFC-2299 for presign_read_with (#2314) - refactor(core): Implement RFC-2299 for presign_write_with (#2320) - refactor(core): Implement RFC-2299 for list_with (#2323) - refactor: Move `ops` to `raw::ops` (#2325) - refactor(bindings/C): align bdd test with the feature tests (#2340) - refactor(bindings/java): narrow unsafe boundary (#2351) ### Fixed - fix(services/supabase): correctly set retryable (#2295) - fix(core): appender complete check (#2298) ### Docs - docs: add service doc for azdfs (#2310) - docs(bidnings/java): how to deploy snapshots (#2311) - docs(bidnings/java): how to deploy snapshots (#2311) - docs: Fixed links of languages to open in same tab (#2327) - docs: Adopt docusaurus pathname protocol (#2330) - docs(bindings/nodejs): update lib desc (#2331) - docs(bindings/java): update the README file (#2338) - docs: add service doc for fs (#2337) - docs: add service doc for cos (#2341) - docs: add service doc for dashmap (#2342) - docs(bindings/java): for BlockingOperator (#2344) ### CI - build(bindings/java): prepare for snapshot release (#2301) - build(bindings/java): support multiple platform java bindings (#2324) - ci(binding/nodejs): Use docker to build nodejs binding (#2328) - build(bindings/java): prepare for automatically multiple platform deploy (#2335) - ci: add bindings java docs and integrate with website (#2346) - ci: avoid copy gitignore to site folder (#2348) - ci(bindings/c): Add diff check (#2359) - ci: Cache librocksdb to speed up CI (#2360) - ci: Don't load rocksdb for all workflows (#2362) - ci: Fix Node.js 12 actions deprecated warning (#2363) - ci: Speed up python docs build (#2364) - ci: Adopt setup-node's cache logic instead (#2365) ### Chore - chore(test): Avoid test names becoming prefixes of other tests (#2333) - chore(bindings/java): improve OpenDALException tests and docs (#2343) - chore(bindings/java): post release 0.1.0 (#2352) - chore(docs): split docs build into small jobs (#2356)' - chore: protect branch gh-pages (#2358) ## [v0.35.0] - 2023-05-23 ### Added - feat(services/onedrive): Implement `list`, `create_dir`, `stat` and upload ing large files (#2231) - feat(bindings/C): Initially support stat in C binding (#2249) - feat(bindings/python): Enable `abi3` to avoid building on different python version (#2255) - feat(bindings/C): support BDD tests using GTest (#2254) - feat(services/sftp): setup integration tests (#2192) - feat(core): Add trait and public API for `append` (#2260) - feat(services/sftp): support copy and rename for sftp (#2263) - feat(services/sftp): support copy and read_seek (#2267) - feat: Add COS service support (#2269) - feat(services/cos): Add support for loading from env (#2271) - feat(core): add presign support for obs (#2253) - feat(services/sftp): setup integration tests (#2192) - feat(core): add presign support for obs (#2253) - feat(core): public API of append (#2284) - test(core): test for append (#2286) - feat(services/oss): add append support (#2279) - feat(bindings/java): implement async ops to pass AsyncStepsTest (#2291) ### Changed - services/gdrive: port code to GdriveCore & add path_2_id cache (#2203) - refactor: Minimize futures dependencies (#2248) - refactor: Add Operator::via_map to support init without generic type parameters (#2280) - refactor(binding/java): build, async and docs (#2276) ### Fixed - fix: Fix bugs that failed wasabi's integration tests (#2273) ### Removed - feat(core): remove `scan` from raw API (#2262) ### Docs - chore(s3): update builder region doc (#2247) - docs: Add services in readme (#2251) - docs: Unify capabilities list for kv services (#2257) - docs(nodejs): fix some example code errors (#2277) - docs(bindings/C): C binding contributing documentation (#2266) - docs: Add new docs that available for all languages (#2285) - docs: Remove unlicensed svg (#2289) - fix(website): double active route (#2290) ### CI - ci: Enable test for cos (#2270) - ci: Add integration tests for supabase (#2272) - ci: replace set-output for docs (#2275) - ci: Fix unit tests (#2282) - ci: Cleanup NOTICE file (#2281) - ci: Fix release not contains incubating (#2292) ### Chore - chore(core): remove unnecessary path prefix (#2265) ## [v0.34.0] - 2023-05-09 ### Added - feat(writer): configurable buffer size of unsized write (#2143) - feat(oay): Add basic s3 list_objects_v2 with start_after support (#2219) - feat: Add typed kv adapter and migrate moka to it (#2222) - feat: migrate service dashmap (#2225) - feat(services/memory): migrate service memory (#2229) - feat: Add assert for public types to ensure Send + Sync (#2237) - feat(services/gcs): Add abort support for writer (#2242) ### Changed - refactor: Replace futures::ready with std::task::ready (#2221) - refactor: Use list without delimiter to replace scan (#2243) ### Fixed - fix(services/gcs): checked_rem_euclid could return Some(0) (#2220) - fix(tests): Etag must be wrapped by `"` (#2226) - fix(services/s3): Return error if credential load fail instead skip (#2233) - fix(services/s3): Return error if region not set for AWS S3 (#2234) - fix(services/gcs): rsa 0.9 breaks gcs presign (#2236) ### Chore - chore: change log subscriber from env_logger to tracing-subscriber (#2238) - chore: Fix build of wasabi (#2241) ## [v0.33.3] - 2023-05-06 ### Added - feat(services/onedrive): Add read and write support for OneDrive (#2129) - test(core): test for `read_with_override_cache_control` (#2155) - feat(http_util): Implement multipart/form-data support (#2157) - feat(http_util): Implement multipart/mixed support (#2161) - RFC-2133: Introduce Append API (#2133) - feat(services/sftp): Add read/write/stat support for sftp (#2186) - feat(services/gdrive): Add read & write & delete support for GoogleDrive (#2184) - feat(services/vercel): Add vercel remote cache support (#2193) - feat(tests): Enable supabase integration tests (#2190) - feat(core): merge scan and list (#2214) ### Changed - refactor(java): refactor java code for java binding (#2145) - refactor(layers/logging): parsing level str (#2160) - refactor: Move not initiated logic to utils instead (#2196) - refactor(services/memcached): Rewrite memecached connection entirely (#2204) ### Fixed - fix(service/s3): set retryable on batch (#2171) - fix(services/supabase): Supabase ci fix (#2200) ### Docs - docs(website): try to add opendal logo (#2159) - doc: update vision to be more clear (#2164) - docs: Refactor `Contributing` and add `Developing` (#2169) - docs: Merge DEVELOPING into CONTRIBUTING (#2172) - docs: fix some grammar errors in the doc of Operator (#2173) - docs(nodejs): Add CONTRIBUTING docs (#2174) - docs: Add CONTRIBUTING for python (#2188) ### CI - ci: Use microsoft rust devcontainer instead (#2165) - ci(devcontainer): Install development deps (#2167) - chore: set workspace default members (#2168) - ci: Setup vercel artifacts integration tests (#2197) - ci: Remove not used odev tools (#2202) - ci: Add tools to generate NOTICE and all deps licenses (#2205) - ci: use Temurin JDK 11 to build the bindings-java (#2213) ### Chore - chore(deps): bump clap from 4.1.11 to 4.2.5 (#2183) - chore(deps): bump futures from 0.3.27 to 0.3.28 (#2181) - chore(deps): bump assert_cmd from 2.0.10 to 2.0.11 (#2180) - chore: Refactor behavior test (#2189) - chore: update readme for more information that is more friendly to newcomers (#2217) ## [v0.33.2] - 2023-04-27 ### Added - feat(core): add test for `stat_with_if_none_match` (#2122) - feat(services/gcs): Add start-after support for list (#2107) - feat(services/azblob): Add supporting presign (#2120) - feat(services/gcs): Add supporting presign support (#2126) - feat(java): connect rust async/await with java future (#2112) - docs: add hdfs classpath related troubleshoot (#2130) - fix(clippy): suppress dead_code check (#2135) - feat(core): Add `cache-control` to Metadata (#2136) - fix(services/gcs): Remove HOST header to avoid gcs RESET connection (#2139) - test(core): test for `write_with_cache_control` (#2131) - test(core): test for `write_with_content_type` (#2140) - test(core): test for `read_with_if_none_match` (#2141) - feat(services/supabase): Add read/write/stat support for supabase (#2119) ### Docs - docs: add hdfs classpath related troubleshoot (#2130) ### CI - ci: Mark job as skipped if owner is not apache (#2128) - ci: Enable native-tls to test gcs presign for workaround (#2138) ## [v0.33.1] - 2023-04-25 ### Added - feat: Add behavior test for read_with_if_match & stat_with_if_match (#2088) - feat(tests): Add fuzz test for writer without content length (#2100) - feat: add if_none_match support for obs (#2103) - feat(services/oss): Add server side encryption support for oss (#2092) - feat(core): update errorKind `PreconditionFailed` to `ConditionNotMatch` (#2104) - feat(services/s3): Add `start-after` support for list (#2096) - feat: gcs support cache control (#2116) ### Fixed - fix(services/gcs): set `content length=0` for gcs initiate_resumable_upload (#2110) - fix(bindings/nodejs): Fix index.d.ts not updated (#2117) ### Docs - chore: improve LoggingLayer docs and pub use log::Level (#2089) - docs(refactor): Add more detailed description of operator, accessor, and builder (#2094) ### CI - chore(bindings/nodejs): update `package.json` repository info (#2078) - ci: Bring hdfs test back (#2114) ## [v0.33.0] - 2023-04-23 ### Added - feat: Add OpenTelemetry Trace Layer (#2001) - feat: add if_none_match support for azblob (#2035) - feat: add if_none_match/if_match for gcs (#2039) - feat: Add size check for sized writer (#2038) - feat(services/azblob): Add if-match support (#2037) - feat(core): add copy&rename to error_context layer (#2040) - feat: add if-match support for OSS (#2034) - feat: Bootstrap new (old) project oay (#2041) - feat(services/OSS): Add override_content_disposition support (#2043) - feat: add IF_MATCH for http (#2044) - feat: add IF_MATCH for http HEAD request (#2047) - feat: add cache control header for azblob and obs (#2049) - feat: Add capability for operation's variant and args (#2057) - feat(azblob): Add override_content_disposition support (#2065) - feat(core): test for read_with_override_content_composition (#2067) - feat(core): Add `start-after` support for list (#2071) ### Changed - refactor: Polish Writer API by merging append and write together (#2036) - refactor(raw/http_util): Add url in error context (#2066) - refactor: Allow reusing the same operator to speed up tests (#2068) ### Fixed - fix(bindings/ruby): use rb_sys_env to help find ruby for building (#2051) - fix: MadsimLayer should be able to built without cfg (#2059) - fix(services/s3): Ignore prefix if it's empty (#2064) ### Docs - docs(bindings/python): ipynb examples for users (#2061) ### CI - ci(bindings/nodejs): publish support `--provenance` (#2046) - ci: upgrade typos to 1.14.8 (#2055) - chore(bindings/C): ignore the formatting of auto-generated opendal.h (#2056) ## [v0.32.0] - 2023-04-18 ### Added - feat: Add wasabi service implementation (#2004) - feat: improve the readability of oli command line error output (#2016) - feat: add If-Match Support for OpRead, OpWrite, OpStat (#2017) - feat: add behavioral test for Write::abort (#2018) - feat: add if-match support for obs (#2023) - feat: Add missing functions for trace layers (#2025) - feat(layer): add madsim layer (#2006) - feat(gcs): add support for gcs append (#1801) ### Changed - refactor: Rename `Create` to `CreateDir` for its behavior changed (#2019) ### Fixed - fix: Cargo lock not updated (#2027) - fix(services/s3): Ignore empty query to make it more compatible (#2028) - fix(services/oss): Fix env not loaded for oss signer (#2029) ### Docs - docs: fix some typos (#2022) - docs: add dev dependency section (#2021) ## [v0.31.1] - 2023-04-17 ### Added - feat(services/azdfs): support rename (#1929) - test: Increate copy/move nested path test depth (#1932) - feat(layers): add a basic minitrace layer (#1931) - feat: add Writer::abort method (#1937) - feat(services/gcs): Allow setting PredefinedAcl (#1989) - feat(services/oss): add oss cache-control header support (#1986) - feat: Add PreconditionFailed Error so users can handle them (#1993) - feat: add http if_none_match support (#1995) - feat: add oss if-none-match support (#1997) - feat(services/gcs): Allow setting default storage_class (#1996) - feat(binding/C): add clang-format for c binding (#2003) ### Changed - refactor: Polish the behavior of scan (#1926) - refactor: Polish the implementation of webhdfs (#1935) ### Fixed - fix: sled should not be enabled by default (#1923) - fix: kv adapter's writer implementation fixed to honour empty writes (#193 4. - fix(services/azblob): fix copy missing content-length (#2000) ### Docs - docs: Adding docs link to python binding (#1921) - docs(bindings/python): fix wrong doc link (#1928) - docs: Add contributing for OpenDAL (#1984) - docs: Add explanation in contributing (#1985) - docs: Feel relax in community and don't hurry (#1987) - docs: update contributing (#1998) - docs(services/memory): Fix wrong explanation (#2002) - docs: Add OpenDAL VISION (#2007) - docs: update VISION and behavior tests doc (#2009) ### CI - ci(bindings/nodejs): Access should be set to public before publish (#1919) - ci: Re-enable webhdfs test (#1925) - chore: add .editorconfig (#1988) - ci: Fix format after adding editorconfig (#1990) ## [v0.31.0] - 2023-04-12 ### Added - feat(bindings/java): add cucumber test (#1809) - feat(bindings/java): setup Java CI (#1823) - feat: add if_none_match support (#1832) - feat: Retry when some of batch operations are failing (#1840) - feat: add writer support for aliyun oss (#1842) - feat(core): Add Copy Support (#1841) - feat(bindings/c): fix c bindings makefile (#1849) - feat(core): add behavior tests for copy & blocking_copy (#1852) - feat(s3): allow users to specify storage_class (#1854) - feat(s3): Support copy (#1856) - Add check for s3 bucket name (#1857) - feat(core): Support rename (#1862) - feat(bindings/nodejs): add `copy` and `rename` (#1866) - feat(azblob): Support copy (#1868) - feat(gcs): copy support for GCS (#1869) - feat(bindings/c): framework of add basic io and init logics (#1861) - feat(webdav): support copy (#1870) - feat(services/oss): Add Copy Support (#1874) - feat(services/obs): Add Copy Support (#1876) - feat(services/webdav): Support Rename (#1878) - binding/c: parse opendal to use typed BlockingOperator (#1881) - binding/c: clean up comments and type assertion for BlockingOperator (#1883) - binding(python): Support python binding benchmark for opendal (#1882) - feat(bindings/c): add support for free heap-allocated operator (#1890) - feat(binding/c): add is_exist to operator (#1892) - feat(bindings/java): add Stat support (#1894) - feat(services/gcs): Add customed token loader support (#1908) - feat(services/oss): remove unused builder prop allow_anonymous (#1913) - feat: Add feature flag for all services (#1915) ### Changed - refactor(core): Simplify the usage of BatchOperation and BatchResults (#1843) - refactor: Use reqwest blocking client instead of ureq (#1853) - refactor: Bump MSRV to 1.64 (#1864) - refactor: Remove not used blocking http client (#1895) - refactor: Change presign to async for future refactor (#1900) - refactor(services/gcs): Migrate to async reqsign (#1906) - refactor(services/azdfs): Migrate to async reqsign (#1903) - refactor(services/azblob): Adopt new reqsign (#1902) - refactor(services/s3): Migrate to async reqsign (#1909) - refactor(services/oss): Migrate to async reqsign (#1911) - refactor: Use chrono instead of time to work well with ecosystem (#1912) - refactor(service/obs): Migrate obs to async reqsign (#1914) ### Fixed - fix: podling website check (#1838) - fix(website): copyright update (#1839) - fix(core): Add checks before doing copy (#1845) - fix(core): S3 Copy should set SSE headers (#1860) - fix: Fix presign related unit tests (#1910) ### Docs - docs(bindings/nodejs): fix build failed (#1819) - docs: fix several typos in the documentation (#1846) - doc(bindings/nodejs): update presign example in doc (#1901) ### CI - ci: Fix build for nodejs binding on macos (#1813) - binding/c: build: add phony to makefile, and some improve (#1850) - ci: upgrade hawkeye action (#1834) ### Chore - chore(bindings/nodejs): add deno benchmark (#1814) - chore: Add CODEOWNERS (#1817) - chore(deps): bump opentelemetry-jaeger from 0.16.0 to 0.18.0 (#1829) - chore(deps): bump opentelemetry from 0.17.0 to 0.19.0 (#1830) - chore(deps): bump tokio from 1.26.0 to 1.27.0 (#1828) - chore(deps): bump napi-derive from 2.12.1 to 2.12.2 (#1827) - chore(deps): bump async-trait from 0.1.67 to 0.1.68 (#1826) - chore: Cleanup code for oss writer (#1847) - chore: Make clippy happy (#1865) - binding(python): Format python code in binding (#1885) ## [v0.30.5] - 2023-03-31 ### Added - feat(oli): implement `oli rm` (#1774) - feat(bindings/nodejs): Support presign (#1772) - feat(oli): implement `oli stat` (#1778) - feat(bindings/object_store): Add support for list and list_with_delimiter (#1784) - feat(oli): implement `oli cp -r` (#1787) - feat(bindings/nodejs): Make PresignedRequest serializable (#1797) - feat(binding/c): add build.rs and cbindgen as dep to gen header (#1793) - feat(bindings/nodejs): Add more APIs and examples (#1799) - feat: reader_with and writer_with (#1803) - feat: add override_cache_control (#1804) - feat: add cache_control to OpWrite (#1806) ### Changed - refactor(oli): switch to `Operator::scan` and `Operator::remove_all` (#1779) - refactor(bindings/nodejs): Polish benchmark to make it more readable (#1810) ### Fixed - fix(oli): set the root of fs service to '/' (#1773) - fix: align WebDAV stat with RFC specification (#1783) - fix(bindings/nodejs): fix read benchmark (#1805) ### CI - ci: Split clippy and docs check (#1785) - ci(bindings/nodejs): Support aarch64-apple-darwin (#1780) - ci(bindings/nodejs): publish with LICENSE & NOTICE (#1790) - ci(services/redis): Add dragonfly test (#1808) ### Chore - chore(bindings/python): update maturin to 0.14.16 (#1777) - chore(bin/oli): Set oli version from package version (#1786) - chore(oli): set cli version in a central place (#1789) - chore: don't pin time version (#1788) - chore(bindings/nodejs): init benchmark (#1798) - chore(bindings/nodejs): Fix generated headers (#1802) ## [v0.30.4] - 2023-03-26 ### Added - feat(oli): add config file to oli (#1706) - feat: make oli support more services (#1717) - feat(bindings/ruby): Setup the integrate with magnus (#1712) - feat(bindings/ruby): setup cucumber tests (#1725) - feat(bindings/python): convert to mixed Python/Rust project layout (#1729) - RFC-1735: Operation Extension (#1735) - feat(oli): load config from both env and file (#1737) - feat(bindings/ruby): support read and write (#1734) - feat(bindings/ruby): support stat, and pass all blocking bdd test (#1743) - feat(bindings/ruby): add namespace (#1745) - feat: Add override_content_disposition for OpRead (#1742) - feat(bindings/java): add java binding (#1736) - feat(oli): implement oli ls (#1755) - feat(oli): implement oli cat (#1759) ### Fixed - fix(bindings/nodejs): Publish sub-package name (#1704) ### Docs - docs: Update comparison vs object_store (#1698) - docs(bindings/python): add pdoc to docs env (#1718) - docs: List working on bindings in README (#1753) ### CI - ci: Fix workflow not triggered when itself changed (#1716) - ci: Remove ROCKSDB_LIB_DIR after we didn't install librocksdb (#1719) - ci: Fix nodejs built failed for "Unexpected token o in JSON at position 0" (#1722) - ci: Split cache into more parts (#1730) - ci: add a basic ci for ruby (#1744) - ci: Remove target from cache (#1764) ### Chore - chore: Fix CHANGELOG not found (#1694) - chore: Remove publish=false of oli (#1697) - chore: Fix a few typos in code comment (#1701) - chore(bindins/nodejs): Update README (#1709) - chore: rename binaries to bin (#1714) - chore: bump rocksdb to resolve dependency conflicts with magnus (#1713) - chore(bindings/nodejs): Remove outdated napi patches (#1727) - chore: Add CITATION file for OpenDAL (#1746) - chore: improve NotADirectory error message with ending slash (#1756) - chore(bindings/python): update pyo3 to 0.18.2 (#1758) ## [v0.30.3] - 2023-03-16 ### Added - feat: Infer storage name based on endpoint (#1551) - feat(bindings/python): implement async file-like reader API (#1570) - feat: website init (#1580) - feat(bindings/python): implement list and scan for AsyncOperator (#1586) - feat: Implement logging/metrics/tracing support for Writer/BlockingWriter (#1588) - feat(bindings/python): expose layers to Python (#1591) - feat(bindings/c): Setup the integrate with cbindgen (#1603) - feat(bindings/nodejs): Auto-generate docs (#1625) - feat: add max_batch_operations for AccessorInfo (#1615) - feat(azblob): Add support for batch operations (#1610) - services/redis: Implement Write::append with native support (#1651) - feat(tests): Introducing BDD tests for all bindings (#1654) - feat(bindings/nodejs): Migrate to BDD test (#1661) - feat(bindings/nodejs): Add generated `index.d.ts` (#1664) - feat(bindings/python): add auto-generated api docs (#1613) - feat(bindings/python): add `__repr__` to `Operator` and `AsyncOperator` (#1683) ### Changed - \*: Change all files licenses to ASF (#1592) - refactor(bindings/python): only enable `pyo3/entension-module` feature when building with maturin (#1680) ### Fixed - fix(bindings/python): Fix the metadata for Python binding (#1568) - fix: Operator::remove_all behaviour on non-existing object fixed (#1587) - fix: reset Reader::SeekState when poll completed (#1609) - fix: Bucket config related error is misleadling (#1684) - fix(services/s3): UploadId should be percent encoded (#1690) ### CI - ci: Fix typo in workflows (#1582) - ci: Don't check dep updates so frequently (#1599) - ci: Setup asf config (#1622) - ci: Use gh-pages instead (#1623) - ci: Update Github homepage (#1627) - ci: Update description for OpenDAL (#1628) - ci: Send notifications to commits@o.a.o (#1629) - ci: set main branch to be protected (#1631) - ci: Add release scripts for OpenDAL (#1637) - ci: Add check scripts (#1638) - ci: Remove rust-cache to allow we can test rust code now (#1643) - ci: Enable license check back (#1663) - ci(bindings/nodejs): Enable formatter (#1665) - ci: Bring our actions back (#1668) - ci: Use korandoru/hawkeye@v1.5.4 instead (#1672) - ci: Fix license header check and doc check (#1674) - infra: Add odev to simplify contributor's setup (#1687) ### Docs - docs: Migrate links to o.a.o (#1630) - docs: update the old address and the LICENSE size. (#1633) - doc: update doc-link (#1642) - docs(blog): Way to Go: OpenDAL successfully entered Apache Incubator (#1652) - docs: Reorganize README of core and whole project (#1676) - doc: Update README.md for quickstart (#1650) - doc: uncomment the use expr for operator example (#1685) ### Website - website: Let's deploy our new website (#1581) - website: Fix CNAME not set (#1590) - website: Fix website publish (#1626) - website: Add GitHub entry (#1636) - website: move some content of footer to navbar. (#1660) ### Chore - chore(bindings/nodejs): fix missing files to publish (#1569) - chore(deps): bump lazy-regex from 2.4.1 to 2.5.0 (#1573) - chore(deps): bump tokio from 1.25.0 to 1.26.0 (#1577) - chore(deps): bump hyper from 0.14.24 to 0.14.25 (#1575) - chore(deps): bump serde from 1.0.152 to 1.0.155 (#1576) - chore(deps): bump peaceiris/actions-gh-pages from 3.9.0 to 3.9.2 (#1593) - chore(deps): bump async-trait from 0.1.64 to 0.1.66 (#1594) - chore(deps): bump serde_json from 1.0.93 to 1.0.94 (#1596) - chore(deps): bump paste from 1.0.11 to 1.0.12 (#1598) - chore(deps): bump napi from 2.11.2 to 2.11.3 (#1595) - chore(deps): bump serde from 1.0.155 to 1.0.156 (#1600) - chore: fix the remaining license (#1605) - chore: add a basic workflow for c bindings (#1608) - chore: manage deps with maturin (#1611) - chore: Rename files to yaml (#1624) - chore: remove PULL_REQUEST_TEMPLATE (#1634) - chore: add NOTICE and DISCLAIMER (#1635) - chore(operator): apply max_batch_limit for async operator (#1641) - chore: replace datafuselabs/opendal with apache/opendal (#1647) - chore: make check.sh be executable and update gitignore (#1648) - chore(automation): fix release.sh packaging sha512sum (#1649) - chore: Update metadata (#1666) - chore(website): Remove authors.yml (#1669) - chore: Move opendal related staffs to core (#1673) - chore: Remove not needed ignore from licenserc (#1677) - chore: Ignore generated docs from git (#1686) ## [v0.30.2] - 2023-03-10 ### CI - ci(bindings/nodejs): Fix nodejs package can't uploaded (#1564) ## [v0.30.1] - 2023-03-10 ### Docs - docs: Fix Operator::create() has been removed (#1560) ### CI - ci: Fix python & nodejs not released correctly (#1559) ### Chore - chore(bindings/nodejs): update license in package.json (#1556) ## [v0.30.0] - 2023-03-10 ### Added - RFC-1477: Remove Object Concept (#1477) - feat(bindings/nodejs): fs Operator (#1485) - feat(service/dashmap): Add scan support (#1492) - feat(bindings/nodejs): Add Writer Support (#1490) - feat: Add dummy implementation for accessor and builder (#1503) - feat(bindings/nodejs): Support List & append all default services (#1505) - feat(bindings/python): Setup operator init logic (#1513) - feat(bindings/nodejs): write support string (#1520) - feat(bindings/python): add support for services that opendal enables by default (#1522) - feat(bindings/nodejs): Remove Operator.writer until we are ready (#1528) - feat(bindings/nodejs): Support Operator.create_dir (#1529) - feat(bindings/python): implement create_dir (#1534) - feat(bindings/python): implement delete and export more metadata fields (#1539) - feat(bindings/python): implement blocking list and scan (#1541) - feat: Append EntryMode to Entry (#1543) - feat: Entry refactoring to allow external creation (#1547) - feat(bindings/nodejs): Support Operator.scanSync & Operator.listSync (#1546) - feat: remove_via can delete files concurrently (#1495) ### Changed - refactor: Split operator APIs into different part (#1483) - refactor: Remove Object prefix for public API (#1488) - refactor: Remove the concept of Object (#1496) - refactor: remove ReadDir in FTP service (#1504) - refactor: rename public api create to create_dir (#1512) - refactor(bindings/python): return bytes directly and add type stub file (#1514) - tests: Remove not needed create file test (#1516) - refactor: improve the python binding implementation (#1517) - refactor(bindings/nodejs): Remove scheme from bindings (#1552) ### Fixed - fix(services/s3): Make sure the ureq's body has been consumed (#1497) - fix(services/s3): Allow retry error RequestTimeout (#1532) ### Docs - docs: Remove all references to object (#1500) - docs(bindings/python): Add building docs (#1526) - docs(bindings/nodejs): update readme (#1527) - docs: Add detailed docs for create_dir (#1537) ### CI - ci: Don't run binding tests if only services changes (#1498) - ci: Improve rocksdb build speed by link dynamic libs (#1502) - ci: Fix bindings CI not running on PR (#1530) - ci: Polish scripts and prepare for releasing (#1553) ### Chore - chore: Re-organize the project layout (#1489) - chore: typo & clippy (#1499) - chore: typo (#1501) - chore: Move memcache-async into opendal (#1544) ## [v0.29.1] - 2023-03-05 ### Added - feat(bindings/python): Add basic IO support (#1464) - feat(binding/node.js): basic IO (#1416) - feat(bindings/nodejs): Align to OpenDAL exports (#1466) - chore(bindings/nodejs): remove duplicate attribute & unused comment (#1478) ### Changed - refactor: Promote operator as a mod for further refactor (#1479) ### Docs - docs: Add convert from m\*n to m+n (#1454) - docs: Polish comments for public types (#1455) - docs: Add discord chat link (#1474) ### Chore - chore: fix typo (#1456) - chore: fix typo (#1459) - benches: Generate into Bytes instead (#1463) - chore(bindings/nodjes): Don't check-in binaries (#1469) - chore(binding/nodejs): specific package manager version with hash (#1470) ## [v0.29.0] - 2023-03-01 ### Added - RFC-1420: Object Writer (#1420) - feat: oss backend support http protocol (#1432) - feat: Implement ObjectWriter Support (#1431) - feat/layers/retry: Add Write Retry support (#1447) - feat: Add Write append tests (#1448) ### Changed - refactor: Decouple decompress read feature from opendal (#1406) - refactor: Cleanup pager related implementation (#1439) - refactor: Polish the implement details for Writer (#1445) - refactor: Remove `io::input` and Rename `io::output` to `oio` (#1446) ### Fixed - fix(services/s3): Fix part number for AWS S3 (#1450) ### CI - ci: Consistently apply license header (#1411) - ci: add typos check (#1425) ### Docs - docs: Add services-dashmap feature (#1404) - docs: Fix incorrect indent for title (#1405) - docs: Add internal sections of Accessor and Layer (#1408) - docs: Add more guide for Accessor (#1409) - docs: Add tutorial of building a duck storage service (#1410) - docs: Add a basic object example (#1422) ### Chore - chore: typo fix (#1418) - chore: Make license check happy (#1423) - chore: typo-fix (#1434) ## [v0.28.0] - 2023-02-22 ### Added - feat: add dashmap support (#1390) ### Changed - refactor: Implement query based object metadata cache (#1395) - refactor: Store complete inside bits and add more examples (#1397) - refactor: Trigger panic if users try to visit not fetched metadata (#1399) - refactor: Polish the implement of Query Based Metadata Cache (#1400) ### Docs - RFC-1391: Object Metadataer (#1391) - RFC-1398: Query Based Metadata (#1398) ## [v0.27.2] - 2023-02-20 ### Added - feat: Add batch API for Accessor (#1339) - feat: add Content-Disposition for inner API (#1347) - feat: add content-disposition support for services (#1350) - feat: webdav service support bearer token (#1349) - feat: support auth for HttpBackend (#1359) - feat: Add batch delete support (#1357) - feat(webdav): add list and improve create (#1330) - feat: Integrate batch with existing ecosystem better (#1378) - feat: Add batch delete support for oss (#1385) ### Changed - refactor: Authorization logic for WebdavBackend (#1348) - refactor(webhdfs): handle 307 redirection instead of noredirect (#1358) - refactor: Polish http authorization related logic (#1367) - refactor: Cleanup duplicated code (#1373) - refactor: Cleanup some not needed error context (#1374) ### Docs - docs: Fix broken links (#1344) - docs: clarify about opendal user defined client (#1356) ### Fixed - fix(webhdfs): should prepend http:// scheme (#1354) ### Infra - ci: Pin time <= 0.3.17 until we decide to bump MSRV (#1361) - ci: Only run service test on changing (#1363) - ci: run tests with nextest (#1370) ## [v0.27.1] - 2023-02-13 ### Added - feat: Add username and password support for WebDAV (#1323) - ci: Add test case for webdav with basic auth (#1327) - feat(oli): support s3 uri without profile (#1328) - feat: Add scan support for kv adapter (#1333) - feat: Add scan support for sled (#1334) ### Changed - chore(deps): update moka requirement from 0.9 to 0.10 (#1331) - chore(deps): update rocksdb requirement from 0.19 to 0.20 (#1332) ### Fixed - fix(services/oss,s3): Metadata should be marked as complete (#1335) ## [v0.27.0] - 2023-02-11 ### Added - feat: Add Retryable Pager Support (#1304) - feat: Add Sled support (#1305) - feat: Add Object::scan() support (#1314) - feat: Add object page size support (#1318) ### Changed - refactor: Hide backon from our public API (#1302) - refactor: Don't expose ops structs to users directly (#1303) - refactor: Move and rename ObjectPager and ObjectEntry for more clear semantics (#1308) - refactor: Implement strong typed pager (#1311) - deps: remove unused deps (#1321) - refactor: Extract scan as a new API and remove ListStyle (#1324) ### Docs - docs: Add risingwave in projects (#1322) ### Fixed - ci: Fix dev container Dockerfile (#1298) - fix: Rocksdb's scheme not output correctly (#1300) - chore: fix name typo in oss backend (#1316) - chore: Add typos-cli and fix typos (#1320) ## [v0.26.2] - 2023-02-07 ### Added - feat: Add ChaosLayer to inject errors into underlying services (#1287) - feat: Implement retry reader (#1291) - feat: use std::path::Path for fs backend (#1100) - feat: Implement services webhdfs (#1263) ### Changed - refactor: Split CompleteReaderLayer from TypeEraserLayer (#1290) - refactor(services/fs): Remove not needed generic (#1292) ### Docs - docs: fix typo (#1285) - docs: Polish docs for better reading (#1288) ### Fixed - fix: FsBuilder can't be used with empty root anymore (#1293) - fix: Fix retry happened in seek's read ahead logic (#1294) ## [v0.26.1] - 2023-02-05 ### Changed - refactor: Remove not used layer subdir (#1280) ### Docs - docs: Add v0.26 upgrade guide (#1276) - docs: Add feature sets in services (#1277) - docs: Migrate all docs in rustdoc instead (#1281) - docs: Fix index page not redirected (#1282) ## [v0.26.0] - 2023-02-04 ### Added - feat: Add benchmarks for blocking_seek operations (#1258) - feat: add dev container (#1261) - feat: Zero Cost OpenDAL (#1260) - feat: Allow dynamic dispatch layer (#1273) ### Changed - refactor: remove the duplicated dependency in dev-dependencies (#1257) - refactor: some code in GitHub Actions (#1269) - refactor: Don't expose services mod directly (#1271) - refactor: Polish Builder API (#1272) ## [v0.25.2] - 2023-01-30 ### Added - feat: Add basic object_store support (#1243) - feat: Implement webdav support (#1246) - feat: Allow passing content_type to OSS presign (#1252) - feat: Make sure short functions have been inlined (#1253) ### Changed - refacor(services/fs): Make normalized path check optional (#1242) ### Docs - docs(http): remove out-dated comments (#1240) - docs: Add bindings in README (#1244) - docs: Add docs for webdav and http services (#1248) - docs: Add webdav in lib docs (#1249) ### Fixed - fix(services/ghac): Fix log message for ghac_upload in write (#1239) ## [v0.25.1] - 2023-01-27 ### Added - ci: Setup benchmark workflow (#1200) - feat: Let's try play with python (#1205) - feat: Let's try play with Node.js (#1206) - feat: Allow retry sending read request (#1212) - ci: Make sure opendal is buildable on windows (#1221) - ci: Remove not needed audit checks (#1226) ### Changed - refactor: Remove observe read/write (#1202) - refactor: Remove not used unwind safe feature (#1218) - cleanup: Move oli and oay into binaries (#1227) - cleanup: Move testdata into tests/data (#1228) - refactor(layers/metrics): Defer initiation of error counters (#1232) ### Fixed - fix: Retry for read and write should at ObjectReader level (#1211) ## [v0.25.0] - 2023-01-18 ### Added - feat: Add dns cache for std dns resolver (#1191) - feat: Allow setting http client that built from external (#1192) - feat: Implement BlockingObjectReader (#1194) ### Changed - chore(deps): replace dotenv with dotenvy (#1187) - refactor: Avoid calling detect region if we know the region (#1188) - chore: ensure minimal version buildable (#1193) ## [v0.24.6] - 2023-01-12 ### Added - feat: implement tokio::io::{AsyncRead, AsyncSeek} for ObjectReader (#1175) - feat(services/hdfs): Evaluating the new async implementation (#1176) - feat(services/ghac): Handling too many requests error (#1181) ### Fixed - doc: fix name change in README (#1179) ## [v0.24.5] - 2023-01-09 ### Fixed - fix(services/memcached): TcpStream should only accept host:port (#1170) ## [v0.24.4] - 2023-01-09 ### Added - feat: Add presign endpoint option for OSS (#1135) - feat: Reset state while returning error so that we can retry IO (#1166) ### Changed - chore(deps): update base64 requirement from 0.20 to 0.21 (#1164) ### Fixed - fix: Memcached can't work on windows (#1165) ## [v0.24.3] - 2023-01-09 ### Added - feat: Implement memcached service support (#1161) ## [v0.24.2] - 2023-01-08 ### Changed - refactor: Use dep: to make our features more clean (#1153) ### Fixed - fix: ghac shall return ObjectAlreadyExists for writing the same path (#1156) - fix: futures read_to_end will lead to performance regression (#1158) ## [v0.24.1] - 2023-01-08 ### Fixed - fix: Allow range_read to be retired (#1149) ## [v0.24.0] - 2023-01-07 ### Added - Add support for SAS tokens in Azure blob storage (#1124) - feat: Add github action cache service support (#1111) - docs: Add docs for ghac service (#1126) - feat: Implement offset seekable reader for zero cost read (#1133) - feat: Implement fuzz test on ObjectReader (#1140) ### Changed - chore(deps): update quick-xml requirement from 0.26 to 0.27 (#1101) - ci: Enable rust cache for CI (#1107) - deps(oay,oli): Update dependences of oay and oli (#1122) - refactor: Only add content length hint if we already know length (#1123) - refactor: Redesign outpu bytes reader trait (#1127) - refactor: Remove open related APIs (#1129) - refactor: Merge and cleanup io & io_util modules (#1136) ### Fixed - ci: Fix build for oay and oli (#1097) - fix: Fix rustls support for suppaftp (#1102) - fix(services/ghac): Fix pkg version not used correctly (#1125) ## [v0.23.0] - 2022-12-22 ### Added - feat: Implement object handler so that we can do seek on file (#1091) - feat: Implement blocking for hdfs (#1092) - feat(services/hdfs): Implement open and blocking open (#1093) - docs: Add mozilla/sccache into projects (#1094) ## [v0.22.6] - 2022-12-20 ### Added - feat(io): make BlockingBytesRead Send + Sync (#1083) - feat(fs): skip seek if offset is 0 (#1082) - RFC-1085: Object Handler (#1085) - feat(services/s3,gcs): Allow accepting signer directly (#1087) ## [v0.22.5] - 2022-12-13 ### Added - feat: Add service account support for gcs (#1076) ## [v0.22.4] - 2022-12-13 ### Added - improve blocking read use read_to_end (#1072) - feat(services/gcs): Fully implement default credential support (#1073) ### Fixed - fix: read a large range without error and add test (#1068) - fix(services/oss): Enable standard behavior for oss range (#1070) ## [v0.22.3] - 2022-12-11 ### Added - feat(layers/metrics): Merge error and failure counters together (#1058) - feat: Set MSRV to 1.60 (#1060) - feat: Add unwind safe flag for operator (#1061) - feat(azblob): Add build from connection string support (#1064) ### Fixed - fix(services/moka): Don't print all content in cache (#1057) ## [v0.22.2] - 2022-12-07 ### Added - feat(presign): support presign head method for s3 and oss (#1049) ## [v0.22.1] - 2022-12-05 ### Fixed - fix(services/s3): Allow disable loading from imds_v2 and assume_role (#1044) ## [v0.22.0] - 2022-12-05 ### Added - feat: improve temp file organization when enable atomic write in fs (#1017) - feat: Allow configure LoggingLayer's level (#1021) - feat: Enable users to specify the cache policy (#1024) - feat: Implement presign for oss (#1035) ### Changed - refactor: Polish error handling of different services (#1018) - refactor: Merge metadata and content cache together (#1020) - refactor(layer/cache): Allow users implement cache by themselves (#1040) ### Fixed - fix(services/fs): Make sure writing file is truncated (#1036) ## [v0.21.2] - 2022-11-27 ### Added - feat: Add azdfs support (#1009) - feat: Set MSRV of opendal to 1.60 (#1012) ### Docs - docs: Fix docs for azdfs service (#1010) ## [v0.21.1] - 2022-11-26 ### Added - feat: Export ObjectLister as public type (#1006) ### Changed - deps: Remove not used thiserror and num-trait (#1005) ## [v0.21.0] - 2022-11-25 ### Added - docs: Add greptimedb and mars into projects (#975) - RFC-0977: Refactor Error (#977) - feat: impl atomic write for fs service (#991) - feat: Add OperatorMetadata to avoid expose AccessorMetadata (#997) - feat: Improve display for error (#1002) ### Changed - refactor: Use separate Error instead of std::io::Error to avoid confusing (#976) - refactor: Return ReplyCreate for create operation (#981) - refactor: Add ReplyRead for read operation (#982) - refactor: Add RpWrite for write operation (#983) - refactor: Add RpStat for stat operation (#984) - refactor: Add RpDelete for delete operations (#985) - refactor: Add RpPresign for presign operation (#986) - refactor: Add reply for all multipart operations (#988) - refactor: Add Reply for all blocking operations (#989) - refactor: Avoid accessor in object entry (#992) - refactor: Move accessor into raw apis (#994) - refactor: Move io to raw (#996) - refactor: Move {path,wrapper,http_util,io_util} into raw modules (#998) - refactor: Move ObjectEntry and ObjectPage into raw (#999) - refactor: Accept Operator instead of `Arc` (#1001) ### Fixed - fix: RetryAccessor is too verbose (#980) ## [v0.20.1] - 2022-11-18 ### Added - feat: Implement blocking operations for cache services (#970) ### Fixed - fix: Use std Duration as args instead (#966) - build: Make opendal buildable on 1.60 (#968) - fix: Avoid cache missing after write (#971) ## [v0.20.0] - 2022-11-17 ### Added - RFC-0926: Object Reader (#926) - feat: Implement Object Reader (#928) - feat(services/s3): Return Object Meta for Read operation (#932) - feat: Implement Bytes Content Range (#933) - feat: Add Content Range support in ObjectMetadata (#935) - feat(layers/content_cache): Implement WholeCacheReader (#936) - feat: CompressAlgorithm derive serde. (#939) - feat: Allow using opendal without tls support (#945) - refactor: Refactor OpRead with BytesRange (#946) - feat: Allow using opendal with native tls support (#949) - docs: add docs for tls dependencies features (#951) - feat: Make ObjectReader content_length returned for all services (#954) - feat(layers): Implement fixed content cache (#953) - feat: Enable default_ttl support for redis (#960) ### Changed - refactor: Return ObjectReader in Accessor::read (#929) - refactor(oay,oli): drop unnecessary patch.crates-io from `Cargo.toml` - refactor: Polish bytes range (#950) - refactor: Use simplified kv adapter instead (#959) ### Fixed - fix(ops): Fix suffix range behavior of bytes range (#942) - fix: Fix cache path not used correctly (#958) ## [v0.19.8] - 2022-11-13 ### Added - feat(services/moka): Use entry's bytes as capacity weigher (#914) - feat: Implement rocksdb service (#913) ### Changed - refactor: Reduce backend builder log level to debug (#907) - refactor: Remove deprecated features (#920) - refactor: use moka::sync::SegmentedCache (#921) ### Fixed - fix(http): Check already read size before returning (#919) ## [v0.19.7] - 2022-10-31 ### Added - feat: Implement content type support for stat (#891) ### Changed - refactor(layers/metrics): Holding all metrics handlers to avoid lock (#894) - refactor(layers/metrics): Only update metrics while dropping readers (#896) ## [v0.19.6] - 2022-10-25 ### Fixed - fix: Metrics blocking reader doesn't handle operation correctly (#887) ## [v0.19.5] - 2022-10-24 ### Added - feat: add a feature named trust-dns (#879) - feat: implement write_with (#880) - feat: `content-type` configuration (#878) ### Fixed - fix: Allow forward layers' acesser operations to inner (#884) ## [v0.19.4] - 2022-10-15 ### Added - feat: Improve into_stream by reduce zero byte fill (#864) - debug: Add log for sync http client (#865) - feat: Add debug log for finishing read (#867) - feat: Try to use trust-dns-resolver (#869) - feat: Add log for dropping reader and streamer (#870) ### Changed - refactor: replace md5 with md-5 (#862) - refactor: replace the hard code to X_AMZ_BUCKET_REGION constant (#866) ## [v0.19.3] - 2022-10-13 ### Fixed - fix: Retry for write is not implemented correctly (#860) ## [v0.19.2] - 2022-10-13 ### Added - feat(experiment): Allow user to config http connection pool (#843) - feat: Add concurrent limit layer (#848) - feat: Allow kv services implemented without list support (#850) - feat: Implement service for moka (#852) - docs: Add docs for moka service and concurrent limit layer (#857) ## [v0.19.1] - 2022-10-11 ### Added - feat: Allow retry read and write (#826) - feat: Convert interrupted error to permanent after retry (#827) - feat(services/ftp): Add connection pool for FTP (#832) - feat: Implement retry for write operation (#831) - feat: Bump reqsign to latest version (#837) - feat(services/s3): Add role_arn and external_id for assume_role (#838) ### Changed - test: accelerate behaviour test `test_list_rich_dir` (#828) ### Fixed - fix: ObjectEntry returned in batch operator doesn't have correct accessor (#839) - fix: Accessor in layers not set correctly (#840) ## [v0.19.0] - 2022-10-08 ### Added - feat: Implement object page stream for services like s3 (#787) - RFC-0793: Generic KV Services (#793) - feat(services/kv): Implement Scoped Key (#796) - feat: Add scan in KeyValueAccessor (#797) - feat: Implement basic kv services support (#799) - feat: Introduce kv adapter for opendal (#802) - feat: Add integration test for redis (#804) - feat: Add OSS Service Support (#801) - feat: Add integration tests for OSS (#814) ### Changed - refactor: Move object to mod (#786) - refactor: Implement azblob dir stream based on ObjectPageStream (#790) - refactor: Implement memory services by generic kv (#800) - refactor: Don't expose backend to users (#816) - tests: allow running tests when env is `true` (#818) - refactor: Remove deprecated type aliases (#819) - test: list rich dir (#820) ### Fixed - fix(services/redis): MATCH can't handle correctly (#803) - fix: Disable ipfs redirection (#809) - fix(services/ipfs): Use ipfs files API to copy data (#811) - fix(services/hdfs): Allow retrying would block (#815) ## [v0.18.2] - 2022-10-01 ### Added - feat: Enable retry layer by default (#781) ### Changed - ci: Enable IPFS NoFecth to avoid networking timeout (#780) - ci: Build all feature in release to prevent build failure under release profile (#783) ### Fixed - fix: Fix build error under release profile (#782) ## [v0.18.1] - 2022-10-01 ### Fixed - fix(services/s3): Content MD5 not set during list (#775) - test: Add a test for ObjectEntry metadata cache (#776) ## [v0.18.0] - 2022-10-01 ### Added - feat: Add Metadata Cache Layer (#739) - feat: Bump reqsign version to 0.5 (#741) - feat: Derive Hash, Eq, PartialEq for Operation (#749) - feat: Make AccessorMetadata public so outer users can use (#750) - feat: Expose AccessorCapability to users (#751) - feat: Expose opendal's http util to users (#753) - feat: Implement convert from PresignedRequest (#756) - feat: Make ObjectMetadata setter public (#758) - feat: Implement cached metadata for ObjectEntry (#761) - feat: Assign unique name for memory backend (#769) ### Changed - refactor: replace error::other with new_other_object_error (#738) - chore(compress): log with trace level instead of debug. (#752) - refactor: Rename DirXxxx to ObjectXxxx instead (#759) ### Fixed - fix(http_util): Disable auto compress and enable http proxy (#731) - deps: Fix build after bump deps of oli and oay (#766) ## [v0.17.4] - 2022-09-27 ### Fixed - fix(http_util): Allow retry more errors (#724) - fix(services/ftp): Suffix endpoints with default port (#726) ## [v0.17.3] - 2022-09-26 ### Added - feat: Add SubdirLayer to allowing switch directory (#718) - feat(layers/retry): Add warning log while retry happened (#721) ### Fixed - fix: update metrics on result (#716) - fix: SubdirLayer should handle dir correctly (#720) ## [v0.17.2] - 2022-09-26 ### Add - feat: implement basic cp command (#688) - chore: also parse 'FTPS' to Scheme::Ftp (#704) ### Changed - refactor: remove `enable_secure` in FTP service (#709) - oli: refactor copy implementation (#710) ### Fixed - fix: Handle slash normalized false positives properly (#702) - fix: Tracing is too verbose (#707) - chore: fix error message in ftp service (#705) ## [v0.17.1] - 2022-09-19 ### Added - feat: redis service implement (#679) - feat: Implement AsyncBufRead for IntoReader (#690) - feat: expose security token of s3 (#693) ### Changed - refactor: avoid unnecessary parent creating in Redis service (#692) - refactor: Refactor HTTP Client to split sending and incoming logic (#695) ### Fixed - fix: Handle write data in async way for IPMFS (#694) ## [v0.17.0] - 2022-09-15 ### Added - RFC: Path In Accessor (#661) - feat: Implement RFC-0661: Path In Accessor (#664) - feat: Hide http client internal details from users (#672) - feat: make rustls the default tls implementation (#674) - feat: Implement benches for layers (#681) ### Docs - docs: Add how to implement service docs (#665) - refactor: update redis support rfc (#676) - docs: update metrics documentation (#684) ### Fixed - fix: Immutable Index Layer could return duplicated paths (#671) - fix: Remove not needed type parameter for immutable_layer (#677) - fix: Don't trace buf field in poll_read (#682) - fix: List non-exist dir should return empty (#683) - fix: Add path validation for fs backend (#685) ## [v0.16.0] - 2022-09-12 ### Added - feat: Implement tests for read-only services (#634) - feat(services/ftp): Implemented multi connection (#637) - feat: Finalize FTP read operation (#644) - feat: Implement service for IPFS HTTP Gateway (#645) - feat: Add ImmutableIndexLayer (#651) - feat: derive Hash for Scheme (#653) - feat(services/ftp): Setup integration tests (#648) ### Changed - refactor: Migrate all behavior tests to capability based (#635) - refactor: Remove list support from http service (#639) - refactor: Replace isahc with reqwest and ureq (#642) ### Deps - deps: Bump reqsign to v0.4 (#643) - deps: Remove not used features (#658) - chore(deps): Update criterion requirement from 0.3 to 0.4 (#656) - chore(deps): Update quick-xml requirement from 0.24 to 0.25 (#657) ### Docs - docs: Add docs for ipfs (#649) - docs: Fix typo (#650) - docs: Add docs for ftp services (#655) ### RFCs - RFC-0623: Redis Service (#623) ## [v0.15.0] - 2022-09-05 ### Added - RFC-0599: Blocking API (#599) - feat: Add blocking API in Accessor (#604) - feat: Implement blocking API for fs (#606) - feat: improve observability of `BytesReader` and `DirStreamer` (#603) - feat: Add behavior tests for blocking operations (#607) - feat: Add integration tests for ipfs (#610) - feat: implemented ftp backend (#581) - RFC-0627: Split Capabilities (#627) ### Changed - refactor: Extrace normalize_root functions (#619) - refactor: Extrace build_abs_path and build_rooted_abs_path (#620) - refactor: Extract build_rel_path (#621) - feat: Rename ipfs to ipmfs to better reflect its naming (#629) ## [v0.14.1] - 2022-08-30 ### Added - feat: Add IPFS backend (#481) - refactor: IPFS service cleanup (#590) ### Docs - docs: Add obs in OpenDAL lib docs (#585) ### Fixed - fix(services/s3): If input range is `0..`, don't insert range header (#592) ## [v0.14.0] - 2022-08-28 ### Added - RFC-0554: Write Refactor (#554) - feat: Implement huaweicloud obs service other op support (#557) - feat: Add new operations in Accessor (#564) - feat: Implement obs create and write (#565) - feat(services/s3): Implement Multipart support (#571) - feat: Implement MultipartObject public API (#574) - feat: Implement integration tests for multipart (#575) - feat: Implement presign for write multipart (#576) - test: Add assert of public struct size (#578) - feat: List metadata reuse (#577) - feat: Implement integration test for obs (#572) ### Changed - refactor(ops): Promote ops as a parent mod (#553) - refactor: Implement RFC-0554 Write Refactor (#556) - refactor: Remove all unused qualifications (#560) - refactor: Fix typo in azblob backend (#569) - refactor: change ObjectError's op from &'static str to Operation (#580) ### Deleted - refactor: Remove deprecated APIs (#582) ### Docs - docs: Add docs for obs service (#579) ## [v0.13.1] - 2022-08-22 ### Added - feat: Add walk for BatchOperator (#543) - feat: Mark Scheme non_exhaustive and extendable (#544) - feat: Try to limit the max_connections for http client (#545) - feat: Implement huaweicloud obs service read support (#540) ### Docs - docs: Fix gcs is missing from index (#546) ## [v0.13.0] - 2022-08-17 ### Added - feat: Refactor metrics and hide under feature layers-metrics (#521) - feat(layer): Add TracingLayer support (#523) - feature: Google Cloud Storage support skeleton (#513) - feat: Add LoggingLayer to replace service internal logs (#526) - feat: Implement integration tests for gcs (#532) - docs: Add docs for new layers (#534) - docs: Add docs for gcs backend (#535) ### Changed - refactor: Rewrite retry layer support (#522) ### Fixed - fix: Make ProtocolViolation a retryable error (#528) ## [v0.12.0] - 2022-08-12 ### Added - RFC-0501: New Builder (#501) - feat: Implement RFC-0501 New Builder (#510) ### Changed - feat: Use isahc to replace hyper (#471) - refactor: make parse http error code public (#511) - refactor: Extrace new http error APIs (#515) - refactor: Simplify the error response parse logic (#516) ### Removed - refactor: Remove deprecated struct Metadata (#503) ## [v0.11.4] - 2022-08-02 ### Added - feat: Support using rustls for TLS (#491) ### Changed - feat: try to support epoll (#478) - deps: Lower the requirement of deps (#495) - Revert "feat: try to support epoll" (#496) ### Fixed - fix: Uri encode continuation-token before signing (#494) ### Docs - docs: Add downloads in README (#485) - docs: Update slogan for OpenDAL (#486) ## [v0.11.3] - 2022-07-26 ### Changed - build: Remove not used features (#472) ### Fixed - fix: Disable connection pool as workaround for async runtime hang (#474) ### Dependencies - chore(deps): Bump clap from 3.2.12 to 3.2.15 in /oay (#461) - chore(deps): Bump clap from 3.2.12 to 3.2.15 in /oli (#460) - chore(deps): Update metrics requirement from 0.19.0 to 0.20.0 (#462) - chore(deps): Bump tokio from 1.20.0 to 1.20.1 in /oay (#468) ## [v0.11.2] - 2022-07-19 ### Fixed - fix: Service HTTP deosn't handle dir correctly (#455) - fix: Service HTTP inserted with wrong key (#457) ## [v0.11.1] - 2022-07-19 ### Added - RFC-0438: Multipart (#438) - RFC-0443: Gateway (#443) - feat: Add basic oay support for http (#445) - feat: BytesRange supports parsing from range and content-range (#449) - feat(oay): Implement range support (#450) - feat(services-http): Implement write and delete for testing (#451) ## [v0.11.0] - 2022-07-11 ### Added - feat: derive Deserialize/Serialize for ObjectMetaData (#420) - RFC-0423: Command Line Interface (#423) - feat: optimize range read (#425) - feat(oli): Add basic layout for oli (#426) - RFC-0429: Init From Iter (#429) - feat: Implement RFC-0429 Init From Iter (#432) - feat(oli): Add cp command layout (#428) ### Docs - docs: Update description of OpenDAL (#434) ## [v0.10.0] - 2022-07-04 ### Added - RFC-0409: Accessor Capabilities (#409) - feat: Implement RFC-0409 Accessor Capabilities (#411) - RFC-0413: Presign (#413) - feat: Implement presign support for s3 (#414) ### Docs - docs: Add new RFCs in list (#415) ### Dependencies - chore(deps): Update reqsign requirement from 0.1.1 to 0.2.0 (#412) ## [v0.9.1] - 2022-06-27 ### Added - feat(object): Add ETag support (#381) - feat: Convert retryable hyper errors into Interrupted (#396) ### Changed - build: Exclude docs from publish (#383) - ci: Don't run CI on not needed push (#395) - refactor: Use list for check instead of stat (#399) ### Dependencies - chore(deps): Update size requirement from 0.1.2 to 0.2.0 (#385) - Upgrade dev-dependency `size` to 0.4 (#392) ### Fixed - fix: Special chars not handled correctly (#398) ## [v0.9.0] - 2022-06-14 ### Added - feat: Implement http service support (#368) - feat: Add http_header to handle HTTP header parse (#369) - feat(services/s3): Add virtual host API style support (#374) ### Changed - refactor: Use the same http client across project (#364) - refactor(services/{s3,azblob}): Make sure error response parsed correctly and safely (#375) ### Docs - docs: Add concepts for Accessor, Operator and Object (#354) - docs: Aad docs for batch operations (#363) ## [v0.8.0] - 2022-06-09 ### Added - RFC-0337: Dir Entry (#337) - feat: Implement RFC-0337: Dir Entry (#342) - feat: Add batch operation support (#346) ### Changed - refactor: Rename Metadata to ObjectMetadata for clarify (#339) ### Others - chore(deps): Bump actions/setup-python from 3 to 4 (#343) - chore(deps): Bump amondnet/vercel-action from 20 to 25 (#344) ## [v0.7.3] - 2022-06-03 ### Fixed - fix(services/s3,hdfs): List empty dir should not return itself (#327) - fix(services/hdfs): Root path not cleaned correctly (#330) ## [v0.7.2] - 2022-06-01 ### Added - feat(io_util): Improve debug logging for compress (#310) - feat(services/s3): Add disable_credential_loader support (#317) - feat: Allow check user input (#318) - docs: Add services and features docs (#319) - feat: Add name to object metadata (#304) - fix(io_util/compress): Fix decoder's buf not all consumed (#323) ### Changed - chore(deps): Update metrics requirement from 0.18.1 to 0.19.0 (#314) - docs: Update README to reflect current status (#321) - refactor(object): Make Metadata::name() return &str (#322) ### Fixed - docs: Fix typo in examples (#320) - fix(services): Don't throw error message for stat operation (#324) ## [v0.7.1] - 2022-05-29 ### Fixed - publish: Fix git version not allowed (#306) - fix(io_util/compress): Decompress read exit too early (#308) ## [v0.7.0] - 2022-05-29 ### Added - feat: Add support for blocking decompress_read (#289) - feat: Add check for operator (#290) - docs: Use mdbook to generate documentation (#291) - proposal: Object ID (#293) - feat: Implement operator metadata support (#296) - feat: Implement RFC-0293 Object ID (#298) ### Changed - chore(deps): Update quick-xml requirement from 0.22.0 to 0.23.0 (#286) - feat(io_util): Refactor decompress decoder (#302) - ci: Adopt amondnet/vercel-action (#303) ### Fixed - fix(services/aws): Increase retry times for AWS STS (#299) ## [v0.6.3] - 2022-05-25 ### Added - ci: Add all issues into databend-storage project (#277) - feat(services/s3): Add retry in load_credential (#281) - feat(services): Allow endpoint has trailing slash (#282) - feat(services): Attach more context in error messages (#283) ## [v0.6.2] - 2022-05-12 ### Fixed - fix(azblob): Request URL not construct correctly (#270) ## [v0.6.1] - 2022-05-09 ### Added - feat: Add hdfs scheme (#266) ## [v0.6.0] - 2022-05-07 ### Added - docs: Improve docs to 100% coverage (#246) - RFC-0247: Retryable Error (#247) - feat: Implement retry layers (#249) - feat: Implement retryable errors for azblob and s3 (#254) - feat: Implement hdfs service support (#255) - docs: Add docs for hdfs services (#262) ### Changed - docs: Make sure code examples are formatted (#251) - chore(deps): Update uuid requirement from 0.8.2 to 1.0.0 (#252) - refactor: Remove deprecated modules (#259) ### Fixed - ci: Fix docs build (#260) - fix: HDFS jar not load (#261) ## [v0.5.2] - 2022-04-08 ### Changed - chore: Build all features for docs.rs (#238) - ci: Enable auto dependence upgrade (#239) - chore(deps): Bump actions/checkout from 2 to 3 (#240) - docs: Refactor examples (#241) ### Fixed - fix(services/s3): Endpoint without scheme should also supported (#242) ## [v0.5.1] - 2022-04-08 ### Added - docs: Add behavior docs for create operation (#235) ### Fixed - fix(services/fs): Create on existing dir should succeed (#234) ## [v0.5.0] - 2022-04-07 ### Added - feat: Improve error message (#220) - RFC-0221: Create Dir (#221) - feat: Simplify create API (#225) - feat: Implement decompress read support (#227) - ci: Enable behavior test for azblob (#229) - docs: Add docs for azblob's public structs (#230) ### Changed - refactor: Move op.objects() to o.list() (#224) - refactor: Improve behavior_tests so that cargo test works without --all-features (#231) ### Fixed - fix: Azblob should pass all behavior tests now (#228) ## [v0.4.2] - 2022-04-03 ### Added - feat: Add seekable_reader on Object (#215) ### Fixed - fix: Object last_modified should carry timezone (#217) ## [v0.4.1] - 2022-04-02 ### Added - feat: Export SeekableReader (#212) ## [v0.4.0] - 2022-04-02 **Refer to [Upgrade](./docs/upgrade.md) `From v0.3 to v0.4` section for more upgrade details.** ### Added - feat(services/azblob): Implement list support (#193) - feat: Implement io_util like into_sink and into_stream (#197) - docs: Add docs for all newly added public functions (#199) - feat(io_util): Implement observer for sink and stream (#198) - docs: Add docs for public types (#206) ### Changed - refactor: Make read return BytesStream instead (#192) - RFC-0191: Async Streaming IO (#191) - refactor: New public API design (#201) - refactor: Adopt io::Result instead (#204) - refactor: Rollback changes around async streaming io (#205) - refactor: Refactor behavior tests with macro_rules (#207) ### Fixed - deps: Bump to reqsign to fix s3 url encode issue (#202) ### Removed - RFC-0203: Remove Credential (#203) ## [v0.3.0] - 2022-03-25 ### Added - feat: Add azure blob support (#165) - feat: Add tracing support via minitrace (#175) - feat(service/s3): Implement server side encryption support (#182) ### Changed - chore: Level down some log entry to debug (#181) ### Fixed - fix(service/s3): Endpoint template should be applied if region exists (#180) ## [v0.2.5] - 2022-03-22 ### Added - feat: Adopt quick_xml to parse xml (#164) - test: Add behavior test for not exist object (#166) - feat: Allow user input region (#168) ### Changed - feat: Improve error handle for s3 service (#169) - feat: Read error response for better debugging (#170) - examples: Improve examples for s3 (#171) ## [v0.2.4] - 2022-03-18 ### Added - feat: Add content_md5 and last_modified in metadata (#158) ### Changed - refactor: Say goodbye to aws-s3-sdk (#152) ## [v0.2.3] - 2022-03-14 ### Added - feat: Export BoxedObjectStream so that users can implement Layer (#147) ## [v0.2.2] - 2022-03-14 ### Fixed - services/fs: Refactor via tokio::fs (#142) - fix: Stat root should return a dir object (#143) ## [v0.2.1] - 2022-03-10 ### Added - \*: Implement logging support (#122) - feat(service): Add service memory read support (#121) - services: Add basic metrics (#127) - services: Add full memory support (#134) ### Changed - benches: Refactor to support more read pattern (#126) - services: Refactor into directories (#131) ### Docs - docs: Cover all public types and functions (#128) - docs: Update README (#129) - ci: Generate main docs to (#132) - docs: Enrich README (#133) - Add examples for object (#135) ## [v0.2.0] - 2022-03-08 ### Added - RFC-112: Path Normalization (#112) - examples: Add more examples for services and operations (#113) ### Changed - benches: Refactor to make code more readable (#104) - object: Refactor ObjectMode into enum (#114) ## [v0.1.4] - 2022-03-04 ### Added - services/s3: Implement anonymous read support (#97) - bench: Add parallel_read bench (#100) - services/s3: Add test for anonymous support (#99) ## [v0.1.3] - 2022-03-02 ### Added - RFC and implementations for limited reader (#90) - readers: Implement observe reader support (#92) ### Changed - deps: Bump s3 sdk to 0.8 (#87) - bench: Improve logic (#89) ### New RFCs - [limited_reader](https://github.com/apache/opendal/blob/main/docs/rfcs/0090-limited-reader.md) ## [v0.1.2] - 2022-03-01 ### Changed - object: Polish API for Metadata (#80) ## [v0.1.1] - 2022-03-01 ### Added - RFC and implementation of feature Object Stream (#69) - services/s3: Implement List support (#76) - credential: Add Plain variant to allow more input (#78) ### Changed - backend/s3: Change from lazy_static to once_cell (#62) - backend/s3: Enable test on AWS S3 (#64) ## [v0.1.0] - 2022-02-24 ### Added - docs: Add README for behavior test and ops benchmarks (#53) - RFC-0057: Auto Region (#57) - backend/s3: Implement RFC-57 Auto Region (#59) ### Changed - io: Rename BoxedAsyncRead to BoxedAsyncReader (#55) - \*: Refactor tests (#60) ## [v0.0.5] - 2022-02-23 ### Fixed - io: Remove not used debug print (#48) ## [v0.0.4] - 2022-02-23 ### Added - readers: Allow config prefetch size (#31) - RFC-0041: Object Native API (#41) - \*: Implement RFC-0041 Object Native API (#35) - RFC-0044: Error Handle (#44) - error: Implement RFC-0044 Error Handle (#43) ### Changed - services/fs: Use separate dedicated thread pool instead (#42) ## [v0.0.3] - 2022-02-16 ### Added - benches: Implement benches for ops (#26) ### Changed - services/s3: Don't load_from_env if users already inputs (#23) - readers: Improve seekable performance (#25) ## [v0.0.2] - 2022-02-15 ### Added - tests: Implement behavior tests (#13) - services/s3: Add support for endpoints without scheme (#15) - tests: Implement integration tests for s3 (#18) ### Fixed - services/s3: Allow set endpoint and region while input value is valid (#17) ## v0.0.1 - 2022-02-14 ### Added Hello, OpenDAL! [v0.52.0]: https://github.com/apache/opendal/compare/v0.51.2...v0.52.0 [v0.51.2]: https://github.com/apache/opendal/compare/v0.51.1...v0.51.2 [v0.51.1]: https://github.com/apache/opendal/compare/v0.51.0...v0.51.1 [v0.51.0]: https://github.com/apache/opendal/compare/v0.50.2...v0.51.0 [v0.50.2]: https://github.com/apache/opendal/compare/v0.50.1...v0.50.2 [v0.50.1]: https://github.com/apache/opendal/compare/v0.50.0...v0.50.1 [v0.50.0]: https://github.com/apache/opendal/compare/v0.49.2...v0.50.0 [v0.49.2]: https://github.com/apache/opendal/compare/v0.49.1...v0.49.2 [v0.49.1]: https://github.com/apache/opendal/compare/v0.49.0...v0.49.1 [v0.49.0]: https://github.com/apache/opendal/compare/v0.48.0...v0.49.0 [v0.48.0]: https://github.com/apache/opendal/compare/v0.47.3...v0.48.0 [v0.47.3]: https://github.com/apache/opendal/compare/v0.47.2...v0.47.3 [v0.47.2]: https://github.com/apache/opendal/compare/v0.47.1...v0.47.2 [v0.47.1]: https://github.com/apache/opendal/compare/v0.47.0...v0.47.1 [v0.47.0]: https://github.com/apache/opendal/compare/v0.46.0...v0.47.0 [v0.46.0]: https://github.com/apache/opendal/compare/v0.45.1...v0.46.0 [v0.45.1]: https://github.com/apache/opendal/compare/v0.45.0...v0.45.1 [v0.45.0]: https://github.com/apache/opendal/compare/v0.44.2...v0.45.0 [v0.44.2]: https://github.com/apache/opendal/compare/v0.44.1...v0.44.2 [v0.44.1]: https://github.com/apache/opendal/compare/v0.44.0...v0.44.1 [v0.44.0]: https://github.com/apache/opendal/compare/v0.43.0...v0.44.0 [v0.43.0]: https://github.com/apache/opendal/compare/v0.42.0...v0.43.0 [v0.42.0]: https://github.com/apache/opendal/compare/v0.41.0...v0.42.0 [v0.41.0]: https://github.com/apache/opendal/compare/v0.40.0...v0.41.0 [v0.40.0]: https://github.com/apache/opendal/compare/v0.39.1...v0.40.0 [v0.39.0]: https://github.com/apache/opendal/compare/v0.38.1...v0.39.0 [v0.38.1]: https://github.com/apache/opendal/compare/v0.38.0...v0.38.1 [v0.38.0]: https://github.com/apache/opendal/compare/v0.37.0...v0.38.0 [v0.37.0]: https://github.com/apache/opendal/compare/v0.36.0...v0.37.0 [v0.36.0]: https://github.com/apache/opendal/compare/v0.35.0...v0.36.0 [v0.35.0]: https://github.com/apache/opendal/compare/v0.34.0...v0.35.0 [v0.34.0]: https://github.com/apache/opendal/compare/v0.33.3...v0.34.0 [v0.33.3]: https://github.com/apache/opendal/compare/v0.33.2...v0.33.3 [v0.33.2]: https://github.com/apache/opendal/compare/v0.33.1...v0.33.2 [v0.33.1]: https://github.com/apache/opendal/compare/v0.33.0...v0.33.1 [v0.33.0]: https://github.com/apache/opendal/compare/v0.32.0...v0.33.0 [v0.32.0]: https://github.com/apache/opendal/compare/v0.31.1...v0.32.0 [v0.31.1]: https://github.com/apache/opendal/compare/v0.31.0...v0.31.1 [v0.31.0]: https://github.com/apache/opendal/compare/v0.30.5...v0.31.0 [v0.30.5]: https://github.com/apache/opendal/compare/v0.30.4...v0.30.5 [v0.30.4]: https://github.com/apache/opendal/compare/v0.30.3...v0.30.4 [v0.30.3]: https://github.com/apache/opendal/compare/v0.30.2...v0.30.3 [v0.30.2]: https://github.com/apache/opendal/compare/v0.30.1...v0.30.2 [v0.30.1]: https://github.com/apache/opendal/compare/v0.30.0...v0.30.1 [v0.30.0]: https://github.com/apache/opendal/compare/v0.29.1...v0.30.0 [v0.29.1]: https://github.com/apache/opendal/compare/v0.29.0...v0.29.1 [v0.29.0]: https://github.com/apache/opendal/compare/v0.28.0...v0.29.0 [v0.28.0]: https://github.com/apache/opendal/compare/v0.27.2...v0.28.0 [v0.27.2]: https://github.com/apache/opendal/compare/v0.27.1...v0.27.2 [v0.27.1]: https://github.com/apache/opendal/compare/v0.27.0...v0.27.1 [v0.27.0]: https://github.com/apache/opendal/compare/v0.26.2...v0.27.0 [v0.26.2]: https://github.com/apache/opendal/compare/v0.26.1...v0.26.2 [v0.26.1]: https://github.com/apache/opendal/compare/v0.26.0...v0.26.1 [v0.26.0]: https://github.com/apache/opendal/compare/v0.25.2...v0.26.0 [v0.25.2]: https://github.com/apache/opendal/compare/v0.25.1...v0.25.2 [v0.25.1]: https://github.com/apache/opendal/compare/v0.25.0...v0.25.1 [v0.25.0]: https://github.com/apache/opendal/compare/v0.24.6...v0.25.0 [v0.24.6]: https://github.com/apache/opendal/compare/v0.24.5...v0.24.6 [v0.24.5]: https://github.com/apache/opendal/compare/v0.24.4...v0.24.5 [v0.24.4]: https://github.com/apache/opendal/compare/v0.24.3...v0.24.4 [v0.24.3]: https://github.com/apache/opendal/compare/v0.24.2...v0.24.3 [v0.24.2]: https://github.com/apache/opendal/compare/v0.24.1...v0.24.2 [v0.24.1]: https://github.com/apache/opendal/compare/v0.24.0...v0.24.1 [v0.24.0]: https://github.com/apache/opendal/compare/v0.23.0...v0.24.0 [v0.23.0]: https://github.com/apache/opendal/compare/v0.22.6...v0.23.0 [v0.22.6]: https://github.com/apache/opendal/compare/v0.22.5...v0.22.6 [v0.22.5]: https://github.com/apache/opendal/compare/v0.22.4...v0.22.5 [v0.22.4]: https://github.com/apache/opendal/compare/v0.22.3...v0.22.4 [v0.22.3]: https://github.com/apache/opendal/compare/v0.22.2...v0.22.3 [v0.22.2]: https://github.com/apache/opendal/compare/v0.22.1...v0.22.2 [v0.22.1]: https://github.com/apache/opendal/compare/v0.22.0...v0.22.1 [v0.22.0]: https://github.com/apache/opendal/compare/v0.21.2...v0.22.0 [v0.21.2]: https://github.com/apache/opendal/compare/v0.21.1...v0.21.2 [v0.21.1]: https://github.com/apache/opendal/compare/v0.21.0...v0.21.1 [v0.21.0]: https://github.com/apache/opendal/compare/v0.20.1...v0.21.0 [v0.20.1]: https://github.com/apache/opendal/compare/v0.20.0...v0.20.1 [v0.20.0]: https://github.com/apache/opendal/compare/v0.19.8...v0.20.0 [v0.19.8]: https://github.com/apache/opendal/compare/v0.19.7...v0.19.8 [v0.19.7]: https://github.com/apache/opendal/compare/v0.19.6...v0.19.7 [v0.19.6]: https://github.com/apache/opendal/compare/v0.19.5...v0.19.6 [v0.19.5]: https://github.com/apache/opendal/compare/v0.19.4...v0.19.5 [v0.19.4]: https://github.com/apache/opendal/compare/v0.19.3...v0.19.4 [v0.19.3]: https://github.com/apache/opendal/compare/v0.19.2...v0.19.3 [v0.19.2]: https://github.com/apache/opendal/compare/v0.19.1...v0.19.2 [v0.19.1]: https://github.com/apache/opendal/compare/v0.19.0...v0.19.1 [v0.19.0]: https://github.com/apache/opendal/compare/v0.18.2...v0.19.0 [v0.18.2]: https://github.com/apache/opendal/compare/v0.18.1...v0.18.2 [v0.18.1]: https://github.com/apache/opendal/compare/v0.18.0...v0.18.1 [v0.18.0]: https://github.com/apache/opendal/compare/v0.17.4...v0.18.0 [v0.17.4]: https://github.com/apache/opendal/compare/v0.17.3...v0.17.4 [v0.17.3]: https://github.com/apache/opendal/compare/v0.17.2...v0.17.3 [v0.17.2]: https://github.com/apache/opendal/compare/v0.17.1...v0.17.2 [v0.17.1]: https://github.com/apache/opendal/compare/v0.17.0...v0.17.1 [v0.17.0]: https://github.com/apache/opendal/compare/v0.16.0...v0.17.0 [v0.16.0]: https://github.com/apache/opendal/compare/v0.15.0...v0.16.0 [v0.15.0]: https://github.com/apache/opendal/compare/v0.14.1...v0.15.0 [v0.14.1]: https://github.com/apache/opendal/compare/v0.14.0...v0.14.1 [v0.14.0]: https://github.com/apache/opendal/compare/v0.13.1...v0.14.0 [v0.13.1]: https://github.com/apache/opendal/compare/v0.13.0...v0.13.1 [v0.13.0]: https://github.com/apache/opendal/compare/v0.12.0...v0.13.0 [v0.12.0]: https://github.com/apache/opendal/compare/v0.11.4...v0.12.0 [v0.11.4]: https://github.com/apache/opendal/compare/v0.11.3...v0.11.4 [v0.11.3]: https://github.com/apache/opendal/compare/v0.11.2...v0.11.3 [v0.11.2]: https://github.com/apache/opendal/compare/v0.11.1...v0.11.2 [v0.11.1]: https://github.com/apache/opendal/compare/v0.11.0...v0.11.1 [v0.11.0]: https://github.com/apache/opendal/compare/v0.10.0...v0.11.0 [v0.10.0]: https://github.com/apache/opendal/compare/v0.9.1...v0.10.0 [v0.9.1]: https://github.com/apache/opendal/compare/v0.9.0...v0.9.1 [v0.9.0]: https://github.com/apache/opendal/compare/v0.8.0...v0.9.0 [v0.8.0]: https://github.com/apache/opendal/compare/v0.7.3...v0.8.0 [v0.7.3]: https://github.com/apache/opendal/compare/v0.7.2...v0.7.3 [v0.7.2]: https://github.com/apache/opendal/compare/v0.7.1...v0.7.2 [v0.7.1]: https://github.com/apache/opendal/compare/v0.7.0...v0.7.1 [v0.7.0]: https://github.com/apache/opendal/compare/v0.6.3...v0.7.0 [v0.6.3]: https://github.com/apache/opendal/compare/v0.6.2...v0.6.3 [v0.6.2]: https://github.com/apache/opendal/compare/v0.6.1...v0.6.2 [v0.6.1]: https://github.com/apache/opendal/compare/v0.6.0...v0.6.1 [v0.6.0]: https://github.com/apache/opendal/compare/v0.5.2...v0.6.0 [v0.5.2]: https://github.com/apache/opendal/compare/v0.5.1...v0.5.2 [v0.5.1]: https://github.com/apache/opendal/compare/v0.5.0...v0.5.1 [v0.5.0]: https://github.com/apache/opendal/compare/v0.4.2...v0.5.0 [v0.4.2]: https://github.com/apache/opendal/compare/v0.4.1...v0.4.2 [v0.4.1]: https://github.com/apache/opendal/compare/v0.4.0...v0.4.1 [v0.4.0]: https://github.com/apache/opendal/compare/v0.3.0...v0.4.0 [v0.3.0]: https://github.com/apache/opendal/compare/v0.2.5...v0.3.0 [v0.2.5]: https://github.com/apache/opendal/compare/v0.2.4...v0.2.5 [v0.2.4]: https://github.com/apache/opendal/compare/v0.2.3...v0.2.4 [v0.2.3]: https://github.com/apache/opendal/compare/v0.2.2...v0.2.3 [v0.2.2]: https://github.com/apache/opendal/compare/v0.2.1...v0.2.2 [v0.2.1]: https://github.com/apache/opendal/compare/v0.2.0...v0.2.1 [v0.2.0]: https://github.com/apache/opendal/compare/v0.1.4...v0.2.0 [v0.1.4]: https://github.com/apache/opendal/compare/v0.1.3...v0.1.4 [v0.1.3]: https://github.com/apache/opendal/compare/v0.1.2...v0.1.3 [v0.1.2]: https://github.com/apache/opendal/compare/v0.1.1...v0.1.2 [v0.1.1]: https://github.com/apache/opendal/compare/v0.1.0...v0.1.1 [v0.1.0]: https://github.com/apache/opendal/compare/v0.0.5...v0.1.0 [v0.0.5]: https://github.com/apache/opendal/compare/v0.0.4...v0.0.5 [v0.0.4]: https://github.com/apache/opendal/compare/v0.0.3...v0.0.4 [v0.0.3]: https://github.com/apache/opendal/compare/v0.0.2...v0.0.3 [v0.0.2]: https://github.com/apache/opendal/compare/v0.0.1...v0.0.2 opendal-0.52.0/CONTRIBUTING.md000064400000000000000000000021321046102023000135610ustar 00000000000000# Contributing ## Get Started - `cargo check` to analyze the current package and report errors. - `cargo build` to compile the current package. - `cargo clippy` to catch common mistakes and improve code. - `cargo test` to run unit tests. - `cargo bench` to run benchmark tests. Useful tips: - Check/Build/Test/Clippy all code: `cargo --tests --benches --examples` - Test specific function: `cargo test tests::it::services::fs` ## Tests We have unit tests and behavior tests. ### Unit Tests Unit tests are placed under `src/tests`, organized by mod. To run all unit tests: ```shell cargo test ``` ### Behavior Tests Behavior Tests are used to make sure every service works correctly. ```shell # Setup env cp .env.example .env # Run tests cargo test ``` Please visit [Behavior Test README](./tests/behavior/README.md) for more details. ## Benchmark We use Ops Benchmark Tests to measure every operation's performance on the target platform. ```shell # Setup env cp .env.example .env # Run benches cargo bench ``` Please visit [Ops Benchmark README](./benches/ops/README.md) for more details. opendal-0.52.0/Cargo.lock0000644000006433420000000000100105310ustar # This file is automatically @generated by Cargo. # It is not intended for manual editing. version = 3 [[package]] name = "Inflector" version = "0.11.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3" dependencies = [ "lazy_static", "regex", ] [[package]] name = "addr" version = "0.15.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a93b8a41dbe230ad5087cc721f8d41611de654542180586b315d9f4cf6b72bef" dependencies = [ "psl-types", ] [[package]] name = "addr2line" version = "0.24.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dfbe277e56a376000877090da837660b4427aad530e3028d44e0bffe4f89a1c1" dependencies = [ "gimli", ] [[package]] name = "adler2" version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "512761e0bb2578dd7380c6baaa0f4ce03e84f95e960231d1dec8bf4d7d6e2627" [[package]] name = "aes" version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" dependencies = [ "cfg-if", "cipher", "cpufeatures", ] [[package]] name = "ahash" version = "0.7.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "891477e0c6a8957309ee5c45a6368af3ae14bb510732d2684ffa19af310920f9" dependencies = [ "getrandom 0.2.15", "once_cell", "version_check", ] [[package]] name = "ahash" version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e89da841a80418a9b391ebaea17f5c112ffaaa96f621d2c285b5174da76b9011" dependencies = [ "cfg-if", "getrandom 0.2.15", "once_cell", "version_check", "zerocopy", ] [[package]] name = "aho-corasick" version = "1.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916" dependencies = [ "memchr", ] [[package]] name = "aliasable" version = "0.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "250f629c0161ad8107cf89319e990051fae62832fd343083bea452d93e2205fd" [[package]] name = "aligned-array" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e05c92d086290f52938013f6242ac62bf7d401fab8ad36798a609faa65c3fd2c" dependencies = [ "generic-array", ] [[package]] name = "allocator-api2" version = "0.2.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" [[package]] name = "ammonia" version = "4.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ab99eae5ee58501ab236beb6f20f6ca39be615267b014899c89b2f0bc18a459" dependencies = [ "html5ever", "maplit", "once_cell", "tendril", "url", ] [[package]] name = "android-tzdata" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e999941b234f3131b00bc13c22d06e8c5ff726d1b6318ac7eb276997bbb4fef0" [[package]] name = "android_system_properties" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" dependencies = [ "libc", ] [[package]] name = "anes" version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299" [[package]] name = "anstream" version = "0.6.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8acc5369981196006228e28809f761875c0327210a891e941f4c683b3a99529b" dependencies = [ "anstyle", "anstyle-parse", "anstyle-query", "anstyle-wincon", "colorchoice", "is_terminal_polyfill", "utf8parse", ] [[package]] name = "anstyle" version = "1.0.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "55cc3b69f167a1ef2e161439aa98aed94e6028e5f9a59be9a6ffb47aef1651f9" [[package]] name = "anstyle-parse" version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b2d16507662817a6a20a9ea92df6652ee4f94f914589377d69f3b21bc5798a9" dependencies = [ "utf8parse", ] [[package]] name = "anstyle-query" version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "79947af37f4177cfead1110013d678905c37501914fba0efea834c3fe9a8d60c" dependencies = [ "windows-sys 0.59.0", ] [[package]] name = "anstyle-wincon" version = "3.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2109dbce0e72be3ec00bed26e6a7479ca384ad226efdd66db8fa2e3a38c83125" dependencies = [ "anstyle", "windows-sys 0.59.0", ] [[package]] name = "any_ascii" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ea50b14b7a4b9343f8c627a7a53c52076482bd4bdad0a24fd3ec533ed616cc2c" [[package]] name = "anyhow" version = "1.0.95" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "34ac096ce696dc2fcabef30516bb13c0a68a11d30131d3df6f04711467681b04" [[package]] name = "approx" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f2a05fd1bd10b2527e20a2cd32d8873d115b8b39fe219ee25f42a8aca6ba278" dependencies = [ "num-traits", ] [[package]] name = "approx" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cab112f0a86d568ea0e627cc1d6be74a1e9cd55214684db5561995f6dad897c6" dependencies = [ "num-traits", ] [[package]] name = "arbitrary" version = "1.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dde20b3d026af13f561bdd0f15edf01fc734f0dafcedbaf42bba506a9517f223" [[package]] name = "arc-swap" version = "1.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69f7f8c3906b62b754cd5326047894316021dcfe5a194c8ea52bdd94934a3457" [[package]] name = "argon2" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3c3610892ee6e0cbce8ae2700349fcf8f98adb0dbfbee85aec3c9179d29cc072" dependencies = [ "base64ct", "blake2", "cpufeatures", "password-hash", ] [[package]] name = "arrayref" version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" [[package]] name = "arrayvec" version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "ascii-canvas" version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8824ecca2e851cec16968d54a01dd372ef8f95b244fb84b84e70128be347c3c6" dependencies = [ "term", ] [[package]] name = "async-backtrace" version = "0.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4dcb391558246d27a13f195c1e3a53eda422270fdd452bd57a5aa9c1da1bb198" dependencies = [ "async-backtrace-attributes", "dashmap 5.5.3", "futures", "loom 0.5.6", "once_cell", "pin-project-lite", "rustc-hash 1.1.0", "static_assertions", ] [[package]] name = "async-backtrace-attributes" version = "0.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "affbba0d438add06462a0371997575927bc05052f7ec486e7a4ca405c956c3d7" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "async-channel" version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "81953c529336010edd6d8e358f886d9581267795c61b19475b71314bffa46d35" dependencies = [ "concurrent-queue", "event-listener 2.5.3", "futures-core", ] [[package]] name = "async-channel" version = "2.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "89b47800b0be77592da0afd425cc03468052844aff33b84e33cc696f64e77b6a" dependencies = [ "concurrent-queue", "event-listener-strategy", "futures-core", "pin-project-lite", ] [[package]] name = "async-compat" version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7bab94bde396a3f7b4962e396fdad640e241ed797d4d8d77fc8c237d14c58fc0" dependencies = [ "futures-core", "futures-io", "once_cell", "pin-project-lite", "tokio", ] [[package]] name = "async-executor" version = "1.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "30ca9a001c1e8ba5149f91a74362376cc6bc5b919d92d988668657bd570bdcec" dependencies = [ "async-task", "concurrent-queue", "fastrand", "futures-lite", "slab", ] [[package]] name = "async-global-executor" version = "2.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "05b1b633a2115cd122d73b955eadd9916c18c8f510ec9cd1686404c60ad1c29c" dependencies = [ "async-channel 2.3.1", "async-executor", "async-io", "async-lock", "blocking", "futures-lite", "once_cell", ] [[package]] name = "async-graphql" version = "7.0.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "59fd6bd734afb8b6e4d0f84a3e77305ce0a7ccc60d70f6001cb5e1c3f38d8ff1" dependencies = [ "async-graphql-derive", "async-graphql-parser", "async-graphql-value", "async-stream", "async-trait", "base64 0.22.1", "bytes", "fnv", "futures-timer", "futures-util", "http 1.2.0", "indexmap 2.7.0", "mime", "multer", "num-traits", "pin-project-lite", "regex", "serde", "serde_json", "serde_urlencoded", "static_assertions_next", "thiserror 1.0.69", ] [[package]] name = "async-graphql-derive" version = "7.0.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac38b4dd452d529d6c0248b51df23603f0a875770352e26ae8c346ce6c149b3e" dependencies = [ "Inflector", "async-graphql-parser", "darling", "proc-macro-crate", "proc-macro2", "quote", "strum", "syn 2.0.95", "thiserror 1.0.69", ] [[package]] name = "async-graphql-parser" version = "7.0.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42d271ddda2f55b13970928abbcbc3423cfc18187c60e8769b48f21a93b7adaa" dependencies = [ "async-graphql-value", "pest", "serde", "serde_json", ] [[package]] name = "async-graphql-value" version = "7.0.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "aefe909173a037eaf3281b046dc22580b59a38b765d7b8d5116f2ffef098048d" dependencies = [ "bytes", "indexmap 2.7.0", "serde", "serde_json", ] [[package]] name = "async-io" version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "43a2b323ccce0a1d90b449fd71f2a06ca7faa7c54c2751f06c9bd851fc061059" dependencies = [ "async-lock", "cfg-if", "concurrent-queue", "futures-io", "futures-lite", "parking", "polling", "rustix", "slab", "tracing", "windows-sys 0.59.0", ] [[package]] name = "async-lock" version = "3.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff6e472cdea888a4bd64f342f09b3f50e1886d32afe8df3d663c01140b811b18" dependencies = [ "event-listener 5.4.0", "event-listener-strategy", "pin-project-lite", ] [[package]] name = "async-recursion" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d7d78656ba01f1b93024b7c3a0467f1608e4be67d725749fdcd7d2c7678fd7a2" dependencies = [ "proc-macro2", "quote", "syn 1.0.109", ] [[package]] name = "async-recursion" version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "async-sleep" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c327a532ed3acb8ad885b50bb2ea5fc7c132a396dd990cf855d2825fbdc16c6c" dependencies = [ "futures-util", "tokio", ] [[package]] name = "async-std" version = "1.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c634475f29802fde2b8f0b505b1bd00dfe4df7d4a000f0b36f7671197d5c3615" dependencies = [ "async-channel 1.9.0", "async-global-executor", "async-io", "async-lock", "crossbeam-utils", "futures-channel", "futures-core", "futures-io", "futures-lite", "gloo-timers", "kv-log-macro", "log", "memchr", "once_cell", "pin-project-lite", "pin-utils", "slab", "wasm-bindgen-futures", ] [[package]] name = "async-stream" version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476" dependencies = [ "async-stream-impl", "futures-core", "pin-project-lite", ] [[package]] name = "async-stream-impl" version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "async-task" version = "4.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b75356056920673b02621b35afd0f7dda9306d03c79a30f5c56c44cf256e3de" [[package]] name = "async-tls" version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b2ae3c9eba89d472a0e4fe1dea433df78fbbe63d2b764addaf2ba3a6bde89a5e" dependencies = [ "futures-core", "futures-io", "rustls 0.21.12", "rustls-pemfile 1.0.4", "webpki-roots 0.22.6", ] [[package]] name = "async-trait" version = "0.1.85" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f934833b4b7233644e5848f235df3f57ed8c80f1528a26c3dfa13d2147fa056" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "async_io_stream" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6d7b9decdf35d8908a7e3ef02f64c5e9b1695e230154c0e8de3969142d9b94c" dependencies = [ "futures", "pharos", "rustc_version", ] [[package]] name = "atoi" version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" dependencies = [ "num-traits", ] [[package]] name = "atomic-waker" version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" [[package]] name = "atomic_lib" version = "0.39.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e00b300ec3db6984694cd41cd84a738feeb4f6f916a13aa2ab733074319daecc" dependencies = [ "base64 0.21.7", "rand 0.8.5", "regex", "ring", "serde", "serde_jcs", "serde_json", "tracing", "ulid", "ureq", "url", "urlencoding", ] [[package]] name = "auto-const-array" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "62f7df18977a1ee03650ee4b31b4aefed6d56bac188760b6e37610400fe8d4bb" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "autocfg" version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ace50bade8e6234aa140d9a2f552bbee1db4d353f69b8217bc503490fc1a9f26" [[package]] name = "await-tree" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0c2d7aec54383fa38ac2f9c28435a02f7312f7174e470c7d5566d2b7e17f9a8d" dependencies = [ "coarsetime", "derive_builder", "flexstr", "indextree", "itertools 0.12.1", "parking_lot 0.12.3", "pin-project", "tokio", "tracing", "weak-table", ] [[package]] name = "awaitable" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "70af449c9a763cb655c6a1e5338b42d99c67190824ff90658c1e30be844c0775" dependencies = [ "awaitable-error", "cfg-if", ] [[package]] name = "awaitable-error" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d5b3469636cdf8543cceab175efca534471f36eee12fb8374aba00eb5e7e7f8a" [[package]] name = "axum" version = "0.6.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b829e4e32b91e643de6eafe82b1d90675f5874230191a4ffbc1b336dec4d6bf" dependencies = [ "async-trait", "axum-core 0.3.4", "bitflags 1.3.2", "bytes", "futures-util", "http 0.2.12", "http-body 0.4.6", "hyper 0.14.32", "itoa", "matchit", "memchr", "mime", "percent-encoding", "pin-project-lite", "rustversion", "serde", "sync_wrapper 0.1.2", "tower 0.4.13", "tower-layer", "tower-service", ] [[package]] name = "axum" version = "0.7.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f" dependencies = [ "async-trait", "axum-core 0.4.5", "bytes", "futures-util", "http 1.2.0", "http-body 1.0.1", "http-body-util", "itoa", "matchit", "memchr", "mime", "percent-encoding", "pin-project-lite", "rustversion", "serde", "sync_wrapper 1.0.2", "tower 0.5.2", "tower-layer", "tower-service", ] [[package]] name = "axum-core" version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "759fa577a247914fd3f7f76d62972792636412fbfd634cd452f6a385a74d2d2c" dependencies = [ "async-trait", "bytes", "futures-util", "http 0.2.12", "http-body 0.4.6", "mime", "rustversion", "tower-layer", "tower-service", ] [[package]] name = "axum-core" version = "0.4.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199" dependencies = [ "async-trait", "bytes", "futures-util", "http 1.2.0", "http-body 1.0.1", "http-body-util", "mime", "pin-project-lite", "rustversion", "sync_wrapper 1.0.2", "tower-layer", "tower-service", ] [[package]] name = "backon" version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba5289ec98f68f28dd809fd601059e6aa908bb8f6108620930828283d4ee23d7" dependencies = [ "fastrand", "gloo-timers", "tokio", ] [[package]] name = "backtrace" version = "0.3.74" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8d82cb332cdfaed17ae235a638438ac4d4839913cc2af585c3c6746e8f8bee1a" dependencies = [ "addr2line", "cfg-if", "libc", "miniz_oxide", "object", "rustc-demangle", "windows-targets 0.52.6", ] [[package]] name = "base64" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b41b7ea54a0c9d92199de89e20e58d49f02f8e699814ef3fdf266f6f748d15c7" [[package]] name = "base64" version = "0.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9e1b586273c5702936fe7b7d6896644d8be71e6314cfe09d3167c95f712589e8" [[package]] name = "base64" version = "0.21.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9d297deb1925b89f2ccc13d7635fa0714f12c87adce1c75356b39ca9b7178567" [[package]] name = "base64" version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "base64ct" version = "1.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8c3c1a368f70d6cf7302d78f8f7093da241fb8e8807c05cc9e51a125895a6d5b" [[package]] name = "bb8" version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d89aabfae550a5c44b43ab941844ffcd2e993cb6900b342debf59e9ea74acdb8" dependencies = [ "async-trait", "futures-util", "parking_lot 0.12.3", "tokio", ] [[package]] name = "bcrypt" version = "0.15.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e65938ed058ef47d92cf8b346cc76ef48984572ade631927e9937b5ffc7662c7" dependencies = [ "base64 0.22.1", "blowfish", "getrandom 0.2.15", "subtle", "zeroize", ] [[package]] name = "bincode" version = "1.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b1f45e9417d87227c7a56d22e471c6206462cba514c7590c09aff4cf6d1ddcad" dependencies = [ "serde", ] [[package]] name = "bindgen" version = "0.65.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cfdf7b466f9a4903edc73f95d6d2bcd5baf8ae620638762244d3f60143643cc5" dependencies = [ "bitflags 1.3.2", "cexpr", "clang-sys", "lazy_static", "lazycell", "peeking_take_while", "prettyplease", "proc-macro2", "quote", "regex", "rustc-hash 1.1.0", "shlex", "syn 2.0.95", ] [[package]] name = "bindgen" version = "0.70.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f49d8fed880d473ea71efb9bf597651e77201bdd4893efe54c9e5d65ae04ce6f" dependencies = [ "bitflags 2.6.0", "cexpr", "clang-sys", "itertools 0.13.0", "log", "prettyplease", "proc-macro2", "quote", "regex", "rustc-hash 1.1.0", "shlex", "syn 2.0.95", ] [[package]] name = "bit-set" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0700ddab506f33b20a03b13996eccd309a48e5ff77d0d95926aa0210fb4e95f1" dependencies = [ "bit-vec", ] [[package]] name = "bit-vec" version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb" [[package]] name = "bitflags" version = "1.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" [[package]] name = "bitflags" version = "2.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b048fb63fd8b5923fc5aa7b340d8e156aec7ec02f0c78fa8a6ddc2613f6f71de" dependencies = [ "serde", ] [[package]] name = "bitvec" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c" dependencies = [ "funty", "radium", "tap", "wyz", ] [[package]] name = "blake2" version = "0.10.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" dependencies = [ "digest", ] [[package]] name = "blake3" version = "1.5.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8ee0c1824c4dea5b5f81736aff91bae041d2c07ee1192bec91054e10e3e601e" dependencies = [ "arrayref", "arrayvec", "cc", "cfg-if", "constant_time_eq", ] [[package]] name = "block-buffer" version = "0.10.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" dependencies = [ "generic-array", ] [[package]] name = "block-padding" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93" dependencies = [ "generic-array", ] [[package]] name = "blocking" version = "1.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "703f41c54fc768e63e091340b424302bb1c29ef4aa0c7f10fe849dfb114d29ea" dependencies = [ "async-channel 2.3.1", "async-task", "futures-io", "futures-lite", "piper", ] [[package]] name = "blowfish" version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e412e2cd0f2b2d93e02543ceae7917b3c70331573df19ee046bcbc35e45e87d7" dependencies = [ "byteorder", "cipher", ] [[package]] name = "borsh" version = "1.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2506947f73ad44e344215ccd6403ac2ae18cd8e046e581a441bf8d199f257f03" dependencies = [ "borsh-derive", "cfg_aliases", ] [[package]] name = "borsh-derive" version = "1.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2593a3b8b938bd68373196c9832f516be11fa487ef4ae745eb282e6a56a7244" dependencies = [ "once_cell", "proc-macro-crate", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "bson" version = "2.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "068208f2b6fcfa27a7f1ee37488d2bb8ba2640f68f5475d08e1d9130696aba59" dependencies = [ "ahash 0.8.11", "base64 0.13.1", "bitvec", "hex", "indexmap 2.7.0", "js-sys", "once_cell", "rand 0.8.5", "serde", "serde_bytes", "serde_json", "time", "uuid", ] [[package]] name = "bufsize" version = "1.0.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f8a4e46ce09fc3179be25d4927caef87469d56cca38a6d6b9b1cadc9fc9ab7d" dependencies = [ "bytes", ] [[package]] name = "bumpalo" version = "3.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "79296716171880943b8470b5f8d03aa55eb2e645a4874bdbb28adb49162e012c" [[package]] name = "bytecheck" version = "0.6.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "23cdc57ce23ac53c931e88a43d06d070a6fd142f2617be5855eb75efc9beb1c2" dependencies = [ "bytecheck_derive", "ptr_meta", "simdutf8", ] [[package]] name = "bytecheck_derive" version = "0.6.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3db406d29fbcd95542e92559bed4d8ad92636d1ca8b3b72ede10b4bcc010e659" dependencies = [ "proc-macro2", "quote", "syn 1.0.109", ] [[package]] name = "bytecount" version = "0.6.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5ce89b21cab1437276d2650d57e971f9d548a2d9037cc231abdc0562b97498ce" [[package]] name = "bytemuck" version = "1.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ef657dfab802224e671f5818e9a4935f9b1957ed18e58292690cc39e7a4092a3" [[package]] name = "byteorder" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "325918d6fe32f23b19878fe4b34794ae41fc19ddbe53b10571a4874d44ffd39b" dependencies = [ "serde", ] [[package]] name = "bzip2-sys" version = "0.1.11+1.0.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "736a955f3fa7875102d57c82b8cac37ec45224a07fd32d58f9f7a186b6cd4cdc" dependencies = [ "cc", "libc", "pkg-config", ] [[package]] name = "cacache" version = "13.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5c5063741c7b2e260bbede781cf4679632dd90e2718e99f7715e46824b65670b" dependencies = [ "digest", "either", "futures", "hex", "libc", "memmap2", "miette", "reflink-copy", "serde", "serde_derive", "serde_json", "sha1", "sha2", "ssri", "tempfile", "thiserror 1.0.69", "tokio", "tokio-stream", "walkdir", ] [[package]] name = "camino" version = "1.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b96ec4966b5813e2c0507c1f86115c8c5abaadc3980879c3424042a02fd1ad3" dependencies = [ "serde", ] [[package]] name = "cargo-platform" version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e35af189006b9c0f00a064685c727031e3ed2d8020f7ba284d78cc2671bd36ea" dependencies = [ "serde", ] [[package]] name = "cargo_metadata" version = "0.14.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4acbb09d9ee8e23699b9634375c72795d095bf268439da88562cf9b501f181fa" dependencies = [ "camino", "cargo-platform", "semver", "serde", "serde_json", ] [[package]] name = "cast" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5" [[package]] name = "castaway" version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0abae9be0aaf9ea96a3b1b8b1b55c602ca751eba1b1500220cea4ecbafe7c0d5" dependencies = [ "rustversion", ] [[package]] name = "cbc" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "26b52a9543ae338f279b96b0b9fed9c8093744685043739079ce85cd58f289a6" dependencies = [ "cipher", ] [[package]] name = "cc" version = "1.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a012a0df96dd6d06ba9a1b29d6402d1a5d77c6befd2566afdc26e10603dc93d7" dependencies = [ "jobserver", "libc", "shlex", ] [[package]] name = "cedar-policy" version = "2.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d91e3b10a0f7f2911774d5e49713c4d25753466f9e11d1cd2ec627f8a2dc857" dependencies = [ "cedar-policy-core", "cedar-policy-validator", "itertools 0.10.5", "lalrpop-util", "ref-cast", "serde", "serde_json", "smol_str", "thiserror 1.0.69", ] [[package]] name = "cedar-policy-core" version = "2.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cd2315591c6b7e18f8038f0a0529f254235fd902b6c217aabc04f2459b0d9995" dependencies = [ "either", "ipnet", "itertools 0.10.5", "lalrpop", "lalrpop-util", "lazy_static", "miette", "regex", "rustc_lexer", "serde", "serde_json", "serde_with", "smol_str", "stacker", "thiserror 1.0.69", ] [[package]] name = "cedar-policy-validator" version = "2.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e756e1b2a5da742ed97e65199ad6d0893e9aa4bd6b34be1de9e70bd1e6adc7df" dependencies = [ "cedar-policy-core", "itertools 0.10.5", "serde", "serde_json", "serde_with", "smol_str", "stacker", "thiserror 1.0.69", "unicode-security", ] [[package]] name = "cexpr" version = "0.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766" dependencies = [ "nom", ] [[package]] name = "cfg-if" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" [[package]] name = "cfg_aliases" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" [[package]] name = "chrono" version = "0.4.39" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7e36cc9d416881d2e24f9a963be5fb1cd90966419ac844274161d10488b3e825" dependencies = [ "android-tzdata", "iana-time-zone", "js-sys", "num-traits", "serde", "wasm-bindgen", "windows-targets 0.52.6", ] [[package]] name = "ciborium" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42e69ffd6f0917f5c029256a24d0161db17cea3997d185db0d35926308770f0e" dependencies = [ "ciborium-io", "ciborium-ll", "serde", ] [[package]] name = "ciborium-io" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "05afea1e0a06c9be33d539b876f1ce3692f4afea2cb41f740e7743225ed1c757" [[package]] name = "ciborium-ll" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "57663b653d948a338bfb3eeba9bb2fd5fcfaecb9e199e87e1eda4d9e8b240fd9" dependencies = [ "ciborium-io", "half", ] [[package]] name = "cipher" version = "0.4.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" dependencies = [ "crypto-common", "inout", ] [[package]] name = "clang-sys" version = "1.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4" dependencies = [ "glob", "libc", "libloading", ] [[package]] name = "clap" version = "4.5.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9560b07a799281c7e0958b9296854d6fafd4c5f31444a7e5bb1ad6dde5ccf1bd" dependencies = [ "clap_builder", "clap_derive", ] [[package]] name = "clap_builder" version = "4.5.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "874e0dd3eb68bf99058751ac9712f622e61e6f393a94f7128fa26e3f02f5c7cd" dependencies = [ "anstream", "anstyle", "clap_lex", "strsim", ] [[package]] name = "clap_derive" version = "4.5.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "54b755194d6389280185988721fffba69495eed5ee9feeee9a599b53db80318c" dependencies = [ "heck 0.5.0", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "clap_lex" version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f46ad14479a25103f283c0f10005961cf086d8dc42205bb44c46ac563475dca6" [[package]] name = "coarsetime" version = "0.1.35" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4252bf230cb600c19826a575b31c8c9c84c6f11acfab6dfcad2e941b10b6f8e2" dependencies = [ "libc", "wasix", "wasm-bindgen", ] [[package]] name = "colorchoice" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5b63caa9aa9397e2d9480a9b13673856c78d8ac123288526c37d7839f2a86990" [[package]] name = "combine" version = "4.6.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba5a308b75df32fe02788e748662718f03fde005016435c444eea572398219fd" dependencies = [ "bytes", "futures-core", "memchr", "pin-project-lite", "tokio", "tokio-util", ] [[package]] name = "compio" version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "de9895d3b1b383334e6dd889618d555ecca48988cfd2be47c7ac8a98b0195c90" dependencies = [ "compio-buf", "compio-dispatcher", "compio-driver", "compio-fs", "compio-io", "compio-log", "compio-net", "compio-runtime", "compio-signal", ] [[package]] name = "compio-buf" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d14413106aad7dd931df3c4724110dabd731c81d52ba18edb4f2d57e7beb611b" dependencies = [ "arrayvec", "bytes", "libc", ] [[package]] name = "compio-dispatcher" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c6f9231b6ee942b5e69eda275fc84d72247055aad197c65ddf3c63cd3cb01d60" dependencies = [ "compio-driver", "compio-runtime", "flume", "futures-channel", ] [[package]] name = "compio-driver" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6be49fe37cd203d925e3850522a47f453b4cb98960846be5e4ebae42e26a64c" dependencies = [ "aligned-array", "cfg-if", "compio-buf", "compio-log", "crossbeam-channel", "crossbeam-queue", "futures-util", "io-uring 0.7.3", "libc", "once_cell", "os_pipe", "paste", "polling", "socket2", "windows-sys 0.52.0", ] [[package]] name = "compio-fs" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "36f645c7bd9c1e1ce5b0ca6aa9a77ec3908d2ed9200c6708a72bccd1c3f875c8" dependencies = [ "cfg-if", "compio-buf", "compio-driver", "compio-io", "compio-runtime", "libc", "os_pipe", "widestring", "windows-sys 0.52.0", ] [[package]] name = "compio-io" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db908087365769933042c157adf860e19bff5a8cdb846ec2b5dd03d0dacf7a35" dependencies = [ "compio-buf", "futures-util", "paste", ] [[package]] name = "compio-log" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fc4e560213c1996b618da369b7c9109564b41af9033802ae534465c4ee4e132f" dependencies = [ "tracing", ] [[package]] name = "compio-net" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b75d9bb79502ac1abb73df8a34e83e51efcb805038cf30c1c48827203a4c6b49" dependencies = [ "cfg-if", "compio-buf", "compio-driver", "compio-io", "compio-runtime", "either", "libc", "socket2", "widestring", "windows-sys 0.52.0", ] [[package]] name = "compio-runtime" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b2d856e9017fdde73918cb1a2f15b6e47fe0aeb93d547201a457b12bb2da74a" dependencies = [ "async-task", "cfg-if", "compio-buf", "compio-driver", "compio-log", "crossbeam-queue", "futures-util", "libc", "once_cell", "os_pipe", "scoped-tls", "slab", "smallvec", "socket2", "windows-sys 0.52.0", ] [[package]] name = "compio-signal" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0cc8476edd2311b8d34cef15eddd0f81a3a9d2dc622dbefd154a39171fc6dba8" dependencies = [ "compio-buf", "compio-driver", "compio-runtime", "libc", "once_cell", "os_pipe", "slab", "windows-sys 0.52.0", ] [[package]] name = "concurrent-queue" version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" dependencies = [ "crossbeam-utils", ] [[package]] name = "concurrent_arena" version = "0.1.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ef19a8d27aefd2a86b39ee61a6e83bc98ee97dacc93ce87770adab2413392a6" dependencies = [ "arc-swap", "parking_lot 0.12.3", "triomphe", ] [[package]] name = "const-cstr" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed3d0b5ff30645a68f35ece8cea4556ca14ef8a1651455f789a099a0513532a6" [[package]] name = "const-oid" version = "0.9.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" [[package]] name = "const-random" version = "0.1.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" dependencies = [ "const-random-macro", ] [[package]] name = "const-random-macro" version = "0.1.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" dependencies = [ "getrandom 0.2.15", "once_cell", "tiny-keccak", ] [[package]] name = "constant_time_eq" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6" [[package]] name = "convert_case" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" [[package]] name = "core-foundation" version = "0.9.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f" dependencies = [ "core-foundation-sys", "libc", ] [[package]] name = "core-foundation-sys" version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" [[package]] name = "cpufeatures" version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "16b80225097f2e5ae4e7179dd2266824648f3e2f49d9134d584b76389d31c4c3" dependencies = [ "libc", ] [[package]] name = "crc" version = "3.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69e6e4d7b33a94f0991c26729976b10ebde1d34c3ee82408fb536164fa10d636" dependencies = [ "crc-catalog", ] [[package]] name = "crc-catalog" version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19d374276b40fb8bbdee95aef7c7fa6b5316ec764510eb64b8dd0e2ed0d7e7f5" [[package]] name = "crc16" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "338089f42c427b86394a5ee60ff321da23a5c89c9d89514c829687b26359fcff" [[package]] name = "crc32c" version = "0.6.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3a47af21622d091a8f0fb295b88bc886ac74efcc613efc19f5d0b21de5c89e47" dependencies = [ "rustc_version", ] [[package]] name = "crc32fast" version = "1.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a97769d94ddab943e4510d138150169a2758b5ef3eb191a9ee688de3e23ef7b3" dependencies = [ "cfg-if", ] [[package]] name = "criterion" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2b12d017a929603d80db1831cd3a24082f8137ce19c69e6447f54f5fc8d692f" dependencies = [ "anes", "cast", "ciborium", "clap", "criterion-plot", "futures", "is-terminal", "itertools 0.10.5", "num-traits", "once_cell", "oorandom", "plotters", "rayon", "regex", "serde", "serde_derive", "serde_json", "tinytemplate", "tokio", "walkdir", ] [[package]] name = "criterion-plot" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1" dependencies = [ "cast", "itertools 0.10.5", ] [[package]] name = "crossbeam-channel" version = "0.5.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "06ba6d68e24814cb8de6bb986db8222d3a027d15872cabc0d18817bc3c0e4471" dependencies = [ "crossbeam-utils", ] [[package]] name = "crossbeam-deque" version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51" dependencies = [ "crossbeam-epoch", "crossbeam-utils", ] [[package]] name = "crossbeam-epoch" version = "0.9.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" dependencies = [ "crossbeam-utils", ] [[package]] name = "crossbeam-queue" version = "0.3.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0f58bbc28f91df819d0aa2a2c00cd19754769c2fad90579b3592b1c9ba7a3115" dependencies = [ "crossbeam-utils", ] [[package]] name = "crossbeam-utils" version = "0.8.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" [[package]] name = "crunchy" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7a81dae078cea95a014a339291cec439d2f232ebe854a9d672b796c6afafa9b7" [[package]] name = "crypto-common" version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3" dependencies = [ "generic-array", "typenum", ] [[package]] name = "ctor" version = "0.1.26" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6d2301688392eb071b0bf1a37be05c469d3cc4dbbd95df672fe28ab021e6a096" dependencies = [ "quote", "syn 1.0.109", ] [[package]] name = "ctr" version = "0.9.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" dependencies = [ "cipher", ] [[package]] name = "darling" version = "0.20.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6f63b86c8a8826a49b8c21f08a2d07338eec8d900540f8630dc76284be802989" dependencies = [ "darling_core", "darling_macro", ] [[package]] name = "darling_core" version = "0.20.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "95133861a8032aaea082871032f5815eb9e98cef03fa916ab4500513994df9e5" dependencies = [ "fnv", "ident_case", "proc-macro2", "quote", "strsim", "syn 2.0.95", ] [[package]] name = "darling_macro" version = "0.20.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d336a2a514f6ccccaa3e09b02d41d35330c07ddf03a62165fcec10bb561c7806" dependencies = [ "darling_core", "quote", "syn 2.0.95", ] [[package]] name = "dashmap" version = "5.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "978747c1d849a7d2ee5e8adc0159961c48fb7e5db2f06af6723b80123bb53856" dependencies = [ "cfg-if", "hashbrown 0.14.5", "lock_api", "once_cell", "parking_lot_core 0.9.10", ] [[package]] name = "dashmap" version = "6.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf" dependencies = [ "cfg-if", "crossbeam-utils", "hashbrown 0.14.5", "lock_api", "once_cell", "parking_lot_core 0.9.10", ] [[package]] name = "data-encoding" version = "2.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e8566979429cf69b49a5c740c60791108e86440e8be149bbea4fe54d2c32d6e2" [[package]] name = "der" version = "0.7.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f55bf8e7b65898637379c1b74eb1551107c8294ed26d855ceb9fd1a09cfc9bc0" dependencies = [ "const-oid", "pem-rfc7468", "zeroize", ] [[package]] name = "deranged" version = "0.3.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b42b6fa04a440b495c8b04d0e71b707c585f83cb9cb28cf8cd0d976c315e31b4" dependencies = [ "powerfmt", "serde", ] [[package]] name = "derive-new" version = "0.5.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3418329ca0ad70234b9735dc4ceed10af4df60eff9c8e7b06cb5e520d92c3535" dependencies = [ "proc-macro2", "quote", "syn 1.0.109", ] [[package]] name = "derive-where" version = "1.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "62d671cc41a825ebabc75757b62d3d168c577f9149b2d49ece1dad1f72119d25" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "derive_builder" version = "0.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947" dependencies = [ "derive_builder_macro", ] [[package]] name = "derive_builder_core" version = "0.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8" dependencies = [ "darling", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "derive_builder_macro" version = "0.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core", "syn 2.0.95", ] [[package]] name = "derive_destructure2" version = "0.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "64b697ac90ff296f0fc031ee5a61c7ac31fb9fff50e3fb32873b09223613fc0c" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "derive_more" version = "0.99.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5f33878137e4dafd7fa914ad4e259e18a4e8e532b9617a2d0150262bf53abfce" dependencies = [ "convert_case", "proc-macro2", "quote", "rustc_version", "syn 2.0.95", ] [[package]] name = "des" version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ffdd80ce8ce993de27e9f063a444a4d53ce8e8db4c1f00cc03af5ad5a9867a1e" dependencies = [ "cipher", ] [[package]] name = "deunicode" version = "1.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "339544cc9e2c4dc3fc7149fd630c5f22263a4fdf18a98afd0075784968b5cf00" [[package]] name = "diff" version = "0.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "56254986775e3233ffa9c4d7d3faaf6d36a2c09d30b20687e9f88bc8bafc16c8" [[package]] name = "digest" version = "0.10.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer", "const-oid", "crypto-common", "subtle", ] [[package]] name = "dirs-next" version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b98cf8ebf19c3d1b223e151f99a4f9f0690dca41414773390fc824184ac833e1" dependencies = [ "cfg-if", "dirs-sys-next", ] [[package]] name = "dirs-sys-next" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ebda144c4fe02d1f7ea1a7d9641b6fc6b580adcfa024ae48797ecdeb6825b4d" dependencies = [ "libc", "redox_users", "winapi", ] [[package]] name = "displaydoc" version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "dlv-list" version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f" dependencies = [ "const-random", ] [[package]] name = "dmp" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bfaa1135a34d26e5cc5b4927a8935af887d4f30a5653a797c33b9a4222beb6d9" dependencies = [ "urlencoding", ] [[package]] name = "dns-lookup" version = "2.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5766087c2235fec47fafa4cfecc81e494ee679d0fd4a59887ea0919bfb0e4fc" dependencies = [ "cfg-if", "libc", "socket2", "windows-sys 0.48.0", ] [[package]] name = "doc-comment" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fea41bba32d969b513997752735605054bc0dfa92b4c56bf1189f2e174be7a10" [[package]] name = "dotenvy" version = "0.15.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1aaf95b3e5c8f23aa320147307562d361db0ae0d51242340f558153b4eb2439b" [[package]] name = "dtoa" version = "1.0.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dcbb2bf8e87535c23f7a8a321e364ce21462d0ff10cb6407820e8e96dfff6653" [[package]] name = "earcutr" version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "79127ed59a85d7687c409e9978547cffb7dc79675355ed22da6b66fd5f6ead01" dependencies = [ "itertools 0.11.0", "num-traits", ] [[package]] name = "either" version = "1.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "60b1af1c220855b6ceac025d3f6ecdd2b7c4894bfe9cd9bda4fbb4bc7c0d4cf0" dependencies = [ "serde", ] [[package]] name = "ena" version = "0.14.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d248bdd43ce613d87415282f69b9bb99d947d290b10962dd6c56233312c2ad5" dependencies = [ "log", ] [[package]] name = "encoding_rs" version = "0.8.35" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3" dependencies = [ "cfg-if", ] [[package]] name = "endian-type" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c34f04666d835ff5d62e058c3995147c06f42fe86ff053337632bca83e42702d" [[package]] name = "enum-as-inner" version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a1e6a265c649f3f5979b601d26f1d05ada116434c87741c9493cb56218f76cbc" dependencies = [ "heck 0.5.0", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "equivalent" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5443807d6dff69373d433ab9ef5378ad8df50ca6298caf15de6e52e24aaf54d5" [[package]] name = "errno" version = "0.3.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "33d852cb9b869c2a9b3df2f71a3074817f01e1844f839a144f5fcef059a4eb5d" dependencies = [ "libc", "windows-sys 0.59.0", ] [[package]] name = "error-chain" version = "0.12.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2d2f06b9cac1506ece98fe3231e3cc9c4410ec3d5b1f24ae1c8946f0742cdefc" dependencies = [ "version_check", ] [[package]] name = "escape8259" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5692dd7b5a1978a5aeb0ce83b7655c58ca8efdcb79d21036ea249da95afec2c6" [[package]] name = "etcd-client" version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39bde3ce50a626efeb1caa9ab1083972d178bebb55ca627639c8ded507dfcbde" dependencies = [ "http 1.2.0", "prost 0.13.4", "tokio", "tokio-stream", "tonic 0.12.3", "tonic-build", "tower 0.4.13", "tower-service", ] [[package]] name = "etcetera" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "136d1b5283a1ab77bd9257427ffd09d8667ced0570b6f938942bc7568ed5b943" dependencies = [ "cfg-if", "home", "windows-sys 0.48.0", ] [[package]] name = "event-listener" version = "2.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0206175f82b8d6bf6652ff7d71a1e27fd2e4efde587fd368662814d6ec1d9ce0" [[package]] name = "event-listener" version = "5.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3492acde4c3fc54c845eaab3eed8bd00c7a7d881f78bfc801e43a93dec1331ae" dependencies = [ "concurrent-queue", "parking", "pin-project-lite", ] [[package]] name = "event-listener-strategy" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3c3e4e0dd3673c1139bf041f3008816d9cf2946bbfac2945c09e523b8d7b05b2" dependencies = [ "event-listener 5.4.0", "pin-project-lite", ] [[package]] name = "fail" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3be3c61c59fdc91f5dbc3ea31ee8623122ce80057058be560654c5d410d181a6" dependencies = [ "lazy_static", "log", "rand 0.7.3", ] [[package]] name = "fastrace" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5787c4baf9b6add08d3cbd45e818e1fdee548bf875f2ed15d8f1b65df24d3119" dependencies = [ "fastrace-macro", "minstant", "once_cell", "parking_lot 0.12.3", "pin-project", "rand 0.8.5", "rtrb", ] [[package]] name = "fastrace-jaeger" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b1458f1184af9f3eeb1e5d9af29be4194c0910964257d477601d66f4cf52e67e" dependencies = [ "fastrace", "log", "thrift_codec", ] [[package]] name = "fastrace-macro" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3141a07a966757217b75f231f59359ee7336170938644ecca7528fe4b3399e3b" dependencies = [ "proc-macro-error2", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "fastrand" version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" [[package]] name = "fixedbitset" version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ce7134b9999ecaf8bcd65542e436736ef32ddca1b3e06094cb6ec5755203b80" [[package]] name = "flate2" version = "1.0.35" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c936bfdafb507ebbf50b8074c54fa31c5be9a1e7e5f467dd659697041407d07c" dependencies = [ "crc32fast", "miniz_oxide", ] [[package]] name = "flexstr" version = "0.9.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4d50aef14619d336a54fca5a592d952eb39037b1a1e7e6afd9f91c892ac7ef65" dependencies = [ "static_assertions", ] [[package]] name = "float_next_after" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8bf7cc16383c4b8d58b9905a8509f02926ce3058053c056376248d958c9df1e8" [[package]] name = "flume" version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "da0e4dd2a88388a1f4ccc7c9ce104604dab68d9f408dc34cd45823d5a9069095" dependencies = [ "futures-core", "futures-sink", "nanorand", "spin", ] [[package]] name = "fnv" version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" [[package]] name = "foldhash" version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a0d2fde1f7b3d48b8395d5f2de76c18a528bd6a9cdde438df747bfcba3e05d6f" [[package]] name = "foreign-types" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1" dependencies = [ "foreign-types-shared", ] [[package]] name = "foreign-types-shared" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b" [[package]] name = "form_urlencoded" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e13624c2627564efccf4934284bdd98cbaa14e79b0b5a141218e507b3a823456" dependencies = [ "percent-encoding", ] [[package]] name = "foundationdb" version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c30b4254cb4c01b50a74dad6aa48e17666726300b3adddb3c959a7e852f286bd" dependencies = [ "async-recursion 1.1.1", "async-trait", "foundationdb-gen", "foundationdb-macros", "foundationdb-sys", "futures", "memchr", "rand 0.8.5", "serde", "serde_bytes", "serde_json", "static_assertions", "uuid", ] [[package]] name = "foundationdb-gen" version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "93b9deedf92107a1076e518d60931b6ed1a632ae0c51e7b491bfd42edb4148ce" dependencies = [ "xml-rs", ] [[package]] name = "foundationdb-macros" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "afc84a5ff0dba78222551017f5625f3365aa09551c78cfaa44136fc6818c2611" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", "try_map", ] [[package]] name = "foundationdb-sys" version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3bae14dba30b8dcc4905a9189ebb18bc9db9744ef0ad8f2b94ef00d21e176964" dependencies = [ "bindgen 0.70.1", "libc", ] [[package]] name = "fs2" version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213" dependencies = [ "libc", "winapi", ] [[package]] name = "fst" version = "0.4.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7ab85b9b05e3978cc9a9cf8fea7f01b494e1a09ed3037e16ba39edc7a29eb61a" [[package]] name = "funty" version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" [[package]] name = "futf" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df420e2e84819663797d1ec6544b13c5be84629e7bb00dc960d6917db2987843" dependencies = [ "mac", "new_debug_unreachable", ] [[package]] name = "futures" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "65bc07b1a8bc7c85c5f2e110c476c7389b4554ba72af57d8445ea63a576b0876" dependencies = [ "futures-channel", "futures-core", "futures-executor", "futures-io", "futures-sink", "futures-task", "futures-util", ] [[package]] name = "futures-channel" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2dff15bf788c671c1934e366d07e30c1814a8ef514e1af724a602e8a2fbe1b10" dependencies = [ "futures-core", "futures-sink", ] [[package]] name = "futures-core" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "05f29059c0c2090612e8d742178b0580d2dc940c837851ad723096f87af6663e" [[package]] name = "futures-executor" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e28d1d997f585e54aebc3f97d39e72338912123a67330d723fdbb564d646c9f" dependencies = [ "futures-core", "futures-task", "futures-util", ] [[package]] name = "futures-intrusive" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d930c203dd0b6ff06e0201a4a2fe9149b43c684fd4420555b26d21b1a02956f" dependencies = [ "futures-core", "lock_api", "parking_lot 0.12.3", ] [[package]] name = "futures-io" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9e5c1b78ca4aae1ac06c48a526a655760685149f0d465d21f37abfe57ce075c6" [[package]] name = "futures-lite" version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cef40d21ae2c515b51041df9ed313ed21e572df340ea58a922a0aefe7e8891a1" dependencies = [ "fastrand", "futures-core", "futures-io", "parking", "pin-project-lite", ] [[package]] name = "futures-macro" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "futures-sink" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e575fab7d1e0dcb8d0c7bcf9a63ee213816ab51902e6d244a95819acacf1d4f7" [[package]] name = "futures-task" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f90f7dce0722e95104fcb095585910c0977252f286e354b5e3bd38902cd99988" [[package]] name = "futures-timer" version = "3.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24" [[package]] name = "futures-util" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9fa08315bb612088cc391249efdc3bc77536f16c91f6cf495e6fbe85b20a4a81" dependencies = [ "futures-channel", "futures-core", "futures-io", "futures-macro", "futures-sink", "futures-task", "memchr", "pin-project-lite", "pin-utils", "slab", ] [[package]] name = "fuzzy-matcher" version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "54614a3312934d066701a80f20f15fa3b56d67ac7722b39eea5b4c9dd1d66c94" dependencies = [ "thread_local", ] [[package]] name = "fxhash" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c31b6d751ae2c7f11320402d34e41349dd1016f8d5d45e48c4312bc8625af50c" dependencies = [ "byteorder", ] [[package]] name = "g2gen" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc3e32f911a41e073b8492473c3595a043e1369ab319a2dbf8c89b1fea06457c" dependencies = [ "g2poly", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "g2p" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1a9afa6efed9af3a5a68ba066429c1497c299d4eafbd948fe630df47a8f2d29f" dependencies = [ "g2gen", "g2poly", ] [[package]] name = "g2poly" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0fd8b261ccf00df8c5cc60c082bb7d7aa64c33a433cfcc091ca244326c924b2c" [[package]] name = "generator" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5cc16584ff22b460a382b7feec54b23d2908d858152e5739a120b949293bd74e" dependencies = [ "cc", "libc", "log", "rustversion", "windows 0.48.0", ] [[package]] name = "generator" version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cc6bd114ceda131d3b1d665eba35788690ad37f5916457286b32ab6fd3c438dd" dependencies = [ "cfg-if", "libc", "log", "rustversion", "windows 0.58.0", ] [[package]] name = "generic-array" version = "0.14.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" dependencies = [ "typenum", "version_check", ] [[package]] name = "geo" version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f811f663912a69249fa620dcd2a005db7254529da2d8a0b23942e81f47084501" dependencies = [ "earcutr", "float_next_after", "geo-types", "geographiclib-rs", "log", "num-traits", "robust", "rstar", "serde", "spade", ] [[package]] name = "geo-types" version = "0.7.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6f47c611187777bbca61ea7aba780213f5f3441fd36294ab333e96cfa791b65" dependencies = [ "approx 0.5.1", "arbitrary", "num-traits", "rstar", "serde", ] [[package]] name = "geographiclib-rs" version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e5ed84f8089c70234b0a8e0aedb6dc733671612ddc0d37c6066052f9781960" dependencies = [ "libm", ] [[package]] name = "getrandom" version = "0.1.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8fc3cb4d91f53b50155bdcfd23f6a4c39ae1969c2ae85982b135750cccaf5fce" dependencies = [ "cfg-if", "libc", "wasi 0.9.0+wasi-snapshot-preview1", ] [[package]] name = "getrandom" version = "0.2.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c4567c8db10ae91089c99af84c68c38da3ec2f087c3f82960bcdbf3656b6f4d7" dependencies = [ "cfg-if", "js-sys", "libc", "wasi 0.11.0+wasi-snapshot-preview1", "wasm-bindgen", ] [[package]] name = "ghac" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a10bd5b898cac1a4de4a882a754b2ccaafead449348cfb420b48cd5c00ffd08b" dependencies = [ "prost 0.13.4", ] [[package]] name = "ghost" version = "0.1.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39b697dbd8bfcc35d0ee91698aaa379af096368ba8837d279cc097b276edda45" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "gimli" version = "0.31.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "07e28edb80900c19c28f1072f2e8aeca7fa06b23cd4169cefe1af5aa3260783f" [[package]] name = "glob" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a8d1add55171497b4705a648c6b583acafb01d58050a51727785f0b2c8e0a2b2" [[package]] name = "gloo-timers" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbb143cf96099802033e0d4f4963b19fd2e0b728bcf076cd9cf7f6634f092994" dependencies = [ "futures-channel", "futures-core", "js-sys", "wasm-bindgen", ] [[package]] name = "governor" version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68a7f542ee6b35af73b06abc0dad1c1bae89964e4e253bc4b587b91c9637867b" dependencies = [ "cfg-if", "dashmap 5.5.3", "futures", "futures-timer", "no-std-compat", "nonzero_ext", "parking_lot 0.12.3", "portable-atomic", "quanta", "rand 0.8.5", "smallvec", "spinning_top", ] [[package]] name = "h2" version = "0.3.26" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "81fe527a889e1532da5c525686d96d4c2e74cdd345badf8dfef9f6b39dd5f5e8" dependencies = [ "bytes", "fnv", "futures-core", "futures-sink", "futures-util", "http 0.2.12", "indexmap 2.7.0", "slab", "tokio", "tokio-util", "tracing", ] [[package]] name = "h2" version = "0.4.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccae279728d634d083c00f6099cb58f01cc99c145b84b8be2f6c74618d79922e" dependencies = [ "atomic-waker", "bytes", "fnv", "futures-core", "futures-sink", "http 1.2.0", "indexmap 2.7.0", "slab", "tokio", "tokio-util", "tracing", ] [[package]] name = "half" version = "2.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6dd08c532ae367adf81c312a4580bc67f1d0fe8bc9c460520283f4c0ff277888" dependencies = [ "cfg-if", "crunchy", ] [[package]] name = "hash32" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" dependencies = [ "byteorder", ] [[package]] name = "hashbrown" version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" dependencies = [ "ahash 0.7.8", ] [[package]] name = "hashbrown" version = "0.14.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" dependencies = [ "ahash 0.8.11", "allocator-api2", ] [[package]] name = "hashbrown" version = "0.15.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bf151400ff0baff5465007dd2f3e717f3fe502074ca563069ce3a6629d07b289" dependencies = [ "allocator-api2", "equivalent", "foldhash", ] [[package]] name = "hashlink" version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7382cf6263419f2d8df38c55d7da83da5c18aef87fc7a7fc1fb1e344edfe14c1" dependencies = [ "hashbrown 0.15.2", ] [[package]] name = "hdfs-native" version = "0.10.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4e72db0dfc43c1e6b7ef6d34f6d38eff079cbd30dbc18924b27108793e47893c" dependencies = [ "aes", "base64 0.21.7", "bitflags 2.6.0", "bytes", "cbc", "chrono", "cipher", "crc", "ctr", "des", "dns-lookup", "futures", "g2p", "hex", "hmac", "libc", "libloading", "log", "md-5", "num-traits", "once_cell", "prost 0.12.6", "prost-types 0.12.6", "rand 0.8.5", "regex", "roxmltree", "socket2", "thiserror 1.0.69", "tokio", "url", "uuid", "whoami", ] [[package]] name = "hdfs-sys" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "33e2d5cefba2d51a26b44d2a493f963a32725a0f6593c91be4a610ad449c49cb" dependencies = [ "cc", "java-locator", ] [[package]] name = "hdrs" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f7c42a693bfe5dc8fcad1f24044c5ec355c5f157b8ce63c7d62f51cecbc7878d" dependencies = [ "blocking", "errno", "futures", "hdfs-sys", "libc", "log", ] [[package]] name = "heapless" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" dependencies = [ "hash32", "stable_deref_trait", ] [[package]] name = "heck" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8" [[package]] name = "heck" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" [[package]] name = "hermit-abi" version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d231dfb89cfffdbc30e7fc41579ed6066ad03abda9e567ccafae602b97ec5024" [[package]] name = "hermit-abi" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fbf6a919d6cf397374f7dfeeea91d974c7c0a7221d0d0f4f20d859d329e53fcc" [[package]] name = "hex" version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" [[package]] name = "hickory-proto" version = "0.24.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "447afdcdb8afb9d0a852af6dc65d9b285ce720ed7a59e42a8bf2e931c67bc1b5" dependencies = [ "async-trait", "cfg-if", "data-encoding", "enum-as-inner", "futures-channel", "futures-io", "futures-util", "idna", "ipnet", "once_cell", "rand 0.8.5", "thiserror 1.0.69", "tinyvec", "tokio", "tracing", "url", ] [[package]] name = "hickory-resolver" version = "0.24.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0a2e2aba9c389ce5267d31cf1e4dace82390ae276b0b364ea55630b1fa1b44b4" dependencies = [ "cfg-if", "futures-util", "hickory-proto", "ipconfig", "lru-cache", "once_cell", "parking_lot 0.12.3", "rand 0.8.5", "resolv-conf", "smallvec", "thiserror 1.0.69", "tokio", "tracing", ] [[package]] name = "hkdf" version = "0.12.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" dependencies = [ "hmac", ] [[package]] name = "hmac" version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" dependencies = [ "digest", ] [[package]] name = "home" version = "0.5.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "589533453244b0995c858700322199b2becb13b627df2851f64a2775d024abcf" dependencies = [ "windows-sys 0.59.0", ] [[package]] name = "hostname" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3c731c3e10504cc8ed35cfe2f1db4c9274c3d35fa486e3b31df46f068ef3e867" dependencies = [ "libc", "match_cfg", "winapi", ] [[package]] name = "html5ever" version = "0.27.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c13771afe0e6e846f1e67d038d4cb29998a6779f93c809212e4e9c32efd244d4" dependencies = [ "log", "mac", "markup5ever", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "http" version = "0.2.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "601cbb57e577e2f5ef5be8e7b83f0f63994f25aa94d673e54a92d5c516d101f1" dependencies = [ "bytes", "fnv", "itoa", ] [[package]] name = "http" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f16ca2af56261c99fba8bac40a10251ce8188205a4c448fbb745a2e4daa76fea" dependencies = [ "bytes", "fnv", "itoa", ] [[package]] name = "http-body" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7ceab25649e9960c0311ea418d17bee82c0dcec1bd053b5f9a66e265a693bed2" dependencies = [ "bytes", "http 0.2.12", "pin-project-lite", ] [[package]] name = "http-body" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" dependencies = [ "bytes", "http 1.2.0", ] [[package]] name = "http-body-util" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "793429d76616a256bcb62c2a2ec2bed781c8307e797e2598c50010f2bee2544f" dependencies = [ "bytes", "futures-util", "http 1.2.0", "http-body 1.0.1", "pin-project-lite", ] [[package]] name = "httparse" version = "1.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d71d3574edd2771538b901e6549113b4006ece66150fb69c0fb6d9a2adae946" [[package]] name = "httpdate" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" [[package]] name = "humantime" version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9a3a5bfb195931eeb336b2a7b4d761daec841b97f947d34394601737a7bba5e4" [[package]] name = "hyper" version = "0.14.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "41dfc780fdec9373c01bae43289ea34c972e40ee3c9f6b3c8801a35f35586ce7" dependencies = [ "bytes", "futures-channel", "futures-core", "futures-util", "h2 0.3.26", "http 0.2.12", "http-body 0.4.6", "httparse", "httpdate", "itoa", "pin-project-lite", "socket2", "tokio", "tower-service", "tracing", "want", ] [[package]] name = "hyper" version = "1.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "256fb8d4bd6413123cc9d91832d78325c48ff41677595be797d90f42969beae0" dependencies = [ "bytes", "futures-channel", "futures-util", "h2 0.4.7", "http 1.2.0", "http-body 1.0.1", "httparse", "httpdate", "itoa", "pin-project-lite", "smallvec", "tokio", "want", ] [[package]] name = "hyper-rustls" version = "0.27.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2d191583f3da1305256f22463b9bb0471acad48a4e534a5218b9963e9c1f59b2" dependencies = [ "futures-util", "http 1.2.0", "hyper 1.5.2", "hyper-util", "rustls 0.23.20", "rustls-pki-types", "tokio", "tokio-rustls 0.26.1", "tower-service", "webpki-roots 0.26.7", ] [[package]] name = "hyper-timeout" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbb958482e8c7be4bc3cf272a766a2b0bf1a6755e7a6ae777f017a31d11b13b1" dependencies = [ "hyper 0.14.32", "pin-project-lite", "tokio", "tokio-io-timeout", ] [[package]] name = "hyper-timeout" version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0" dependencies = [ "hyper 1.5.2", "hyper-util", "pin-project-lite", "tokio", "tower-service", ] [[package]] name = "hyper-util" version = "0.1.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df2dcfbe0677734ab2f3ffa7fa7bfd4706bfdc1ef393f2ee30184aed67e631b4" dependencies = [ "bytes", "futures-channel", "futures-util", "http 1.2.0", "http-body 1.0.1", "hyper 1.5.2", "pin-project-lite", "socket2", "tokio", "tower-service", "tracing", ] [[package]] name = "iana-time-zone" version = "0.1.61" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "235e081f3925a06703c2d0117ea8b91f042756fd6e7a6e5d901e8ca1a996b220" dependencies = [ "android_system_properties", "core-foundation-sys", "iana-time-zone-haiku", "js-sys", "wasm-bindgen", "windows-core 0.52.0", ] [[package]] name = "iana-time-zone-haiku" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" dependencies = [ "cc", ] [[package]] name = "icu_collections" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db2fa452206ebee18c4b5c2274dbf1de17008e874b4dc4f0aea9d01ca79e4526" dependencies = [ "displaydoc", "yoke", "zerofrom", "zerovec", ] [[package]] name = "icu_locid" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "13acbb8371917fc971be86fc8057c41a64b521c184808a698c02acc242dbf637" dependencies = [ "displaydoc", "litemap", "tinystr", "writeable", "zerovec", ] [[package]] name = "icu_locid_transform" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "01d11ac35de8e40fdeda00d9e1e9d92525f3f9d887cdd7aa81d727596788b54e" dependencies = [ "displaydoc", "icu_locid", "icu_locid_transform_data", "icu_provider", "tinystr", "zerovec", ] [[package]] name = "icu_locid_transform_data" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fdc8ff3388f852bede6b579ad4e978ab004f139284d7b28715f773507b946f6e" [[package]] name = "icu_normalizer" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19ce3e0da2ec68599d193c93d088142efd7f9c5d6fc9b803774855747dc6a84f" dependencies = [ "displaydoc", "icu_collections", "icu_normalizer_data", "icu_properties", "icu_provider", "smallvec", "utf16_iter", "utf8_iter", "write16", "zerovec", ] [[package]] name = "icu_normalizer_data" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8cafbf7aa791e9b22bec55a167906f9e1215fd475cd22adfcf660e03e989516" [[package]] name = "icu_properties" version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "93d6020766cfc6302c15dbbc9c8778c37e62c14427cb7f6e601d849e092aeef5" dependencies = [ "displaydoc", "icu_collections", "icu_locid_transform", "icu_properties_data", "icu_provider", "tinystr", "zerovec", ] [[package]] name = "icu_properties_data" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "67a8effbc3dd3e4ba1afa8ad918d5684b8868b3b26500753effea8d2eed19569" [[package]] name = "icu_provider" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ed421c8a8ef78d3e2dbc98a973be2f3770cb42b606e3ab18d6237c4dfde68d9" dependencies = [ "displaydoc", "icu_locid", "icu_provider_macros", "stable_deref_trait", "tinystr", "writeable", "yoke", "zerofrom", "zerovec", ] [[package]] name = "icu_provider_macros" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ec89e9337638ecdc08744df490b221a7399bf8d164eb52a665454e60e075ad6" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "ident_case" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39" [[package]] name = "idna" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "686f825264d630750a544639377bae737628043f20d38bbc029e8f29ea968a7e" dependencies = [ "idna_adapter", "smallvec", "utf8_iter", ] [[package]] name = "idna_adapter" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "daca1df1c957320b2cf139ac61e7bd64fed304c5040df000a745aa1de3b4ef71" dependencies = [ "icu_normalizer", "icu_properties", ] [[package]] name = "indexmap" version = "1.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99" dependencies = [ "autocfg", "hashbrown 0.12.3", "serde", ] [[package]] name = "indexmap" version = "2.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "62f822373a4fe84d4bb149bf54e584a7f4abec90e072ed49cda0edea5b95471f" dependencies = [ "equivalent", "hashbrown 0.15.2", "serde", ] [[package]] name = "indextree" version = "4.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "91f3e68a01402c3404bfb739079f38858325bc7ad775b07922278a8a415b1a3f" dependencies = [ "indextree-macros", ] [[package]] name = "indextree-macros" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "477e2e7ec7379407656293ff74902caea786a1dda427ca1f84b923c4fdeb7659" dependencies = [ "either", "itertools 0.13.0", "proc-macro2", "quote", "strum", "syn 2.0.95", "thiserror 1.0.69", ] [[package]] name = "inout" version = "0.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a0c10553d664a4d0bcff9f4215d0aac67a639cc68ef660840afe309b807bc9f5" dependencies = [ "block-padding", "generic-array", ] [[package]] name = "instant" version = "0.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" dependencies = [ "cfg-if", ] [[package]] name = "io-uring" version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "595a0399f411a508feb2ec1e970a4a30c249351e30208960d58298de8660b0e5" dependencies = [ "bitflags 1.3.2", "libc", ] [[package]] name = "io-uring" version = "0.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c5d5b4a5e02a58296749114728ea3644f9a4cd5669c243896e445b90bd299ad6" dependencies = [ "bitflags 2.6.0", "cfg-if", "libc", ] [[package]] name = "ipconfig" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b58db92f96b720de98181bbbe63c831e87005ab460c1bf306eb2622b4707997f" dependencies = [ "socket2", "widestring", "windows-sys 0.48.0", "winreg", ] [[package]] name = "ipnet" version = "2.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ddc24109865250148c2e0f3d25d4f0f479571723792d3802153c60922a4fb708" [[package]] name = "is-terminal" version = "0.4.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "261f68e344040fbd0edea105bef17c66edf46f984ddb1115b775ce31be948f4b" dependencies = [ "hermit-abi 0.4.0", "libc", "windows-sys 0.52.0", ] [[package]] name = "is_terminal_polyfill" version = "1.70.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7943c866cc5cd64cbc25b2e01621d07fa8eb2a1a23160ee81ce38704e97b8ecf" [[package]] name = "itertools" version = "0.10.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b0fd2260e829bddf4cb6ea802289de2f86d6a7a690192fbe91b3f46e0f2c8473" dependencies = [ "either", ] [[package]] name = "itertools" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57" dependencies = [ "either", ] [[package]] name = "itertools" version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba291022dbbd398a455acf126c1e341954079855bc60dfdda641363bd6922569" dependencies = [ "either", ] [[package]] name = "itertools" version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" dependencies = [ "either", ] [[package]] name = "itoa" version = "1.0.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d75a2a4b1b190afb6f5425f10f6a8f959d2ea0b9c2b1d79553551850539e4674" [[package]] name = "java-locator" version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6f25f28894af6a5dd349ed5ec46e178654e75f62edb6717ac74007102a57deb5" dependencies = [ "glob", ] [[package]] name = "jobserver" version = "0.1.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "48d1dbcbbeb6a7fec7e059840aa538bd62aaccf972c7346c4d9d2059312853d0" dependencies = [ "libc", ] [[package]] name = "js-sys" version = "0.3.76" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6717b6b5b077764fb5966237269cb3c64edddde4b14ce42647430a78ced9e7b7" dependencies = [ "once_cell", "wasm-bindgen", ] [[package]] name = "jsonwebtoken" version = "9.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9ae10193d25051e74945f1ea2d0b42e03cc3b890f7e4cc5faa44997d808193f" dependencies = [ "base64 0.21.7", "js-sys", "pem", "ring", "serde", "serde_json", "simple_asn1", ] [[package]] name = "kv-log-macro" version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0de8b303297635ad57c9f5059fd9cee7a47f8e8daa09df0fcd07dd39fb22977f" dependencies = [ "log", ] [[package]] name = "lalrpop" version = "0.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "55cb077ad656299f160924eb2912aa147d7339ea7d69e1b5517326fdcec3c1ca" dependencies = [ "ascii-canvas", "bit-set", "ena", "itertools 0.11.0", "lalrpop-util", "petgraph", "pico-args", "regex", "regex-syntax 0.8.5", "string_cache", "term", "tiny-keccak", "unicode-xid", "walkdir", ] [[package]] name = "lalrpop-util" version = "0.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "507460a910eb7b32ee961886ff48539633b788a36b65692b95f225b844c82553" dependencies = [ "regex-automata 0.4.9", ] [[package]] name = "lazy-regex" version = "3.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "60c7310b93682b36b98fa7ea4de998d3463ccbebd94d935d6b48ba5b6ffa7126" dependencies = [ "lazy-regex-proc_macros", "once_cell", "regex", ] [[package]] name = "lazy-regex-proc_macros" version = "3.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ba01db5ef81e17eb10a5e0f2109d1b3a3e29bac3070fdbd7d156bf7dbd206a1" dependencies = [ "proc-macro2", "quote", "regex", "syn 2.0.95", ] [[package]] name = "lazy_static" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" dependencies = [ "spin", ] [[package]] name = "lazycell" version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55" [[package]] name = "lexicmp" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7378d131ddf24063b32cbd7e91668d183140c4b3906270635a4d633d1068ea5d" dependencies = [ "any_ascii", ] [[package]] name = "libc" version = "0.2.169" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5aba8db14291edd000dfcc4d620c7ebfb122c613afb886ca8803fa4e128a20a" [[package]] name = "libloading" version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fc2f4eb4bc735547cfed7c0a4922cbd04a4655978c09b54f1f7b228750664c34" dependencies = [ "cfg-if", "windows-targets 0.52.6", ] [[package]] name = "libm" version = "0.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8355be11b20d696c8f18f6cc018c4e372165b1fa8126cef092399c9951984ffa" [[package]] name = "libredox" version = "0.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c0ff37bd590ca25063e35af745c343cb7a0271906fb7b37e4813e8f79f00268d" dependencies = [ "bitflags 2.6.0", "libc", ] [[package]] name = "librocksdb-sys" version = "0.11.0+8.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3386f101bcb4bd252d8e9d2fb41ec3b0862a15a62b478c355b2982efa469e3e" dependencies = [ "bindgen 0.65.1", "bzip2-sys", "cc", "glob", "libc", "libz-sys", ] [[package]] name = "libsqlite3-sys" version = "0.30.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2e99fb7a497b1e3339bc746195567ed8d3e24945ecd636e3619d20b9de9e9149" dependencies = [ "cc", "pkg-config", "vcpkg", ] [[package]] name = "libtest-mimic" version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5297962ef19edda4ce33aaa484386e0a5b3d7f2f4e037cbeee00503ef6b29d33" dependencies = [ "anstream", "anstyle", "clap", "escape8259", ] [[package]] name = "libz-sys" version = "1.1.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df9b68e50e6e0b26f672573834882eb57759f6db9b3be2ea3c35c91188bb4eaa" dependencies = [ "cc", "pkg-config", "vcpkg", ] [[package]] name = "linfa-linalg" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "56e7562b41c8876d3367897067013bb2884cc78e6893f092ecd26b305176ac82" dependencies = [ "ndarray", "num-traits", "rand 0.8.5", "thiserror 1.0.69", ] [[package]] name = "linked-hash-map" version = "0.5.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0717cef1bc8b636c6e1c1bbdefc09e6322da8a9321966e8928ef80d20f7f770f" [[package]] name = "linux-raw-sys" version = "0.4.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d26c52dbd32dccf2d10cac7725f8eae5296885fb5703b261f7d0a0739ec807ab" [[package]] name = "litemap" version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ee93343901ab17bd981295f2cf0026d4ad018c7c31ba84549a4ddbb47a45104" [[package]] name = "lock_api" version = "0.4.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "07af8b9cdd281b7915f413fa73f29ebd5d55d0d3f0155584dade1ff18cea1b17" dependencies = [ "autocfg", "scopeguard", ] [[package]] name = "log" version = "0.4.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a7a70ba024b9dc04c27ea2f0c0548feb474ec5c54bba33a7f72f873a39d07b24" dependencies = [ "value-bag", ] [[package]] name = "loom" version = "0.5.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff50ecb28bb86013e935fb6683ab1f6d3a20016f123c76fd4c27470076ac30f5" dependencies = [ "cfg-if", "generator 0.7.5", "scoped-tls", "tracing", "tracing-subscriber", ] [[package]] name = "loom" version = "0.7.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "419e0dc8046cb947daa77eb95ae174acfbddb7673b4151f56d1eed8e93fbfaca" dependencies = [ "cfg-if", "generator 0.8.4", "scoped-tls", "tracing", "tracing-subscriber", ] [[package]] name = "lru-cache" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "31e24f1ad8321ca0e8a1e0ac13f23cb668e6f5466c2c57319f6a5cf1cc8e3b1c" dependencies = [ "linked-hash-map", ] [[package]] name = "mac" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c41e0c4fef86961ac6d6f8a82609f55f31b05e4fce149ac5710e439df7619ba4" [[package]] name = "maplit" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3e2e65a1a2e43cfcb47a895c4c8b10d1f4a61097f9f254f183aee60cad9c651d" [[package]] name = "markup5ever" version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "16ce3abbeba692c8b8441d036ef91aea6df8da2c6b6e21c7e14d3c18e526be45" dependencies = [ "log", "phf", "phf_codegen", "string_cache", "string_cache_codegen", "tendril", ] [[package]] name = "match_cfg" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ffbee8634e0d45d258acb448e7eaab3fce7a0a467395d4d9f228e3c1f01fb2e4" [[package]] name = "matchers" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8263075bb86c5a1b1427b5ae862e8889656f126e9f77c484496e8b47cf5c5558" dependencies = [ "regex-automata 0.1.10", ] [[package]] name = "matchit" version = "0.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94" [[package]] name = "matrixmultiply" version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9380b911e3e96d10c1f415da0876389aaf1b56759054eeb0de7df940c456ba1a" dependencies = [ "autocfg", "rawpointer", ] [[package]] name = "md-5" version = "0.10.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" dependencies = [ "cfg-if", "digest", ] [[package]] name = "memchr" version = "2.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "78ca9ab1a0babb1e7d5695e3530886289c18cf2f87ec19a575a0abdce112e3a3" [[package]] name = "memmap2" version = "0.5.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "83faa42c0a078c393f6b29d5db232d8be22776a891f8f56e5284faee4a20b327" dependencies = [ "libc", ] [[package]] name = "memoffset" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5de893c32cde5f383baa4c04c5d6dbdd735cfd4a794b0debdb2bb1b421da5ff4" dependencies = [ "autocfg", ] [[package]] name = "metrics" version = "0.24.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7a7deb012b3b2767169ff203fadb4c6b0b82b947512e5eb9e0b78c2e186ad9e3" dependencies = [ "ahash 0.8.11", "portable-atomic", ] [[package]] name = "miette" version = "5.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "59bb584eaeeab6bd0226ccf3509a69d7936d148cf3d036ad350abe35e8c6856e" dependencies = [ "miette-derive", "once_cell", "thiserror 1.0.69", "unicode-width", ] [[package]] name = "miette-derive" version = "5.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "49e7bc1560b95a3c4a25d03de42fe76ca718ab92d1a22a55b9b4cf67b3ae635c" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "mime" version = "0.3.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" [[package]] name = "mime_guess" version = "2.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e" dependencies = [ "mime", "unicase", ] [[package]] name = "mini-moka" version = "0.10.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c325dfab65f261f386debee8b0969da215b3fa0037e74c8a1234db7ba986d803" dependencies = [ "crossbeam-channel", "crossbeam-utils", "dashmap 5.5.3", "skeptic", "smallvec", "tagptr", "triomphe", ] [[package]] name = "minimal-lexical" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" [[package]] name = "miniz_oxide" version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ffbe83022cedc1d264172192511ae958937694cd57ce297164951b8b3568394" dependencies = [ "adler2", ] [[package]] name = "minstant" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fb9b5c752f145ac5046bccc3c4f62892e3c950c1d1eab80c5949cd68a2078db" dependencies = [ "ctor", "web-time", ] [[package]] name = "mio" version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a4a650543ca06a924e8b371db273b2756685faae30f8487da1b56505a8f78b0c" dependencies = [ "libc", "log", "wasi 0.11.0+wasi-snapshot-preview1", "windows-sys 0.48.0", ] [[package]] name = "mio" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2886843bf800fba2e3377cff24abf6379b4c4d5c6681eaf9ea5b0d15090450bd" dependencies = [ "libc", "wasi 0.11.0+wasi-snapshot-preview1", "windows-sys 0.52.0", ] [[package]] name = "moka" version = "0.12.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a9321642ca94a4282428e6ea4af8cc2ca4eac48ac7a6a4ea8f33f76d0ce70926" dependencies = [ "async-lock", "crossbeam-channel", "crossbeam-epoch", "crossbeam-utils", "event-listener 5.4.0", "futures-util", "loom 0.7.2", "parking_lot 0.12.3", "portable-atomic", "rustc_version", "smallvec", "tagptr", "thiserror 1.0.69", "uuid", ] [[package]] name = "mongodb" version = "3.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff1f6edf7fe8828429647a2200f684681ca6d5a33b45edc3140c81390d852301" dependencies = [ "async-trait", "base64 0.13.1", "bitflags 1.3.2", "bson", "chrono", "derive-where", "derive_more", "futures-core", "futures-executor", "futures-io", "futures-util", "hex", "hickory-proto", "hickory-resolver", "hmac", "md-5", "mongodb-internal-macros", "once_cell", "pbkdf2 0.11.0", "percent-encoding", "rand 0.8.5", "rustc_version_runtime", "rustls 0.21.12", "rustls-pemfile 1.0.4", "serde", "serde_bytes", "serde_with", "sha-1", "sha2", "socket2", "stringprep", "strsim", "take_mut", "thiserror 1.0.69", "tokio", "tokio-rustls 0.24.1", "tokio-util", "typed-builder", "uuid", "webpki-roots 0.25.4", ] [[package]] name = "mongodb-internal-macros" version = "3.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b07bfd601af78e39384707a8e80041946c98260e3e0190e294ee7435823e6bf" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "monoio" version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3bd0f8bcde87b1949f95338b547543fcab187bc7e7a5024247e359a5e828ba6a" dependencies = [ "auto-const-array", "bytes", "flume", "fxhash", "io-uring 0.6.4", "libc", "memchr", "mio 0.8.11", "monoio-macros", "nix", "once_cell", "pin-project-lite", "socket2", "threadpool", "windows-sys 0.48.0", ] [[package]] name = "monoio-macros" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "176a5f5e69613d9e88337cf2a65e11135332b4efbcc628404a7c555e4452084c" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "multer" version = "3.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "83e87776546dc87511aa5ee218730c92b666d7264ab6ed41f9d215af9cd5224b" dependencies = [ "bytes", "encoding_rs", "futures-util", "http 1.2.0", "httparse", "memchr", "mime", "spin", "version_check", ] [[package]] name = "multimap" version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "defc4c55412d89136f966bbb339008b474350e5e6e78d2714439c386b3137a03" [[package]] name = "nanoid" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3ffa00dec017b5b1a8b7cf5e2c008bfda1aa7e0697ac1508b491fdf2622fb4d8" dependencies = [ "rand 0.8.5", ] [[package]] name = "nanorand" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a51313c5820b0b02bd422f4b44776fbf47961755c74ce64afc73bfad10226c3" dependencies = [ "getrandom 0.2.15", ] [[package]] name = "native-tls" version = "0.2.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a8614eb2c83d59d1c8cc974dd3f920198647674a0a035e1af1fa58707e317466" dependencies = [ "libc", "log", "openssl", "openssl-probe", "openssl-sys", "schannel", "security-framework", "security-framework-sys", "tempfile", ] [[package]] name = "ndarray" version = "0.15.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "adb12d4e967ec485a5f71c6311fe28158e9d6f4bc4a447b474184d0f91a8fa32" dependencies = [ "approx 0.4.0", "matrixmultiply", "num-complex", "num-integer", "num-traits", "rawpointer", ] [[package]] name = "ndarray-stats" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af5a8477ac96877b5bd1fd67e0c28736c12943aba24eda92b127e036b0c8f400" dependencies = [ "indexmap 1.9.3", "itertools 0.10.5", "ndarray", "noisy_float", "num-integer", "num-traits", "rand 0.8.5", ] [[package]] name = "new_debug_unreachable" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "650eef8c711430f1a879fdd01d4745a7deea475becfb90269c06775983bbf086" [[package]] name = "nibble_vec" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77a5d83df9f36fe23f0c3648c6bbb8b0298bb5f1939c8f2704431371f4b84d43" dependencies = [ "smallvec", ] [[package]] name = "nix" version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "598beaf3cc6fdd9a5dfb1630c2800c7acd31df7aaf0f565796fba2b53ca1af1b" dependencies = [ "bitflags 1.3.2", "cfg-if", "libc", "memoffset", "pin-utils", ] [[package]] name = "no-std-compat" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b93853da6d84c2e3c7d730d6473e8817692dd89be387eb01b94d7f108ecb5b8c" [[package]] name = "noisy_float" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "978fe6e6ebc0bf53de533cd456ca2d9de13de13856eda1518a285d7705a213af" dependencies = [ "num-traits", ] [[package]] name = "nom" version = "7.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" dependencies = [ "memchr", "minimal-lexical", ] [[package]] name = "nonzero_ext" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "38bf9645c8b145698bb0b18a4637dcacbc421ea49bef2317e4fd8065a387cf21" [[package]] name = "nu-ansi-term" version = "0.46.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77a8165726e8236064dbb45459242600304b42a5ea24ee2948e18e023bf7ba84" dependencies = [ "overload", "winapi", ] [[package]] name = "num-bigint" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" dependencies = [ "num-integer", "num-traits", ] [[package]] name = "num-bigint-dig" version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc84195820f291c7697304f3cbdadd1cb7199c0efc917ff5eafd71225c136151" dependencies = [ "byteorder", "lazy_static", "libm", "num-integer", "num-iter", "num-traits", "rand 0.8.5", "smallvec", "zeroize", ] [[package]] name = "num-complex" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" dependencies = [ "num-traits", ] [[package]] name = "num-conv" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9" [[package]] name = "num-derive" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "876a53fff98e03a936a674b29568b0e605f06b29372c2489ff4de23f1949743d" dependencies = [ "proc-macro2", "quote", "syn 1.0.109", ] [[package]] name = "num-derive" version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed3955f1a9c7c0c15e092f9c887db08b1fc683305fdf6eb6684f22555355e202" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "num-integer" version = "0.1.46" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" dependencies = [ "num-traits", ] [[package]] name = "num-iter" version = "0.1.45" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1429034a0490724d0075ebb2bc9e875d6503c3cf69e235a8941aa757d83ef5bf" dependencies = [ "autocfg", "num-integer", "num-traits", ] [[package]] name = "num-traits" version = "0.2.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" dependencies = [ "autocfg", "libm", ] [[package]] name = "num_cpus" version = "1.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4161fcb6d602d4d2081af7c3a45852d875a03dd337a6bfdd6e06407b61342a43" dependencies = [ "hermit-abi 0.3.9", "libc", ] [[package]] name = "object" version = "0.36.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "62948e14d923ea95ea2c7c86c71013138b66525b86bdc08d2dcc262bdb497b87" dependencies = [ "memchr", ] [[package]] name = "object_store" version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6da452820c715ce78221e8202ccc599b4a52f3e1eb3eedb487b680c81a8e3f3" dependencies = [ "async-trait", "bytes", "chrono", "futures", "humantime", "itertools 0.13.0", "parking_lot 0.12.3", "percent-encoding", "snafu", "tokio", "tracing", "url", "walkdir", ] [[package]] name = "once_cell" version = "1.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1261fe7e33c73b354eab43b1273a57c8f967d0391e80353e51f764ac02cf6775" [[package]] name = "oorandom" version = "11.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b410bbe7e14ab526a0e86877eb47c6996a2bd7746f027ba551028c925390e4e9" [[package]] name = "opendal" version = "0.52.0" dependencies = [ "anyhow", "async-backtrace", "async-tls", "async-trait", "atomic_lib", "await-tree", "backon", "base64 0.22.1", "bb8", "bytes", "cacache", "chrono", "compio", "crc32c", "criterion", "dashmap 6.1.0", "dotenvy", "etcd-client", "fastrace", "fastrace-jaeger", "flume", "foundationdb", "futures", "getrandom 0.2.15", "ghac", "governor", "hdfs-native", "hdrs", "hmac", "http 1.2.0", "libtest-mimic", "log", "md-5", "metrics", "mime_guess", "mini-moka", "moka", "mongodb", "mongodb-internal-macros", "monoio", "once_cell", "openssh", "openssh-sftp-client", "opentelemetry", "opentelemetry-otlp", "opentelemetry_sdk", "ouroboros", "percent-encoding", "persy", "pretty_assertions", "probe", "prometheus", "prometheus-client", "prost 0.13.4", "quick-xml 0.36.2", "rand 0.8.5", "redb", "redis", "reqsign", "reqwest", "rocksdb", "rust-nebula", "serde", "serde_json", "sha1", "sha2", "size", "sled", "snowflaked", "sqlx", "suppaftp", "surrealdb", "tikv-client", "tokio", "tracing", "tracing-opentelemetry", "tracing-subscriber", "uuid", ] [[package]] name = "openssh" version = "0.11.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fabb3ca0c2ac5024fe5658a732b4dc36d1d9c49409f7d1774c9df2764143499f" dependencies = [ "libc", "once_cell", "shell-escape", "tempfile", "thiserror 2.0.9", "tokio", ] [[package]] name = "openssh-sftp-client" version = "0.15.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9efd4eeca3f1e754e93b4c23157395f70be0595131335b7311bc7b3ee1e12d3f" dependencies = [ "bytes", "derive_destructure2", "futures-core", "once_cell", "openssh", "openssh-sftp-client-lowlevel", "openssh-sftp-error", "pin-project", "scopeguard", "tokio", "tokio-io-utility", "tokio-util", "tracing", ] [[package]] name = "openssh-sftp-client-lowlevel" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0e6971c272aeccb246f25279a0bb06cfd642184c93f9d614f64987c3b12924da" dependencies = [ "awaitable", "bytes", "concurrent_arena", "derive_destructure2", "openssh-sftp-error", "openssh-sftp-protocol", "pin-project", "tokio", "tokio-io-utility", ] [[package]] name = "openssh-sftp-error" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "12a702f18f0595b4578b21fd120ae7aa45f4298a8b28ddcb2397ace6f5a8251a" dependencies = [ "awaitable-error", "openssh", "openssh-sftp-protocol-error", "ssh_format_error", "thiserror 2.0.9", "tokio", ] [[package]] name = "openssh-sftp-protocol" version = "0.24.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a9c862e0c56553146306507f55958c11ff554e02c46de287e6976e50d815b350" dependencies = [ "bitflags 2.6.0", "num-derive 0.4.2", "num-traits", "openssh-sftp-protocol-error", "serde", "ssh_format", "vec-strings", ] [[package]] name = "openssh-sftp-protocol-error" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42b54df62ccfd9a7708a83a9d60c46293837e478f9f4c0829360dcfa60ede8d2" dependencies = [ "serde", "thiserror 2.0.9", "vec-strings", ] [[package]] name = "openssl" version = "0.10.68" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6174bc48f102d208783c2c84bf931bb75927a617866870de8a4ea85597f871f5" dependencies = [ "bitflags 2.6.0", "cfg-if", "foreign-types", "libc", "once_cell", "openssl-macros", "openssl-sys", ] [[package]] name = "openssl-macros" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "openssl-probe" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff011a302c396a5197692431fc1948019154afc178baf7d8e37367442a4601cf" [[package]] name = "openssl-sys" version = "0.9.104" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "45abf306cbf99debc8195b66b7346498d7b10c210de50418b5ccd7ceba08c741" dependencies = [ "cc", "libc", "pkg-config", "vcpkg", ] [[package]] name = "opentelemetry" version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "236e667b670a5cdf90c258f5a55794ec5ac5027e960c224bff8367a59e1e6426" dependencies = [ "futures-core", "futures-sink", "js-sys", "pin-project-lite", "thiserror 2.0.9", "tracing", ] [[package]] name = "opentelemetry-http" version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a8863faf2910030d139fb48715ad5ff2f35029fc5f244f6d5f689ddcf4d26253" dependencies = [ "async-trait", "bytes", "http 1.2.0", "opentelemetry", "reqwest", "tracing", ] [[package]] name = "opentelemetry-otlp" version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5bef114c6d41bea83d6dc60eb41720eedd0261a67af57b66dd2b84ac46c01d91" dependencies = [ "async-trait", "futures-core", "http 1.2.0", "opentelemetry", "opentelemetry-http", "opentelemetry-proto", "opentelemetry_sdk", "prost 0.13.4", "reqwest", "thiserror 2.0.9", "tokio", "tonic 0.12.3", "tracing", ] [[package]] name = "opentelemetry-proto" version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "56f8870d3024727e99212eb3bb1762ec16e255e3e6f58eeb3dc8db1aa226746d" dependencies = [ "opentelemetry", "opentelemetry_sdk", "prost 0.13.4", "tonic 0.12.3", ] [[package]] name = "opentelemetry_sdk" version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "84dfad6042089c7fc1f6118b7040dc2eb4ab520abbf410b79dc481032af39570" dependencies = [ "async-trait", "futures-channel", "futures-executor", "futures-util", "glob", "opentelemetry", "percent-encoding", "rand 0.8.5", "serde_json", "thiserror 2.0.9", "tokio", "tokio-stream", "tracing", ] [[package]] name = "ordered-float" version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3305af35278dd29f46fcdd139e0b1fbfae2153f0e5928b39b035542dd31e37b7" dependencies = [ "num-traits", "serde", ] [[package]] name = "ordered-multimap" version = "0.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79" dependencies = [ "dlv-list", "hashbrown 0.14.5", ] [[package]] name = "os_pipe" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5ffd2b0a5634335b135d5728d84c5e0fd726954b87111f7506a61c502280d982" dependencies = [ "libc", "windows-sys 0.59.0", ] [[package]] name = "ouroboros" version = "0.18.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "944fa20996a25aded6b4795c6d63f10014a7a83f8be9828a11860b08c5fc4a67" dependencies = [ "aliasable", "ouroboros_macro", "static_assertions", ] [[package]] name = "ouroboros_macro" version = "0.18.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39b0deead1528fd0e5947a8546a9642a9777c25f6e1e26f34c97b204bbb465bd" dependencies = [ "heck 0.4.1", "itertools 0.12.1", "proc-macro2", "proc-macro2-diagnostics", "quote", "syn 2.0.95", ] [[package]] name = "overload" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b15813163c1d831bf4a13c3610c05c0d03b39feb07f7e09fa234dac9b15aaf39" [[package]] name = "panic-message" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "384e52fd8fbd4cbe3c317e8216260c21a0f9134de108cea8a4dd4e7e152c472d" [[package]] name = "parking" version = "2.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" [[package]] name = "parking_lot" version = "0.11.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d17b78036a60663b797adeaee46f5c9dfebb86948d1255007a1d6be0271ff99" dependencies = [ "instant", "lock_api", "parking_lot_core 0.8.6", ] [[package]] name = "parking_lot" version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1bf18183cf54e8d6059647fc3063646a1801cf30896933ec2311622cc4b9a27" dependencies = [ "lock_api", "parking_lot_core 0.9.10", ] [[package]] name = "parking_lot_core" version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "60a2cfe6f0ad2bfc16aefa463b497d5c7a5ecd44a23efa72aa342d90177356dc" dependencies = [ "cfg-if", "instant", "libc", "redox_syscall 0.2.16", "smallvec", "winapi", ] [[package]] name = "parking_lot_core" version = "0.9.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e401f977ab385c9e4e3ab30627d6f26d00e2c73eef317493c4ec6d468726cf8" dependencies = [ "cfg-if", "libc", "redox_syscall 0.5.8", "smallvec", "windows-targets 0.52.6", ] [[package]] name = "password-hash" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "346f04948ba92c43e8469c1ee6736c7563d71012b17d40745260fe106aac2166" dependencies = [ "base64ct", "rand_core 0.6.4", "subtle", ] [[package]] name = "paste" version = "1.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" [[package]] name = "path-clean" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "17359afc20d7ab31fdb42bb844c8b3bb1dabd7dcf7e68428492da7f16966fcef" [[package]] name = "pbkdf2" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "83a0692ec44e4cf1ef28ca317f14f8f07da2d95ec3fa01f86e4467b725e60917" dependencies = [ "digest", ] [[package]] name = "pbkdf2" version = "0.12.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" dependencies = [ "digest", "hmac", "password-hash", "sha2", ] [[package]] name = "peeking_take_while" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19b17cddbe7ec3f8bc800887bab5e717348c95ea2ca0b1bf0837fb964dc67099" [[package]] name = "pem" version = "3.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e459365e590736a54c3fa561947c84837534b8e9af6fc5bf781307e82658fae" dependencies = [ "base64 0.22.1", "serde", ] [[package]] name = "pem-rfc7468" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" dependencies = [ "base64ct", ] [[package]] name = "percent-encoding" version = "2.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3148f5046208a5d56bcfc03053e3ca6334e51da8dfb19b6cdc8b306fae3283e" [[package]] name = "persy" version = "1.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "58070b019476f812f780e0503b63207c256603f7e8703fdbe15bea7c692bc5be" dependencies = [ "crc", "data-encoding", "fs2", "linked-hash-map", "rand 0.8.5", "thiserror 1.0.69", "unsigned-varint", "zigzag", ] [[package]] name = "pest" version = "2.7.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b7cafe60d6cf8e62e1b9b2ea516a089c008945bb5a275416789e7db0bc199dc" dependencies = [ "memchr", "thiserror 2.0.9", "ucd-trie", ] [[package]] name = "petgraph" version = "0.6.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4c5cc86750666a3ed20bdaf5ca2a0344f9c67674cae0515bec2da16fbaa47db" dependencies = [ "fixedbitset", "indexmap 2.7.0", ] [[package]] name = "pharos" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e9567389417feee6ce15dd6527a8a1ecac205ef62c2932bcf3d9f6fc5b78b414" dependencies = [ "futures", "rustc_version", ] [[package]] name = "phf" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fd6780a80ae0c52cc120a26a1a42c1ae51b247a253e4e06113d23d2c2edd078" dependencies = [ "phf_macros", "phf_shared 0.11.3", ] [[package]] name = "phf_codegen" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "aef8048c789fa5e851558d709946d6d79a8ff88c0440c587967f8e94bfb1216a" dependencies = [ "phf_generator 0.11.3", "phf_shared 0.11.3", ] [[package]] name = "phf_generator" version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5d5285893bb5eb82e6aaf5d59ee909a06a16737a8970984dd7746ba9283498d6" dependencies = [ "phf_shared 0.10.0", "rand 0.8.5", ] [[package]] name = "phf_generator" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3c80231409c20246a13fddb31776fb942c38553c51e871f8cbd687a4cfb5843d" dependencies = [ "phf_shared 0.11.3", "rand 0.8.5", ] [[package]] name = "phf_macros" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f84ac04429c13a7ff43785d75ad27569f2951ce0ffd30a3321230db2fc727216" dependencies = [ "phf_generator 0.11.3", "phf_shared 0.11.3", "proc-macro2", "quote", "syn 2.0.95", "unicase", ] [[package]] name = "phf_shared" version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6796ad771acdc0123d2a88dc428b5e38ef24456743ddb1744ed628f9815c096" dependencies = [ "siphasher 0.3.11", ] [[package]] name = "phf_shared" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "67eabc2ef2a60eb7faa00097bd1ffdb5bd28e62bf39990626a582201b7a754e5" dependencies = [ "siphasher 1.0.1", "unicase", ] [[package]] name = "pico-args" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5be167a7af36ee22fe3115051bc51f6e6c7054c9348e28deb4f49bd6f705a315" [[package]] name = "pin-project" version = "1.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e2ec53ad785f4d35dac0adea7f7dc6f1bb277ad84a680c7afefeae05d1f5916" dependencies = [ "pin-project-internal", ] [[package]] name = "pin-project-internal" version = "1.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d56a66c0c55993aa927429d0f8a0abfd74f084e4d9c192cffed01e418d83eefb" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "pin-project-lite" version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b3cff922bd51709b605d9ead9aa71031d81447142d828eb4a6eba76fe619f9b" [[package]] name = "pin-utils" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" [[package]] name = "piper" version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96c8c490f422ef9a4efd2cb5b42b76c8613d7e7dfc1caf667b8a3350a5acc066" dependencies = [ "atomic-waker", "fastrand", "futures-io", ] [[package]] name = "pkcs1" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8ffb9f10fa047879315e6625af03c164b16962a5368d724ed16323b68ace47f" dependencies = [ "der", "pkcs8", "spki", ] [[package]] name = "pkcs5" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e847e2c91a18bfa887dd028ec33f2fe6f25db77db3619024764914affe8b69a6" dependencies = [ "aes", "cbc", "der", "pbkdf2 0.12.2", "scrypt", "sha2", "spki", ] [[package]] name = "pkcs8" version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" dependencies = [ "der", "pkcs5", "rand_core 0.6.4", "spki", ] [[package]] name = "pkg-config" version = "0.3.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "953ec861398dccce10c670dfeaf3ec4911ca479e9c02154b3a215178c5f566f2" [[package]] name = "plotters" version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5aeb6f403d7a4911efb1e33402027fc44f29b5bf6def3effcc22d7bb75f2b747" dependencies = [ "num-traits", "plotters-backend", "plotters-svg", "wasm-bindgen", "web-sys", ] [[package]] name = "plotters-backend" version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df42e13c12958a16b3f7f4386b9ab1f3e7933914ecea48da7139435263a4172a" [[package]] name = "plotters-svg" version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "51bae2ac328883f7acdfea3d66a7c35751187f870bc81f94563733a154d7a670" dependencies = [ "plotters-backend", ] [[package]] name = "polling" version = "3.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a604568c3202727d1507653cb121dbd627a58684eb09a820fd746bee38b4442f" dependencies = [ "cfg-if", "concurrent-queue", "hermit-abi 0.4.0", "pin-project-lite", "rustix", "tracing", "windows-sys 0.59.0", ] [[package]] name = "portable-atomic" version = "1.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "280dc24453071f1b63954171985a0b0d30058d287960968b9b2aca264c8d4ee6" [[package]] name = "powerfmt" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" [[package]] name = "ppv-lite86" version = "0.2.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77957b295656769bb8ad2b6a6b09d897d94f05c41b069aede1fcdaa675eaea04" dependencies = [ "zerocopy", ] [[package]] name = "precomputed-hash" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "925383efa346730478fb4838dbe9137d2a47675ad789c546d150a6e1dd4ab31c" [[package]] name = "pretty_assertions" version = "1.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3ae130e2f271fbc2ac3a40fb1d07180839cdbbe443c7a27e1e3c13c5cac0116d" dependencies = [ "diff", "yansi", ] [[package]] name = "prettyplease" version = "0.2.27" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "483f8c21f64f3ea09fe0f30f5d48c3e8eefe5dac9129f0075f76593b4c1da705" dependencies = [ "proc-macro2", "syn 2.0.95", ] [[package]] name = "probe" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d8e2d2444b730c8f027344c60f9e1f1554d7a3342df9bdd425142ed119a6e5a3" [[package]] name = "proc-macro-crate" version = "3.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ecf48c7ca261d60b74ab1a7b20da18bede46776b2e55535cb958eb595c5fa7b" dependencies = [ "toml_edit", ] [[package]] name = "proc-macro-error-attr2" version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96de42df36bb9bba5542fe9f1a054b8cc87e172759a1868aa05c1f3acc89dfc5" dependencies = [ "proc-macro2", "quote", ] [[package]] name = "proc-macro-error2" version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "11ec05c52be0a07b08061f7dd003e7d7092e0472bc731b4af7bb1ef876109802" dependencies = [ "proc-macro-error-attr2", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "proc-macro2" version = "1.0.92" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37d3544b3f2748c54e147655edb5025752e2303145b5aefb3c3ea2c78b973bb0" dependencies = [ "unicode-ident", ] [[package]] name = "proc-macro2-diagnostics" version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af066a9c399a26e020ada66a034357a868728e72cd426f3adcd35f80d88d88c8" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", "version_check", "yansi", ] [[package]] name = "procfs" version = "0.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "731e0d9356b0c25f16f33b5be79b1c57b562f141ebfcdb0ad8ac2c13a24293b4" dependencies = [ "bitflags 2.6.0", "hex", "lazy_static", "procfs-core", "rustix", ] [[package]] name = "procfs-core" version = "0.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2d3554923a69f4ce04c4a754260c338f505ce22642d3830e049a399fc2059a29" dependencies = [ "bitflags 2.6.0", "hex", ] [[package]] name = "prometheus" version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d33c28a30771f7f96db69893f78b857f7450d7e0237e9c8fc6427a81bae7ed1" dependencies = [ "cfg-if", "fnv", "lazy_static", "libc", "memchr", "parking_lot 0.12.3", "procfs", "protobuf", "thiserror 1.0.69", ] [[package]] name = "prometheus-client" version = "0.23.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf41c1a7c32ed72abe5082fb19505b969095c12da9f5732a4bc9878757fd087c" dependencies = [ "dtoa", "itoa", "parking_lot 0.12.3", "prometheus-client-derive-encode", ] [[package]] name = "prometheus-client-derive-encode" version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "440f724eba9f6996b75d63681b0a92b06947f1457076d503a4d2e2c8f56442b8" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "prost" version = "0.12.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "deb1435c188b76130da55f17a466d252ff7b1418b2ad3e037d127b94e3411f29" dependencies = [ "bytes", "prost-derive 0.12.6", ] [[package]] name = "prost" version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2c0fef6c4230e4ccf618a35c59d7ede15dea37de8427500f50aff708806e42ec" dependencies = [ "bytes", "prost-derive 0.13.4", ] [[package]] name = "prost-build" version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0f3e5beed80eb580c68e2c600937ac2c4eedabdfd5ef1e5b7ea4f3fba84497b" dependencies = [ "heck 0.5.0", "itertools 0.13.0", "log", "multimap", "once_cell", "petgraph", "prettyplease", "prost 0.13.4", "prost-types 0.13.4", "regex", "syn 2.0.95", "tempfile", ] [[package]] name = "prost-derive" version = "0.12.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "81bddcdb20abf9501610992b6759a4c888aef7d1a7247ef75e2404275ac24af1" dependencies = [ "anyhow", "itertools 0.12.1", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "prost-derive" version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "157c5a9d7ea5c2ed2d9fb8f495b64759f7816c7eaea54ba3978f0d63000162e3" dependencies = [ "anyhow", "itertools 0.13.0", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "prost-types" version = "0.12.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9091c90b0a32608e984ff2fa4091273cbdd755d54935c51d520887f4a1dbd5b0" dependencies = [ "prost 0.12.6", ] [[package]] name = "prost-types" version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cc2f1e56baa61e93533aebc21af4d2134b70f66275e0fcdf3cbe43d77ff7e8fc" dependencies = [ "prost 0.13.4", ] [[package]] name = "protobuf" version = "2.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "106dd99e98437432fed6519dedecfade6a06a73bb7b2a1e019fdd2bee5778d94" [[package]] name = "psl-types" version = "2.0.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "33cb294fe86a74cbcf50d4445b37da762029549ebeea341421c7c70370f86cac" [[package]] name = "psm" version = "0.1.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "200b9ff220857e53e184257720a14553b2f4aa02577d2ed9842d45d4b9654810" dependencies = [ "cc", ] [[package]] name = "ptr_meta" version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0738ccf7ea06b608c10564b31debd4f5bc5e197fc8bfe088f68ae5ce81e7a4f1" dependencies = [ "ptr_meta_derive", ] [[package]] name = "ptr_meta_derive" version = "0.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "16b845dbfca988fa33db069c0e230574d15a3088f147a87b64c7589eb662c9ac" dependencies = [ "proc-macro2", "quote", "syn 1.0.109", ] [[package]] name = "pulldown-cmark" version = "0.9.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "57206b407293d2bcd3af849ce869d52068623f19e1b5ff8e8778e3309439682b" dependencies = [ "bitflags 2.6.0", "memchr", "unicase", ] [[package]] name = "quanta" version = "0.12.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3bd1fe6824cea6538803de3ff1bc0cf3949024db3d43c9643024bfb33a807c0e" dependencies = [ "crossbeam-utils", "libc", "once_cell", "raw-cpuid", "wasi 0.11.0+wasi-snapshot-preview1", "web-sys", "winapi", ] [[package]] name = "quick-error" version = "1.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0" [[package]] name = "quick-xml" version = "0.35.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "86e446ed58cef1bbfe847bc2fda0e2e4ea9f0e57b90c507d4781292590d72a4e" dependencies = [ "memchr", "serde", ] [[package]] name = "quick-xml" version = "0.36.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f7649a7b4df05aed9ea7ec6f628c67c9953a43869b8bc50929569b2999d443fe" dependencies = [ "memchr", "serde", ] [[package]] name = "quick_cache" version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eb55a1aa7668676bb93926cd4e9cdfe60f03bb866553bcca9112554911b6d3dc" dependencies = [ "ahash 0.8.11", "equivalent", "hashbrown 0.14.5", "parking_lot 0.12.3", ] [[package]] name = "quinn" version = "0.11.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "62e96808277ec6f97351a2380e6c25114bc9e67037775464979f3037c92d05ef" dependencies = [ "bytes", "pin-project-lite", "quinn-proto", "quinn-udp", "rustc-hash 2.1.0", "rustls 0.23.20", "socket2", "thiserror 2.0.9", "tokio", "tracing", ] [[package]] name = "quinn-proto" version = "0.11.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a2fe5ef3495d7d2e377ff17b1a8ce2ee2ec2a18cde8b6ad6619d65d0701c135d" dependencies = [ "bytes", "getrandom 0.2.15", "rand 0.8.5", "ring", "rustc-hash 2.1.0", "rustls 0.23.20", "rustls-pki-types", "slab", "thiserror 2.0.9", "tinyvec", "tracing", "web-time", ] [[package]] name = "quinn-udp" version = "0.5.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c40286217b4ba3a71d644d752e6a0b71f13f1b6a2c5311acfcbe0c2418ed904" dependencies = [ "cfg_aliases", "libc", "once_cell", "socket2", "tracing", "windows-sys 0.59.0", ] [[package]] name = "quote" version = "1.0.38" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0e4dccaaaf89514f546c693ddc140f729f958c247918a13380cccc6078391acc" dependencies = [ "proc-macro2", ] [[package]] name = "radium" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" [[package]] name = "radix_trie" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c069c179fcdc6a2fe24d8d18305cf085fdbd4f922c041943e203685d6a1c58fd" dependencies = [ "endian-type", "nibble_vec", "serde", ] [[package]] name = "rand" version = "0.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a6b1679d49b24bbfe0c803429aa1874472f50d9b363131f0e89fc356b544d03" dependencies = [ "getrandom 0.1.16", "libc", "rand_chacha 0.2.2", "rand_core 0.5.1", "rand_hc", ] [[package]] name = "rand" version = "0.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404" dependencies = [ "libc", "rand_chacha 0.3.1", "rand_core 0.6.4", ] [[package]] name = "rand_chacha" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f4c8ed856279c9737206bf725bf36935d8666ead7aa69b52be55af369d193402" dependencies = [ "ppv-lite86", "rand_core 0.5.1", ] [[package]] name = "rand_chacha" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" dependencies = [ "ppv-lite86", "rand_core 0.6.4", ] [[package]] name = "rand_core" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "90bde5296fc891b0cef12a6d03ddccc162ce7b2aff54160af9338f8d40df6d19" dependencies = [ "getrandom 0.1.16", ] [[package]] name = "rand_core" version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" dependencies = [ "getrandom 0.2.15", ] [[package]] name = "rand_hc" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ca3129af7b92a17112d59ad498c6f81eaf463253766b90396d39ea7a39d6613c" dependencies = [ "rand_core 0.5.1", ] [[package]] name = "raw-cpuid" version = "11.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ab240315c661615f2ee9f0f2cd32d5a7343a84d5ebcccb99d46e6637565e7b0" dependencies = [ "bitflags 2.6.0", ] [[package]] name = "rawpointer" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" [[package]] name = "rayon" version = "1.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b418a60154510ca1a002a752ca9714984e21e4241e804d32555251faf8b78ffa" dependencies = [ "either", "rayon-core", ] [[package]] name = "rayon-core" version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1465873a3dfdaa8ae7cb14b4383657caab0b3e8a0aa9ae8e04b044854c8dfce2" dependencies = [ "crossbeam-deque", "crossbeam-utils", ] [[package]] name = "reblessive" version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ffead9d0a0b45f3e0bc063a244b1779fd53a09d2c2f7282c186a016b1f10a778" [[package]] name = "redb" version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ea0a72cd7140de9fc3e318823b883abf819c20d478ec89ce880466dc2ef263c6" dependencies = [ "libc", ] [[package]] name = "redis" version = "0.27.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09d8f99a4090c89cc489a94833c901ead69bfbf3877b4867d5482e321ee875bc" dependencies = [ "arc-swap", "async-trait", "backon", "bytes", "combine", "crc16", "futures", "futures-util", "itertools 0.13.0", "itoa", "log", "native-tls", "num-bigint", "percent-encoding", "pin-project-lite", "rand 0.8.5", "rustls 0.23.20", "rustls-native-certs", "rustls-pemfile 2.2.0", "rustls-pki-types", "ryu", "sha1_smol", "socket2", "tokio", "tokio-native-tls", "tokio-rustls 0.26.1", "tokio-util", "url", ] [[package]] name = "redox_syscall" version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a" dependencies = [ "bitflags 1.3.2", ] [[package]] name = "redox_syscall" version = "0.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "03a862b389f93e68874fbf580b9de08dd02facb9a788ebadaf4a3fd33cf58834" dependencies = [ "bitflags 2.6.0", ] [[package]] name = "redox_users" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba009ff324d1fc1b900bd1fdb31564febe58a8ccc8a6fdbb93b543d33b13ca43" dependencies = [ "getrandom 0.2.15", "libredox", "thiserror 1.0.69", ] [[package]] name = "ref-cast" version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccf0a6f84d5f1d581da8b41b47ec8600871962f2a528115b542b362d4b744931" dependencies = [ "ref-cast-impl", ] [[package]] name = "ref-cast-impl" version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bcc303e793d3734489387d205e9b186fac9c6cfacedd98cbb2e8a5943595f3e6" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "reflink-copy" version = "0.1.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "17400ed684c3a0615932f00c271ae3eea13e47056a1455821995122348ab6438" dependencies = [ "cfg-if", "rustix", "windows 0.58.0", ] [[package]] name = "regex" version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b544ef1b4eac5dc2db33ea63606ae9ffcfac26c1416a2806ae0bf5f56b201191" dependencies = [ "aho-corasick", "memchr", "regex-automata 0.4.9", "regex-syntax 0.8.5", ] [[package]] name = "regex-automata" version = "0.1.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6c230d73fb8d8c1b9c0b3135c5142a8acee3a0558fb8db5cf1cb65f8d7862132" dependencies = [ "regex-syntax 0.6.29", ] [[package]] name = "regex-automata" version = "0.4.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "809e8dc61f6de73b46c85f4c96486310fe304c434cfa43669d7b40f711150908" dependencies = [ "aho-corasick", "memchr", "regex-syntax 0.8.5", ] [[package]] name = "regex-syntax" version = "0.6.29" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f162c6dd7b008981e4d40210aca20b4bd0f9b60ca9271061b07f78537722f2e1" [[package]] name = "regex-syntax" version = "0.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c" [[package]] name = "rend" version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "71fe3824f5629716b1589be05dacd749f6aa084c87e00e016714a8cdfccc997c" dependencies = [ "bytecheck", ] [[package]] name = "reqsign" version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eb0075a66c8bfbf4cc8b70dca166e722e1f55a3ea9250ecbb85f4d92a5f64149" dependencies = [ "anyhow", "async-trait", "base64 0.22.1", "chrono", "form_urlencoded", "getrandom 0.2.15", "hex", "hmac", "home", "http 1.2.0", "jsonwebtoken", "log", "once_cell", "percent-encoding", "quick-xml 0.35.0", "rand 0.8.5", "reqwest", "rsa", "rust-ini", "serde", "serde_json", "sha1", "sha2", ] [[package]] name = "reqwest" version = "0.12.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "43e734407157c3c2034e0258f5e4473ddb361b1e85f95a66690d67264d7cd1da" dependencies = [ "base64 0.22.1", "bytes", "futures-channel", "futures-core", "futures-util", "http 1.2.0", "http-body 1.0.1", "http-body-util", "hyper 1.5.2", "hyper-rustls", "hyper-util", "ipnet", "js-sys", "log", "mime", "mime_guess", "once_cell", "percent-encoding", "pin-project-lite", "quinn", "rustls 0.23.20", "rustls-pemfile 2.2.0", "rustls-pki-types", "serde", "serde_json", "serde_urlencoded", "sync_wrapper 1.0.2", "tokio", "tokio-rustls 0.26.1", "tokio-util", "tower 0.5.2", "tower-service", "url", "wasm-bindgen", "wasm-bindgen-futures", "wasm-streams", "web-sys", "webpki-roots 0.26.7", "windows-registry", ] [[package]] name = "resolv-conf" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52e44394d2086d010551b14b53b1f24e31647570cd1deb0379e2c21b329aba00" dependencies = [ "hostname", "quick-error", ] [[package]] name = "revision" version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "22f53179a035f881adad8c4d58a2c599c6b4a8325b989c68d178d7a34d1b1e4c" dependencies = [ "chrono", "geo", "regex", "revision-derive", "roaring", "rust_decimal", "uuid", ] [[package]] name = "revision-derive" version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5f0ec466e5d8dca9965eb6871879677bef5590cf7525ad96cae14376efb75073" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "ring" version = "0.17.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c17fa4cb658e3583423e915b9f3acc01cceaee1860e33d59ebae66adc3a2dc0d" dependencies = [ "cc", "cfg-if", "getrandom 0.2.15", "libc", "spin", "untrusted", "windows-sys 0.52.0", ] [[package]] name = "rkyv" version = "0.7.45" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9008cd6385b9e161d8229e1f6549dd23c3d022f132a2ea37ac3a10ac4935779b" dependencies = [ "bitvec", "bytecheck", "bytes", "hashbrown 0.12.3", "ptr_meta", "rend", "rkyv_derive", "seahash", "tinyvec", "uuid", ] [[package]] name = "rkyv_derive" version = "0.7.45" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "503d1d27590a2b0a3a4ca4c94755aa2875657196ecbf401a42eff41d7de532c0" dependencies = [ "proc-macro2", "quote", "syn 1.0.109", ] [[package]] name = "rmp" version = "0.8.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "228ed7c16fa39782c3b3468e974aec2795e9089153cd08ee2e9aefb3613334c4" dependencies = [ "byteorder", "num-traits", "paste", ] [[package]] name = "rmpv" version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "58450723cd9ee93273ce44a20b6ec4efe17f8ed2e3631474387bfdecf18bb2a9" dependencies = [ "num-traits", "rmp", ] [[package]] name = "roaring" version = "0.10.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "41589aba99537475bf697f2118357cad1c31590c5a1b9f6d9fc4ad6d07503661" dependencies = [ "bytemuck", "byteorder", "serde", ] [[package]] name = "robust" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cbf4a6aa5f6d6888f39e980649f3ad6b666acdce1d78e95b8a2cb076e687ae30" [[package]] name = "rocksdb" version = "0.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bb6f170a4041d50a0ce04b0d2e14916d6ca863ea2e422689a5b694395d299ffe" dependencies = [ "libc", "librocksdb-sys", ] [[package]] name = "roxmltree" version = "0.18.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "862340e351ce1b271a378ec53f304a5558f7db87f3769dc655a8f6ecbb68b302" dependencies = [ "xmlparser", ] [[package]] name = "rsa" version = "0.9.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47c75d7c5c6b673e58bf54d8544a9f432e3a925b0e80f7cd3602ab5c50c55519" dependencies = [ "const-oid", "digest", "num-bigint-dig", "num-integer", "num-traits", "pkcs1", "pkcs8", "rand_core 0.6.4", "sha2", "signature", "spki", "subtle", "zeroize", ] [[package]] name = "rstar" version = "0.12.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "421400d13ccfd26dfa5858199c30a5d76f9c54e0dba7575273025b43c5175dbb" dependencies = [ "heapless", "num-traits", "smallvec", ] [[package]] name = "rtrb" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ad8388ea1a9e0ea807e442e8263a699e7edcb320ecbcd21b4fa8ff859acce3ba" [[package]] name = "rust-ini" version = "0.21.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4e310ef0e1b6eeb79169a1171daf9abcb87a2e17c03bee2c4bb100b55c75409f" dependencies = [ "cfg-if", "ordered-multimap", "trim-in-place", ] [[package]] name = "rust-nebula" version = "0.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "11a94ea754ca8b05b71ae911b7035180861cebb607c711acd766d14c04f87ae9" dependencies = [ "anyhow", "async-compat", "async-sleep", "async-trait", "base64 0.11.0", "bb8", "bufsize", "bytes", "const-cstr", "futures", "futures-util", "ghost", "num-derive 0.3.3", "num-traits", "ordered-float", "panic-message", "serde", "serde_json", "thiserror 1.0.69", "tokio", "tracing", ] [[package]] name = "rust-stemmers" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e46a2036019fdb888131db7a4c847a1063a7493f971ed94ea82c67eada63ca54" dependencies = [ "serde", "serde_derive", ] [[package]] name = "rust_decimal" version = "1.36.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b082d80e3e3cc52b2ed634388d436fe1f4de6af5786cc2de9ba9737527bdf555" dependencies = [ "arrayvec", "borsh", "bytes", "num-traits", "rand 0.8.5", "rkyv", "serde", "serde_json", ] [[package]] name = "rustc-demangle" version = "0.1.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "719b953e2095829ee67db738b3bfa9fa368c94900df327b3f07fe6e794d2fe1f" [[package]] name = "rustc-hash" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" [[package]] name = "rustc-hash" version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c7fb8039b3032c191086b10f11f319a6e99e1e82889c5cc6046f515c9db1d497" [[package]] name = "rustc_lexer" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c86aae0c77166108c01305ee1a36a1e77289d7dc6ca0a3cd91ff4992de2d16a5" dependencies = [ "unicode-xid", ] [[package]] name = "rustc_version" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" dependencies = [ "semver", ] [[package]] name = "rustc_version_runtime" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2dd18cd2bae1820af0b6ad5e54f4a51d0f3fcc53b05f845675074efcc7af071d" dependencies = [ "rustc_version", "semver", ] [[package]] name = "rustix" version = "0.38.43" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a78891ee6bf2340288408954ac787aa063d8e8817e9f53abb37c695c6d834ef6" dependencies = [ "bitflags 2.6.0", "errno", "libc", "linux-raw-sys", "windows-sys 0.59.0", ] [[package]] name = "rustls" version = "0.21.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f56a14d1f48b391359b22f731fd4bd7e43c97f3c50eee276f3aa09c94784d3e" dependencies = [ "log", "ring", "rustls-webpki 0.101.7", "sct", ] [[package]] name = "rustls" version = "0.23.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5065c3f250cbd332cd894be57c40fa52387247659b14a2d6041d121547903b1b" dependencies = [ "log", "once_cell", "ring", "rustls-pki-types", "rustls-webpki 0.102.8", "subtle", "zeroize", ] [[package]] name = "rustls-native-certs" version = "0.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5bfb394eeed242e909609f56089eecfe5fda225042e8b171791b9c95f5931e5" dependencies = [ "openssl-probe", "rustls-pemfile 2.2.0", "rustls-pki-types", "schannel", "security-framework", ] [[package]] name = "rustls-pemfile" version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c74cae0a4cf6ccbbf5f359f08efdf8ee7e1dc532573bf0db71968cb56b1448c" dependencies = [ "base64 0.21.7", ] [[package]] name = "rustls-pemfile" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50" dependencies = [ "rustls-pki-types", ] [[package]] name = "rustls-pki-types" version = "1.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d2bf47e6ff922db3825eb750c4e2ff784c6ff8fb9e13046ef6a1d1c5401b0b37" dependencies = [ "web-time", ] [[package]] name = "rustls-webpki" version = "0.101.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b6275d1ee7a1cd780b64aca7726599a1dbc893b1e64144529e55c3c2f745765" dependencies = [ "ring", "untrusted", ] [[package]] name = "rustls-webpki" version = "0.102.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "64ca1bc8749bd4cf37b5ce386cc146580777b4e8572c7b97baf22c83f444bee9" dependencies = [ "ring", "rustls-pki-types", "untrusted", ] [[package]] name = "rustversion" version = "1.0.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f7c45b9784283f1b2e7fb61b42047c2fd678ef0960d4f6f1eba131594cc369d4" [[package]] name = "ryu" version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f3cb5ba0dc43242ce17de99c180e96db90b235b8a9fdc9543c96d2209116bd9f" [[package]] name = "ryu-js" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6518fc26bced4d53678a22d6e423e9d8716377def84545fe328236e3af070e7f" [[package]] name = "salsa20" version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97a22f5af31f73a954c10289c93e8a50cc23d971e80ee446f1f6f7137a088213" dependencies = [ "cipher", ] [[package]] name = "same-file" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" dependencies = [ "winapi-util", ] [[package]] name = "schannel" version = "0.1.27" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1f29ebaa345f945cec9fbbc532eb307f0fdad8161f281b6369539c8d84876b3d" dependencies = [ "windows-sys 0.59.0", ] [[package]] name = "scoped-tls" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e1cf6437eb19a8f4a6cc0f7dca544973b0b78843adbfeb3683d1a94a0024a294" [[package]] name = "scopeguard" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" [[package]] name = "scrypt" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0516a385866c09368f0b5bcd1caff3366aace790fcd46e2bb032697bb172fd1f" dependencies = [ "password-hash", "pbkdf2 0.12.2", "salsa20", "sha2", ] [[package]] name = "sct" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "da046153aa2352493d6cb7da4b6e5c0c057d8a1d0a9aa8560baffdd945acd414" dependencies = [ "ring", "untrusted", ] [[package]] name = "seahash" version = "4.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c107b6f4780854c8b126e228ea8869f4d7b71260f962fefb57b996b8959ba6b" [[package]] name = "security-framework" version = "2.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" dependencies = [ "bitflags 2.6.0", "core-foundation", "core-foundation-sys", "libc", "security-framework-sys", ] [[package]] name = "security-framework-sys" version = "2.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "49db231d56a190491cb4aeda9527f1ad45345af50b0851622a7adb8c03b01c32" dependencies = [ "core-foundation-sys", "libc", ] [[package]] name = "semver" version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3cb6eb87a131f756572d7fb904f6e7b68633f09cca868c5df1c4b8d1a694bbba" dependencies = [ "serde", ] [[package]] name = "send_wrapper" version = "0.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cd0b0ec5f1c1ca621c432a25813d8d60c88abe6d3e08a3eb9cf37d97a0fe3d73" [[package]] name = "serde" version = "1.0.217" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "02fc4265df13d6fa1d00ecff087228cc0a2b5f3c0e87e258d8b94a156e984c70" dependencies = [ "serde_derive", ] [[package]] name = "serde-content" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e255eaf9f3814135df4f959c9f404ebb2e67238bae0ed412da10518d0629e7c9" dependencies = [ "serde", ] [[package]] name = "serde_bytes" version = "0.11.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "387cc504cb06bb40a96c8e04e951fe01854cf6bc921053c954e4a606d9675c6a" dependencies = [ "serde", ] [[package]] name = "serde_derive" version = "1.0.217" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5a9bf7cf98d04a2b28aead066b7496853d4779c9cc183c440dbac457641e19a0" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "serde_jcs" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cacecf649bc1a7c5f0e299cc813977c6a78116abda2b93b1ee01735b71ead9a8" dependencies = [ "ryu-js", "serde", "serde_json", ] [[package]] name = "serde_json" version = "1.0.135" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2b0d7ba2887406110130a978386c4e1befb98c674b4fba677954e4db976630d9" dependencies = [ "indexmap 2.7.0", "itoa", "memchr", "ryu", "serde", ] [[package]] name = "serde_urlencoded" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" dependencies = [ "form_urlencoded", "itoa", "ryu", "serde", ] [[package]] name = "serde_with" version = "3.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6b6f7f2fcb69f747921f79f3926bd1e203fce4fef62c268dd3abfb6d86029aa" dependencies = [ "base64 0.22.1", "chrono", "hex", "indexmap 1.9.3", "indexmap 2.7.0", "serde", "serde_derive", "serde_json", "serde_with_macros", "time", ] [[package]] name = "serde_with_macros" version = "3.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8d00caa5193a3c8362ac2b73be6b9e768aa5a4b2f721d8f4b339600c3cb51f8e" dependencies = [ "darling", "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "sha-1" version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f5058ada175748e33390e40e872bd0fe59a19f265d0158daa551c5a88a76009c" dependencies = [ "cfg-if", "cpufeatures", "digest", ] [[package]] name = "sha1" version = "0.10.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" dependencies = [ "cfg-if", "cpufeatures", "digest", ] [[package]] name = "sha1_smol" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbfa15b3dddfee50a0fff136974b3e1bde555604ba463834a7eb7deb6417705d" [[package]] name = "sha2" version = "0.10.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "793db75ad2bcafc3ffa7c68b215fee268f537982cd901d132f89c6343f3a3dc8" dependencies = [ "cfg-if", "cpufeatures", "digest", ] [[package]] name = "sharded-slab" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" dependencies = [ "lazy_static", ] [[package]] name = "shell-escape" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "45bb67a18fa91266cc7807181f62f9178a6873bfad7dc788c42e6430db40184f" [[package]] name = "shlex" version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" [[package]] name = "signal-hook-registry" version = "1.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a9e9e0b4211b72e7b8b6e85c807d36c212bdb33ea8587f7569562a84df5465b1" dependencies = [ "libc", ] [[package]] name = "signature" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" dependencies = [ "digest", "rand_core 0.6.4", ] [[package]] name = "simdutf8" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" [[package]] name = "simple_asn1" version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "adc4e5204eb1910f40f9cfa375f6f05b68c3abac4b6fd879c8ff5e7ae8a0a085" dependencies = [ "num-bigint", "num-traits", "thiserror 1.0.69", "time", ] [[package]] name = "siphasher" version = "0.3.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "38b58827f4464d87d377d175e90bf58eb00fd8716ff0a62f80356b5e61555d0d" [[package]] name = "siphasher" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "56199f7ddabf13fe5074ce809e7d3f42b42ae711800501b5b16ea82ad029c39d" [[package]] name = "size" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9fed904c7fb2856d868b92464fc8fa597fce366edea1a9cbfaa8cb5fe080bd6d" [[package]] name = "skeptic" version = "0.13.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "16d23b015676c90a0f01c197bfdc786c20342c73a0afdda9025adb0bc42940a8" dependencies = [ "bytecount", "cargo_metadata", "error-chain", "glob", "pulldown-cmark", "tempfile", "walkdir", ] [[package]] name = "slab" version = "0.4.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f92a496fb766b417c996b9c5e57daf2f7ad3b0bebe1ccfca4856390e3d3bb67" dependencies = [ "autocfg", ] [[package]] name = "sled" version = "0.34.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f96b4737c2ce5987354855aed3797279def4ebf734436c6aa4552cf8e169935" dependencies = [ "crc32fast", "crossbeam-epoch", "crossbeam-utils", "fs2", "fxhash", "libc", "log", "parking_lot 0.11.2", ] [[package]] name = "smallvec" version = "1.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3c5e1a9a646d36c3599cd173a41282daf47c44583ad367b8e6837255952e5c67" dependencies = [ "serde", ] [[package]] name = "smol_str" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dd538fb6910ac1099850255cf94a94df6551fbdd602454387d0adb2d1ca6dead" dependencies = [ "serde", ] [[package]] name = "snafu" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e4de37ad025c587a29e8f3f5605c00f70b98715ef90b9061a815b9e59e9042d6" dependencies = [ "doc-comment", "snafu-derive", ] [[package]] name = "snafu-derive" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "990079665f075b699031e9c08fd3ab99be5029b96f3b78dc0709e8f77e4efebf" dependencies = [ "heck 0.4.1", "proc-macro2", "quote", "syn 1.0.109", ] [[package]] name = "snap" version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" [[package]] name = "snowflaked" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "398d462c4c454399be452039b24b0aa0ecb4c7a57f6ae615f5d25de2b032f850" dependencies = [ "loom 0.5.6", ] [[package]] name = "socket2" version = "0.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c970269d99b64e60ec3bd6ad27270092a5394c4e309314b18ae3fe575695fbe8" dependencies = [ "libc", "windows-sys 0.52.0", ] [[package]] name = "spade" version = "2.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "93f5ef1f863aca7d1d7dda7ccfc36a0a4279bd6d3c375176e5e0712e25cb4889" dependencies = [ "hashbrown 0.14.5", "num-traits", "robust", "smallvec", ] [[package]] name = "spin" version = "0.9.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67" dependencies = [ "lock_api", ] [[package]] name = "spinning_top" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d96d2d1d716fb500937168cc09353ffdc7a012be8475ac7308e1bdf0e3923300" dependencies = [ "lock_api", ] [[package]] name = "spki" version = "0.7.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" dependencies = [ "base64ct", "der", ] [[package]] name = "sqlx" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4410e73b3c0d8442c5f99b425d7a435b5ee0ae4167b3196771dd3f7a01be745f" dependencies = [ "sqlx-core", "sqlx-macros", "sqlx-mysql", "sqlx-postgres", "sqlx-sqlite", ] [[package]] name = "sqlx-core" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a007b6936676aa9ab40207cde35daab0a04b823be8ae004368c0793b96a61e0" dependencies = [ "bytes", "crc", "crossbeam-queue", "either", "event-listener 5.4.0", "futures-core", "futures-intrusive", "futures-io", "futures-util", "hashbrown 0.15.2", "hashlink", "indexmap 2.7.0", "log", "memchr", "once_cell", "percent-encoding", "rustls 0.23.20", "rustls-pemfile 2.2.0", "serde", "serde_json", "sha2", "smallvec", "thiserror 2.0.9", "tokio", "tokio-stream", "tracing", "url", "webpki-roots 0.26.7", ] [[package]] name = "sqlx-macros" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3112e2ad78643fef903618d78cf0aec1cb3134b019730edb039b69eaf531f310" dependencies = [ "proc-macro2", "quote", "sqlx-core", "sqlx-macros-core", "syn 2.0.95", ] [[package]] name = "sqlx-macros-core" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4e9f90acc5ab146a99bf5061a7eb4976b573f560bc898ef3bf8435448dd5e7ad" dependencies = [ "dotenvy", "either", "heck 0.5.0", "hex", "once_cell", "proc-macro2", "quote", "serde", "serde_json", "sha2", "sqlx-core", "sqlx-mysql", "sqlx-postgres", "sqlx-sqlite", "syn 2.0.95", "tempfile", "tokio", "url", ] [[package]] name = "sqlx-mysql" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4560278f0e00ce64938540546f59f590d60beee33fffbd3b9cd47851e5fff233" dependencies = [ "atoi", "base64 0.22.1", "bitflags 2.6.0", "byteorder", "bytes", "crc", "digest", "dotenvy", "either", "futures-channel", "futures-core", "futures-io", "futures-util", "generic-array", "hex", "hkdf", "hmac", "itoa", "log", "md-5", "memchr", "once_cell", "percent-encoding", "rand 0.8.5", "rsa", "serde", "sha1", "sha2", "smallvec", "sqlx-core", "stringprep", "thiserror 2.0.9", "tracing", "whoami", ] [[package]] name = "sqlx-postgres" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c5b98a57f363ed6764d5b3a12bfedf62f07aa16e1856a7ddc2a0bb190a959613" dependencies = [ "atoi", "base64 0.22.1", "bitflags 2.6.0", "byteorder", "crc", "dotenvy", "etcetera", "futures-channel", "futures-core", "futures-util", "hex", "hkdf", "hmac", "home", "itoa", "log", "md-5", "memchr", "once_cell", "rand 0.8.5", "serde", "serde_json", "sha2", "smallvec", "sqlx-core", "stringprep", "thiserror 2.0.9", "tracing", "whoami", ] [[package]] name = "sqlx-sqlite" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f85ca71d3a5b24e64e1d08dd8fe36c6c95c339a896cc33068148906784620540" dependencies = [ "atoi", "flume", "futures-channel", "futures-core", "futures-executor", "futures-intrusive", "futures-util", "libsqlite3-sys", "log", "percent-encoding", "serde", "serde_urlencoded", "sqlx-core", "tracing", "url", ] [[package]] name = "ssh_format" version = "0.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "24ab31081d1c9097c327ec23550858cb5ffb4af6b866c1ef4d728455f01f3304" dependencies = [ "bytes", "serde", "ssh_format_error", ] [[package]] name = "ssh_format_error" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be3c6519de7ca611f71ef7e8a56eb57aa1c818fecb5242d0a0f39c83776c210c" dependencies = [ "serde", ] [[package]] name = "ssri" version = "9.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "da7a2b3c2bc9693bcb40870c4e9b5bf0d79f9cb46273321bf855ec513e919082" dependencies = [ "base64 0.21.7", "digest", "hex", "miette", "serde", "sha-1", "sha2", "thiserror 1.0.69", "xxhash-rust", ] [[package]] name = "stable_deref_trait" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a8f112729512f8e442d81f95a8a7ddf2b7c6b8a1a6f509a95864142b30cab2d3" [[package]] name = "stacker" version = "0.1.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "799c883d55abdb5e98af1a7b3f23b9b6de8ecada0ecac058672d7635eb48ca7b" dependencies = [ "cc", "cfg-if", "libc", "psm", "windows-sys 0.59.0", ] [[package]] name = "static_assertions" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f" [[package]] name = "static_assertions_next" version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d7beae5182595e9a8b683fa98c4317f956c9a2dec3b9716990d20023cc60c766" [[package]] name = "storekey" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "43c42833834a5d23b344f71d87114e0cc9994766a5c42938f4b50e7b2aef85b2" dependencies = [ "byteorder", "memchr", "serde", "thiserror 1.0.69", ] [[package]] name = "string_cache" version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f91138e76242f575eb1d3b38b4f1362f10d3a43f47d182a5b359af488a02293b" dependencies = [ "new_debug_unreachable", "once_cell", "parking_lot 0.12.3", "phf_shared 0.10.0", "precomputed-hash", "serde", ] [[package]] name = "string_cache_codegen" version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6bb30289b722be4ff74a408c3cc27edeaad656e06cb1fe8fa9231fa59c728988" dependencies = [ "phf_generator 0.10.0", "phf_shared 0.10.0", "proc-macro2", "quote", ] [[package]] name = "stringprep" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b4df3d392d81bd458a8a621b8bffbd2302a12ffe288a9d931670948749463b1" dependencies = [ "unicode-bidi", "unicode-normalization", "unicode-properties", ] [[package]] name = "strsim" version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" [[package]] name = "strum" version = "0.26.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8fec0f0aef304996cf250b31b5a10dee7980c85da9d759361292b8bca5a18f06" dependencies = [ "strum_macros", ] [[package]] name = "strum_macros" version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4c6bee85a5a24955dc440386795aa378cd9cf82acd5f764469152d2270e581be" dependencies = [ "heck 0.5.0", "proc-macro2", "quote", "rustversion", "syn 2.0.95", ] [[package]] name = "subtle" version = "2.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" [[package]] name = "suppaftp" version = "6.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a593e8bdcd2aff8369ccddf7fd42ae7ccaa0f8b6b343c16ed763c99c20926a29" dependencies = [ "async-std", "async-tls", "async-trait", "chrono", "futures-lite", "lazy-regex", "log", "pin-project", "rustls 0.23.20", "thiserror 2.0.9", ] [[package]] name = "surrealdb" version = "2.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6c4b1b800a132d76a9899f813aa382c8866feb40a4637fdb14da5e5f3a9fbc7" dependencies = [ "arrayvec", "async-channel 2.3.1", "bincode", "chrono", "dmp", "futures", "geo", "indexmap 2.7.0", "path-clean", "pharos", "reblessive", "reqwest", "revision", "ring", "rust_decimal", "rustls 0.23.20", "rustls-pki-types", "semver", "serde", "serde-content", "serde_json", "surrealdb-core", "thiserror 1.0.69", "tokio", "tokio-tungstenite", "tokio-util", "tracing", "trice", "url", "uuid", "wasm-bindgen-futures", "wasmtimer", "ws_stream_wasm", ] [[package]] name = "surrealdb-core" version = "2.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "582a63df1d8c2c8fb90cf923c25350946604b0bea9f6a9815fe02d074b136dea" dependencies = [ "addr", "ahash 0.8.11", "ammonia", "any_ascii", "argon2", "async-channel 2.3.1", "async-executor", "async-graphql", "base64 0.21.7", "bcrypt", "bincode", "blake3", "bytes", "castaway", "cedar-policy", "chrono", "ciborium", "dashmap 5.5.3", "deunicode", "dmp", "fst", "futures", "fuzzy-matcher", "geo", "geo-types", "hex", "ipnet", "jsonwebtoken", "lexicmp", "linfa-linalg", "md-5", "nanoid", "ndarray", "ndarray-stats", "num-traits", "num_cpus", "object_store", "pbkdf2 0.12.2", "pharos", "phf", "pin-project-lite", "quick_cache", "radix_trie", "rand 0.8.5", "rayon", "reblessive", "regex", "revision", "ring", "rmpv", "roaring", "rust-stemmers", "rust_decimal", "scrypt", "semver", "serde", "serde-content", "serde_json", "sha1", "sha2", "snap", "storekey", "strsim", "subtle", "surrealdb-derive", "thiserror 1.0.69", "tokio", "tracing", "trice", "ulid", "unicase", "url", "uuid", "vart", "wasm-bindgen-futures", "wasmtimer", "ws_stream_wasm", ] [[package]] name = "surrealdb-derive" version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "aacdb4c58b9ebef0291310afcd63af0012d85610d361f3785952c61b6f1dddf4" dependencies = [ "quote", "syn 1.0.109", ] [[package]] name = "syn" version = "1.0.109" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" dependencies = [ "proc-macro2", "quote", "unicode-ident", ] [[package]] name = "syn" version = "2.0.95" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "46f71c0377baf4ef1cc3e3402ded576dccc315800fbc62dfc7fe04b009773b4a" dependencies = [ "proc-macro2", "quote", "unicode-ident", ] [[package]] name = "sync_wrapper" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2047c6ded9c721764247e62cd3b03c09ffc529b2ba5b10ec482ae507a4a70160" [[package]] name = "sync_wrapper" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" dependencies = [ "futures-core", ] [[package]] name = "synstructure" version = "0.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8af7666ab7b6390ab78131fb5b0fce11d6b7a6951602017c35fa82800708971" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "tagptr" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" [[package]] name = "take_mut" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f764005d11ee5f36500a149ace24e00e3da98b0158b3e2d53a7495660d3f4d60" [[package]] name = "tap" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" [[package]] name = "tempfile" version = "3.15.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9a8a559c81686f576e8cd0290cd2a24a2a9ad80c98b3478856500fcbd7acd704" dependencies = [ "cfg-if", "fastrand", "getrandom 0.2.15", "once_cell", "rustix", "windows-sys 0.59.0", ] [[package]] name = "tendril" version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d24a120c5fc464a3458240ee02c299ebcb9d67b5249c8848b09d639dca8d7bb0" dependencies = [ "futf", "mac", "utf-8", ] [[package]] name = "term" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c59df8ac95d96ff9bede18eb7300b0fda5e5d8d90960e76f8e14ae765eedbf1f" dependencies = [ "dirs-next", "rustversion", "winapi", ] [[package]] name = "thin-vec" version = "0.2.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a38c90d48152c236a3ab59271da4f4ae63d678c5d7ad6b7714d7cb9760be5e4b" [[package]] name = "thiserror" version = "1.0.69" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" dependencies = [ "thiserror-impl 1.0.69", ] [[package]] name = "thiserror" version = "2.0.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f072643fd0190df67a8bab670c20ef5d8737177d6ac6b2e9a236cb096206b2cc" dependencies = [ "thiserror-impl 2.0.9", ] [[package]] name = "thiserror-impl" version = "1.0.69" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "thiserror-impl" version = "2.0.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b50fa271071aae2e6ee85f842e2e28ba8cd2c5fb67f11fcb1fd70b276f9e7d4" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "thread_local" version = "1.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b9ef9bad013ada3808854ceac7b46812a6465ba368859a37e2100283d2d719c" dependencies = [ "cfg-if", "once_cell", ] [[package]] name = "threadpool" version = "1.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d050e60b33d41c19108b32cea32164033a9013fe3b46cbd4457559bfbf77afaa" dependencies = [ "num_cpus", ] [[package]] name = "thrift_codec" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "83d957f535b242b91aa9f47bde08080f9a6fef276477e55b0079979d002759d5" dependencies = [ "byteorder", "trackable", ] [[package]] name = "tikv-client" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "048968e4e3d04db472346770cc19914c6b5ae206fa44677f6a0874d54cd05940" dependencies = [ "async-recursion 0.3.2", "async-trait", "derive-new", "either", "fail", "futures", "lazy_static", "log", "pin-project", "prometheus", "prost 0.12.6", "rand 0.8.5", "regex", "semver", "serde", "serde_derive", "thiserror 1.0.69", "tokio", "tonic 0.10.2", ] [[package]] name = "time" version = "0.3.37" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "35e7868883861bd0e56d9ac6efcaaca0d6d5d82a2a7ec8209ff492c07cf37b21" dependencies = [ "deranged", "itoa", "num-conv", "powerfmt", "serde", "time-core", "time-macros", ] [[package]] name = "time-core" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ef927ca75afb808a4d64dd374f00a2adf8d0fcff8e7b184af886c3c87ec4a3f3" [[package]] name = "time-macros" version = "0.2.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2834e6017e3e5e4b9834939793b282bc03b37a3336245fa820e35e233e2a85de" dependencies = [ "num-conv", "time-core", ] [[package]] name = "tiny-keccak" version = "2.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" dependencies = [ "crunchy", ] [[package]] name = "tinystr" version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9117f5d4db391c1cf6927e7bea3db74b9a1c1add8f7eda9ffd5364f40f57b82f" dependencies = [ "displaydoc", "zerovec", ] [[package]] name = "tinytemplate" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be4d6b5f19ff7664e8c98d03e2139cb510db9b0a60b55f8e8709b689d939b6bc" dependencies = [ "serde", "serde_json", ] [[package]] name = "tinyvec" version = "1.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "022db8904dfa342efe721985167e9fcd16c29b226db4397ed752a761cfce81e8" dependencies = [ "tinyvec_macros", ] [[package]] name = "tinyvec_macros" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" version = "1.42.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5cec9b21b0450273377fc97bd4c33a8acffc8c996c987a7c5b319a0083707551" dependencies = [ "backtrace", "bytes", "libc", "mio 1.0.3", "parking_lot 0.12.3", "pin-project-lite", "signal-hook-registry", "socket2", "tokio-macros", "windows-sys 0.52.0", ] [[package]] name = "tokio-io-timeout" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "30b74022ada614a1b4834de765f9bb43877f910cc8ce4be40e89042c9223a8bf" dependencies = [ "pin-project-lite", "tokio", ] [[package]] name = "tokio-io-utility" version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8d672654d175710e52c7c41f6aec77c62b3c0954e2a7ebce9049d1e94ed7c263" dependencies = [ "bytes", "tokio", ] [[package]] name = "tokio-macros" version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "693d596312e88961bc67d7f1f97af8a70227d9f90c31bba5806eec004978d752" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "tokio-native-tls" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbae76ab933c85776efabc971569dd6119c580d8f5d448769dec1764bf796ef2" dependencies = [ "native-tls", "tokio", ] [[package]] name = "tokio-rustls" version = "0.24.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c28327cf380ac148141087fbfb9de9d7bd4e84ab5d2c28fbc911d753de8a7081" dependencies = [ "rustls 0.21.12", "tokio", ] [[package]] name = "tokio-rustls" version = "0.26.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5f6d0975eaace0cf0fcadee4e4aaa5da15b5c079146f2cffb67c113be122bf37" dependencies = [ "rustls 0.23.20", "tokio", ] [[package]] name = "tokio-stream" version = "0.1.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eca58d7bba4a75707817a2c44174253f9236b2d5fbd055602e9d5c07c139a047" dependencies = [ "futures-core", "pin-project-lite", "tokio", ] [[package]] name = "tokio-tungstenite" version = "0.23.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c6989540ced10490aaf14e6bad2e3d33728a2813310a0c71d1574304c49631cd" dependencies = [ "futures-util", "log", "rustls 0.23.20", "rustls-pki-types", "tokio", "tokio-rustls 0.26.1", "tungstenite", "webpki-roots 0.26.7", ] [[package]] name = "tokio-util" version = "0.7.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d7fcaa8d55a2bdd6b83ace262b016eca0d79ee02818c5c1bcdf0305114081078" dependencies = [ "bytes", "futures-core", "futures-io", "futures-sink", "pin-project-lite", "tokio", ] [[package]] name = "toml_datetime" version = "0.6.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0dd7358ecb8fc2f8d014bf86f6f638ce72ba252a2c3a2572f2a795f1d23efb41" [[package]] name = "toml_edit" version = "0.22.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ae48d6208a266e853d946088ed816055e556cc6028c5e8e2b84d9fa5dd7c7f5" dependencies = [ "indexmap 2.7.0", "toml_datetime", "winnow", ] [[package]] name = "tonic" version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d560933a0de61cf715926b9cac824d4c883c2c43142f787595e48280c40a1d0e" dependencies = [ "async-stream", "async-trait", "axum 0.6.20", "base64 0.21.7", "bytes", "h2 0.3.26", "http 0.2.12", "http-body 0.4.6", "hyper 0.14.32", "hyper-timeout 0.4.1", "percent-encoding", "pin-project", "prost 0.12.6", "rustls 0.21.12", "rustls-pemfile 1.0.4", "tokio", "tokio-rustls 0.24.1", "tokio-stream", "tower 0.4.13", "tower-layer", "tower-service", "tracing", ] [[package]] name = "tonic" version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877c5b330756d856ffcc4553ab34a5684481ade925ecc54bcd1bf02b1d0d4d52" dependencies = [ "async-stream", "async-trait", "axum 0.7.9", "base64 0.22.1", "bytes", "h2 0.4.7", "http 1.2.0", "http-body 1.0.1", "http-body-util", "hyper 1.5.2", "hyper-timeout 0.5.2", "hyper-util", "percent-encoding", "pin-project", "prost 0.13.4", "rustls-pemfile 2.2.0", "socket2", "tokio", "tokio-rustls 0.26.1", "tokio-stream", "tower 0.4.13", "tower-layer", "tower-service", "tracing", ] [[package]] name = "tonic-build" version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9557ce109ea773b399c9b9e5dca39294110b74f1f342cb347a80d1fce8c26a11" dependencies = [ "prettyplease", "proc-macro2", "prost-build", "prost-types 0.13.4", "quote", "syn 2.0.95", ] [[package]] name = "tower" version = "0.4.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c" dependencies = [ "futures-core", "futures-util", "indexmap 1.9.3", "pin-project", "pin-project-lite", "rand 0.8.5", "slab", "tokio", "tokio-util", "tower-layer", "tower-service", "tracing", ] [[package]] name = "tower" version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d039ad9159c98b70ecfd540b2573b97f7f52c3e8d9f8ad57a24b916a536975f9" dependencies = [ "futures-core", "futures-util", "pin-project-lite", "sync_wrapper 1.0.2", "tokio", "tower-layer", "tower-service", ] [[package]] name = "tower-layer" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" [[package]] name = "tower-service" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" [[package]] name = "tracing" version = "0.1.41" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "784e0ac535deb450455cbfa28a6f0df145ea1bb7ae51b821cf5e7927fdcfbdd0" dependencies = [ "log", "pin-project-lite", "tracing-attributes", "tracing-core", ] [[package]] name = "tracing-attributes" version = "0.1.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "395ae124c09f9e6918a2310af6038fba074bcf474ac352496d5910dd59a2226d" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "tracing-core" version = "0.1.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e672c95779cf947c5311f83787af4fa8fffd12fb27e4993211a84bdfd9610f9c" dependencies = [ "once_cell", "valuable", ] [[package]] name = "tracing-log" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3" dependencies = [ "log", "once_cell", "tracing-core", ] [[package]] name = "tracing-opentelemetry" version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "721f2d2569dce9f3dfbbddee5906941e953bfcdf736a62da3377f5751650cc36" dependencies = [ "js-sys", "once_cell", "opentelemetry", "opentelemetry_sdk", "smallvec", "tracing", "tracing-core", "tracing-log", "tracing-subscriber", "web-time", ] [[package]] name = "tracing-subscriber" version = "0.3.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e8189decb5ac0fa7bc8b96b7cb9b2701d60d48805aca84a238004d665fcc4008" dependencies = [ "matchers", "nu-ansi-term", "once_cell", "regex", "sharded-slab", "smallvec", "thread_local", "tracing", "tracing-core", "tracing-log", ] [[package]] name = "trackable" version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b15bd114abb99ef8cee977e517c8f37aee63f184f2d08e3e6ceca092373369ae" dependencies = [ "trackable_derive", ] [[package]] name = "trackable_derive" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebeb235c5847e2f82cfe0f07eb971d1e5f6804b18dac2ae16349cc604380f82f" dependencies = [ "quote", "syn 1.0.109", ] [[package]] name = "trice" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3aaab10ae9fac0b10f392752bf56f0fd20845f39037fec931e8537b105b515a" dependencies = [ "js-sys", "wasm-bindgen", "web-sys", ] [[package]] name = "trim-in-place" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "343e926fc669bc8cde4fa3129ab681c63671bae288b1f1081ceee6d9d37904fc" [[package]] name = "triomphe" version = "0.1.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ef8f7726da4807b58ea5c96fdc122f80702030edc33b35aff9190a51148ccc85" dependencies = [ "arc-swap", "serde", "stable_deref_trait", ] [[package]] name = "try-lock" version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" [[package]] name = "try_map" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fb1626d07cb5c1bb2cf17d94c0be4852e8a7c02b041acec9a8c5bdda99f9d580" [[package]] name = "tungstenite" version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6e2e2ce1e47ed2994fd43b04c8f618008d4cabdd5ee34027cf14f9d918edd9c8" dependencies = [ "byteorder", "bytes", "data-encoding", "http 1.2.0", "httparse", "log", "rand 0.8.5", "rustls 0.23.20", "rustls-pki-types", "sha1", "thiserror 1.0.69", "url", "utf-8", ] [[package]] name = "typed-builder" version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "89851716b67b937e393b3daa8423e67ddfc4bbbf1654bcf05488e95e0828db0c" dependencies = [ "proc-macro2", "quote", "syn 1.0.109", ] [[package]] name = "typenum" version = "1.17.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42ff0bf0c66b8238c6f3b578df37d0b7848e55df8577b3f74f92a69acceeb825" [[package]] name = "ucd-trie" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971" [[package]] name = "ulid" version = "1.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "04f903f293d11f31c0c29e4148f6dc0d033a7f80cebc0282bea147611667d289" dependencies = [ "getrandom 0.2.15", "rand 0.8.5", "serde", "web-time", ] [[package]] name = "unicase" version = "2.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "75b844d17643ee918803943289730bec8aac480150456169e647ed0b576ba539" [[package]] name = "unicode-bidi" version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5" [[package]] name = "unicode-ident" version = "1.0.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "adb9e6ca4f869e1180728b7950e35922a7fc6397f7b641499e8f3ef06e50dc83" [[package]] name = "unicode-normalization" version = "0.1.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5033c97c4262335cded6d6fc3e5c18ab755e1a3dc96376350f3d8e9f009ad956" dependencies = [ "tinyvec", ] [[package]] name = "unicode-properties" version = "0.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e70f2a8b45122e719eb623c01822704c4e0907e7e426a05927e1a1cfff5b75d0" [[package]] name = "unicode-script" version = "0.5.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9fb421b350c9aff471779e262955939f565ec18b86c15364e6bdf0d662ca7c1f" [[package]] name = "unicode-security" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2e4ddba1535dd35ed8b61c52166b7155d7f4e4b8847cec6f48e71dc66d8b5e50" dependencies = [ "unicode-normalization", "unicode-script", ] [[package]] name = "unicode-width" version = "0.1.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af" [[package]] name = "unicode-xid" version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" [[package]] name = "unsigned-varint" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eb066959b24b5196ae73cb057f45598450d2c5f71460e98c49b738086eff9c06" [[package]] name = "untrusted" version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" [[package]] name = "ureq" version = "2.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "02d1a66277ed75f640d608235660df48c8e3c19f3b4edb6a263315626cc3c01d" dependencies = [ "base64 0.22.1", "flate2", "log", "once_cell", "rustls 0.23.20", "rustls-pki-types", "url", "webpki-roots 0.26.7", ] [[package]] name = "url" version = "2.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32f8b686cadd1473f4bd0117a5d28d36b1ade384ea9b5069a1c40aefed7fda60" dependencies = [ "form_urlencoded", "idna", "percent-encoding", ] [[package]] name = "urlencoding" version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da" [[package]] name = "utf-8" version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9" [[package]] name = "utf16_iter" version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8232dd3cdaed5356e0f716d285e4b40b932ac434100fe9b7e0e8e935b9e6246" [[package]] name = "utf8_iter" version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" [[package]] name = "utf8parse" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" version = "1.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b3758f5e68192bb96cc8f9b7e2c2cfdabb435499a28499a42f8f984092adad4b" dependencies = [ "getrandom 0.2.15", "serde", "wasm-bindgen", ] [[package]] name = "valuable" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "830b7e5d4d90034032940e4ace0d9a9a057e7a45cd94e6c007832e39edb82f6d" [[package]] name = "value-bag" version = "1.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3ef4c4aa54d5d05a279399bfa921ec387b7aba77caf7a682ae8d86785b8fdad2" [[package]] name = "vart" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c92195d375eb94995afddeedfd7f246796eb60b85f727c538e42222c4c9b2d3" [[package]] name = "vcpkg" version = "0.2.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" [[package]] name = "vec-strings" version = "0.4.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8509489e2a7ee219522238ad45fd370bec6808811ac15ac6b07453804e77659" dependencies = [ "serde", "thin-vec", ] [[package]] name = "version_check" version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" [[package]] name = "walkdir" version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" dependencies = [ "same-file", "winapi-util", ] [[package]] name = "want" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" dependencies = [ "try-lock", ] [[package]] name = "wasi" version = "0.9.0+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cccddf32554fecc6acb585f82a32a72e28b48f8c4c1883ddfeeeaa96f7d8e519" [[package]] name = "wasi" version = "0.11.0+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423" [[package]] name = "wasite" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" [[package]] name = "wasix" version = "0.12.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c1fbb4ef9bbca0c1170e0b00dd28abc9e3b68669821600cad1caaed606583c6d" dependencies = [ "wasi 0.11.0+wasi-snapshot-preview1", ] [[package]] name = "wasm-bindgen" version = "0.2.99" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a474f6281d1d70c17ae7aa6a613c87fce69a127e2624002df63dcb39d6cf6396" dependencies = [ "cfg-if", "once_cell", "wasm-bindgen-macro", ] [[package]] name = "wasm-bindgen-backend" version = "0.2.99" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5f89bb38646b4f81674e8f5c3fb81b562be1fd936d84320f3264486418519c79" dependencies = [ "bumpalo", "log", "proc-macro2", "quote", "syn 2.0.95", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-futures" version = "0.4.49" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "38176d9b44ea84e9184eff0bc34cc167ed044f816accfe5922e54d84cf48eca2" dependencies = [ "cfg-if", "js-sys", "once_cell", "wasm-bindgen", "web-sys", ] [[package]] name = "wasm-bindgen-macro" version = "0.2.99" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2cc6181fd9a7492eef6fef1f33961e3695e4579b9872a6f7c83aee556666d4fe" dependencies = [ "quote", "wasm-bindgen-macro-support", ] [[package]] name = "wasm-bindgen-macro-support" version = "0.2.99" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "30d7a95b763d3c45903ed6c81f156801839e5ee968bb07e534c44df0fcd330c2" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", "wasm-bindgen-backend", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" version = "0.2.99" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "943aab3fdaaa029a6e0271b35ea10b72b943135afe9bffca82384098ad0e06a6" [[package]] name = "wasm-streams" version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "15053d8d85c7eccdbefef60f06769760a563c7f0a9d6902a13d35c7800b0ad65" dependencies = [ "futures-util", "js-sys", "wasm-bindgen", "wasm-bindgen-futures", "web-sys", ] [[package]] name = "wasmtimer" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c7ed9d8b15c7fb594d72bfb4b5a276f3d2029333cd93a932f376f5937f6f80ee" dependencies = [ "futures", "js-sys", "parking_lot 0.12.3", "pin-utils", "wasm-bindgen", ] [[package]] name = "weak-table" version = "0.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "323f4da9523e9a669e1eaf9c6e763892769b1d38c623913647bfdc1532fe4549" [[package]] name = "web-sys" version = "0.3.76" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "04dd7223427d52553d3702c004d3b2fe07c148165faa56313cb00211e31c12bc" dependencies = [ "js-sys", "wasm-bindgen", ] [[package]] name = "web-time" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" dependencies = [ "js-sys", "wasm-bindgen", ] [[package]] name = "webpki" version = "0.22.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed63aea5ce73d0ff405984102c42de94fc55a6b75765d621c65262469b3c9b53" dependencies = [ "ring", "untrusted", ] [[package]] name = "webpki-roots" version = "0.22.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6c71e40d7d2c34a5106301fb632274ca37242cd0c9d3e64dbece371a40a2d87" dependencies = [ "webpki", ] [[package]] name = "webpki-roots" version = "0.25.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5f20c57d8d7db6d3b86154206ae5d8fba62dd39573114de97c2cb0578251f8e1" [[package]] name = "webpki-roots" version = "0.26.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5d642ff16b7e79272ae451b7322067cdc17cadf68c23264be9d94a32319efe7e" dependencies = [ "rustls-pki-types", ] [[package]] name = "whoami" version = "1.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "372d5b87f58ec45c384ba03563b03544dc5fadc3983e434b286913f5b4a9bb6d" dependencies = [ "redox_syscall 0.5.8", "wasite", "web-sys", ] [[package]] name = "widestring" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7219d36b6eac893fa81e84ebe06485e7dcbb616177469b142df14f1f4deb1311" [[package]] name = "winapi" version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" dependencies = [ "winapi-i686-pc-windows-gnu", "winapi-x86_64-pc-windows-gnu", ] [[package]] name = "winapi-i686-pc-windows-gnu" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" [[package]] name = "winapi-util" version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf221c93e13a30d793f7645a0e7762c55d169dbb0a49671918a2319d289b10bb" dependencies = [ "windows-sys 0.59.0", ] [[package]] name = "winapi-x86_64-pc-windows-gnu" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windows" version = "0.48.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e686886bc078bc1b0b600cac0147aadb815089b6e4da64016cbd754b6342700f" dependencies = [ "windows-targets 0.48.5", ] [[package]] name = "windows" version = "0.58.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dd04d41d93c4992d421894c18c8b43496aa748dd4c081bac0dc93eb0489272b6" dependencies = [ "windows-core 0.58.0", "windows-targets 0.52.6", ] [[package]] name = "windows-core" version = "0.52.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "33ab640c8d7e35bf8ba19b884ba838ceb4fba93a4e8c65a9059d08afcfc683d9" dependencies = [ "windows-targets 0.52.6", ] [[package]] name = "windows-core" version = "0.58.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ba6d44ec8c2591c134257ce647b7ea6b20335bf6379a27dac5f1641fcf59f99" dependencies = [ "windows-implement", "windows-interface", "windows-result", "windows-strings", "windows-targets 0.52.6", ] [[package]] name = "windows-implement" version = "0.58.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2bbd5b46c938e506ecbce286b6628a02171d56153ba733b6c741fc627ec9579b" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "windows-interface" version = "0.58.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "053c4c462dc91d3b1504c6fe5a726dd15e216ba718e84a0e46a88fbe5ded3515" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "windows-registry" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e400001bb720a623c1c69032f8e3e4cf09984deec740f007dd2b03ec864804b0" dependencies = [ "windows-result", "windows-strings", "windows-targets 0.52.6", ] [[package]] name = "windows-result" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d1043d8214f791817bab27572aaa8af63732e11bf84aa21a45a78d6c317ae0e" dependencies = [ "windows-targets 0.52.6", ] [[package]] name = "windows-strings" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4cd9b125c486025df0eabcb585e62173c6c9eddcec5d117d3b6e8c30e2ee4d10" dependencies = [ "windows-result", "windows-targets 0.52.6", ] [[package]] name = "windows-sys" version = "0.48.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9" dependencies = [ "windows-targets 0.48.5", ] [[package]] name = "windows-sys" version = "0.52.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" dependencies = [ "windows-targets 0.52.6", ] [[package]] name = "windows-sys" version = "0.59.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b" dependencies = [ "windows-targets 0.52.6", ] [[package]] name = "windows-targets" version = "0.48.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c" dependencies = [ "windows_aarch64_gnullvm 0.48.5", "windows_aarch64_msvc 0.48.5", "windows_i686_gnu 0.48.5", "windows_i686_msvc 0.48.5", "windows_x86_64_gnu 0.48.5", "windows_x86_64_gnullvm 0.48.5", "windows_x86_64_msvc 0.48.5", ] [[package]] name = "windows-targets" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" dependencies = [ "windows_aarch64_gnullvm 0.52.6", "windows_aarch64_msvc 0.52.6", "windows_i686_gnu 0.52.6", "windows_i686_gnullvm", "windows_i686_msvc 0.52.6", "windows_x86_64_gnu 0.52.6", "windows_x86_64_gnullvm 0.52.6", "windows_x86_64_msvc 0.52.6", ] [[package]] name = "windows_aarch64_gnullvm" version = "0.48.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8" [[package]] name = "windows_aarch64_gnullvm" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" [[package]] name = "windows_aarch64_msvc" version = "0.48.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc" [[package]] name = "windows_aarch64_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" [[package]] name = "windows_i686_gnu" version = "0.48.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e" [[package]] name = "windows_i686_gnu" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" [[package]] name = "windows_i686_gnullvm" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" [[package]] name = "windows_i686_msvc" version = "0.48.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406" [[package]] name = "windows_i686_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" [[package]] name = "windows_x86_64_gnu" version = "0.48.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e" [[package]] name = "windows_x86_64_gnu" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" [[package]] name = "windows_x86_64_gnullvm" version = "0.48.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc" [[package]] name = "windows_x86_64_gnullvm" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" [[package]] name = "windows_x86_64_msvc" version = "0.48.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538" [[package]] name = "windows_x86_64_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" [[package]] name = "winnow" version = "0.6.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39281189af81c07ec09db316b302a3e67bf9bd7cbf6c820b50e35fee9c2fa980" dependencies = [ "memchr", ] [[package]] name = "winreg" version = "0.50.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "524e57b2c537c0f9b1e69f1965311ec12182b4122e45035b1508cd24d2adadb1" dependencies = [ "cfg-if", "windows-sys 0.48.0", ] [[package]] name = "write16" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d1890f4022759daae28ed4fe62859b1236caebfc61ede2f63ed4e695f3f6d936" [[package]] name = "writeable" version = "0.5.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e9df38ee2d2c3c5948ea468a8406ff0db0b29ae1ffde1bcf20ef305bcc95c51" [[package]] name = "ws_stream_wasm" version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7999f5f4217fe3818726b66257a4475f71e74ffd190776ad053fa159e50737f5" dependencies = [ "async_io_stream", "futures", "js-sys", "log", "pharos", "rustc_version", "send_wrapper", "thiserror 1.0.69", "wasm-bindgen", "wasm-bindgen-futures", "web-sys", ] [[package]] name = "wyz" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed" dependencies = [ "tap", ] [[package]] name = "xml-rs" version = "0.8.25" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c5b940ebc25896e71dd073bad2dbaa2abfe97b0a391415e22ad1326d9c54e3c4" [[package]] name = "xmlparser" version = "0.13.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4" [[package]] name = "xxhash-rust" version = "0.8.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fdd20c5420375476fbd4394763288da7eb0cc0b8c11deed431a91562af7335d3" [[package]] name = "yansi" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" [[package]] name = "yoke" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "120e6aef9aa629e3d4f52dc8cc43a015c7724194c97dfaf45180d2daf2b77f40" dependencies = [ "serde", "stable_deref_trait", "yoke-derive", "zerofrom", ] [[package]] name = "yoke-derive" version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2380878cad4ac9aac1e2435f3eb4020e8374b5f13c296cb75b4620ff8e229154" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", "synstructure", ] [[package]] name = "zerocopy" version = "0.7.35" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1b9b4fd18abc82b8136838da5d50bae7bdea537c574d8dc1a34ed098d6c166f0" dependencies = [ "byteorder", "zerocopy-derive", ] [[package]] name = "zerocopy-derive" version = "0.7.35" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fa4f8080344d4671fb4e831a13ad1e68092748387dfc4f55e356242fae12ce3e" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "zerofrom" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cff3ee08c995dee1859d998dea82f7374f2826091dd9cd47def953cae446cd2e" dependencies = [ "zerofrom-derive", ] [[package]] name = "zerofrom-derive" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "595eed982f7d355beb85837f651fa22e90b3c044842dc7f2c2842c086f295808" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", "synstructure", ] [[package]] name = "zeroize" version = "1.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ced3678a2879b30306d323f4542626697a464a97c0a07c9aebf7ebca65cd4dde" [[package]] name = "zerovec" version = "0.10.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "aa2b893d79df23bfb12d5461018d408ea19dfafe76c2c7ef6d4eba614f8ff079" dependencies = [ "yoke", "zerofrom", "zerovec-derive", ] [[package]] name = "zerovec-derive" version = "0.10.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6eafa6dfb17584ea3e2bd6e76e0cc15ad7af12b09abdd1ca55961bed9b1063c6" dependencies = [ "proc-macro2", "quote", "syn 2.0.95", ] [[package]] name = "zigzag" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "70b40401a28d86ce16a330b863b86fd7dbee4d7c940587ab09ab8c019f9e3fdf" dependencies = [ "num-traits", ] opendal-0.52.0/Cargo.toml0000644000000271460000000000100105520ustar # THIS FILE IS AUTOMATICALLY GENERATED BY CARGO # # When uploading crates to the registry Cargo will automatically # "normalize" Cargo.toml files for maximal compatibility # with all versions of Cargo and also rewrite `path` dependencies # to registry (e.g., crates.io) dependencies. # # If you are reading this file be aware that the original Cargo.toml # will likely look very different (and much more reasonable). # See Cargo.toml.orig for the original contents. [package] edition = "2021" rust-version = "1.75" name = "opendal" version = "0.52.0" authors = ["Apache OpenDAL "] build = false exclude = ["/tests/"] autolib = false autobins = false autoexamples = false autotests = false autobenches = false description = "Apache OpenDAL™: One Layer, All Storage." homepage = "https://opendal.apache.org/" readme = "README.md" keywords = [ "storage", "fs", "s3", "azblob", "gcs", ] categories = ["filesystem"] license = "Apache-2.0" repository = "https://github.com/apache/opendal" [package.metadata.docs.rs] all-features = true [features] default = [ "reqwest/rustls-tls", "executors-tokio", "services-memory", ] executors-tokio = ["tokio/rt"] internal-path-cache = ["dep:moka"] internal-tokio-rt = ["tokio/rt-multi-thread"] layers-async-backtrace = ["dep:async-backtrace"] layers-await-tree = ["dep:await-tree"] layers-blocking = ["internal-tokio-rt"] layers-chaos = ["dep:rand"] layers-dtrace = ["dep:probe"] layers-fastrace = ["dep:fastrace"] layers-metrics = ["dep:metrics"] layers-mime-guess = ["dep:mime_guess"] layers-otel-metrics = [ "dep:opentelemetry", "opentelemetry/metrics", ] layers-otel-trace = [ "dep:opentelemetry", "opentelemetry/trace", ] layers-prometheus = ["dep:prometheus"] layers-prometheus-client = ["dep:prometheus-client"] layers-throttle = ["dep:governor"] layers-tracing = ["dep:tracing"] services-aliyun-drive = [] services-alluxio = [] services-atomicserver = ["dep:atomic_lib"] services-azblob = [ "dep:sha2", "dep:reqsign", "reqsign?/services-azblob", "reqsign?/reqwest_request", ] services-azdls = [ "dep:reqsign", "reqsign?/services-azblob", "reqsign?/reqwest_request", ] services-azfile = [ "dep:reqsign", "reqsign?/services-azblob", "reqsign?/reqwest_request", ] services-b2 = [] services-cacache = ["dep:cacache"] services-chainsafe = [] services-cloudflare-kv = [] services-compfs = ["dep:compio"] services-cos = [ "dep:reqsign", "reqsign?/services-tencent", "reqsign?/reqwest_request", ] services-d1 = [] services-dashmap = ["dep:dashmap"] services-dbfs = [] services-dropbox = [] services-etcd = [ "dep:etcd-client", "dep:bb8", ] services-foundationdb = ["dep:foundationdb"] services-fs = [ "tokio/fs", "internal-tokio-rt", ] services-ftp = [ "dep:suppaftp", "dep:bb8", "dep:async-tls", ] services-gcs = [ "dep:reqsign", "reqsign?/services-google", "reqsign?/reqwest_request", ] services-gdrive = ["internal-path-cache"] services-ghac = [ "dep:ghac", "dep:prost", "services-azblob", ] services-github = [] services-gridfs = [ "dep:mongodb", "dep:mongodb-internal-macros", ] services-hdfs = ["dep:hdrs"] services-hdfs-native = ["hdfs-native"] services-http = [] services-huggingface = [] services-icloud = ["internal-path-cache"] services-ipfs = ["dep:prost"] services-ipmfs = [] services-koofr = [] services-lakefs = [] services-memcached = ["dep:bb8"] services-memory = [] services-mini-moka = ["dep:mini-moka"] services-moka = ["dep:moka"] services-mongodb = [ "dep:mongodb", "dep:mongodb-internal-macros", ] services-monoiofs = [ "dep:monoio", "dep:flume", ] services-mysql = [ "dep:sqlx", "sqlx?/mysql", ] services-nebula-graph = [ "dep:rust-nebula", "dep:bb8", "dep:snowflaked", ] services-obs = [ "dep:reqsign", "reqsign?/services-huaweicloud", "reqsign?/reqwest_request", ] services-onedrive = [] services-oss = [ "dep:reqsign", "reqsign?/services-aliyun", "reqsign?/reqwest_request", ] services-pcloud = [] services-persy = [ "dep:persy", "internal-tokio-rt", ] services-postgresql = [ "dep:sqlx", "sqlx?/postgres", ] services-redb = [ "dep:redb", "internal-tokio-rt", ] services-redis = [ "dep:redis", "dep:bb8", "redis?/tokio-rustls-comp", ] services-redis-native-tls = [ "services-redis", "redis?/tokio-native-tls-comp", ] services-rocksdb = [ "dep:rocksdb", "internal-tokio-rt", ] services-s3 = [ "dep:reqsign", "reqsign?/services-aws", "reqsign?/reqwest_request", "dep:crc32c", ] services-seafile = [] services-sftp = [ "dep:openssh", "dep:openssh-sftp-client", "dep:bb8", ] services-sled = [ "dep:sled", "internal-tokio-rt", ] services-sqlite = [ "dep:sqlx", "sqlx?/sqlite", "dep:ouroboros", ] services-supabase = [] services-surrealdb = ["dep:surrealdb"] services-swift = [] services-tikv = ["tikv-client"] services-upyun = [ "dep:hmac", "dep:sha1", ] services-vercel-artifacts = [] services-vercel-blob = [] services-webdav = [] services-webhdfs = [] services-yandex-disk = [] tests = [ "dep:rand", "dep:sha2", "dep:dotenvy", "layers-blocking", "services-azblob", "services-fs", "services-http", "services-memory", "internal-tokio-rt", "services-s3", ] [lib] name = "opendal" path = "src/lib.rs" bench = false [[bench]] name = "ops" path = "benches/ops/main.rs" harness = false required-features = ["tests"] [[bench]] name = "types" path = "benches/types/main.rs" harness = false required-features = ["tests"] [dependencies.anyhow] version = "1.0.30" features = ["std"] [dependencies.async-backtrace] version = "0.2.6" optional = true [dependencies.async-tls] version = "0.13.0" optional = true [dependencies.async-trait] version = "0.1.68" [dependencies.atomic_lib] version = "0.39.0" optional = true [dependencies.await-tree] version = "0.2" optional = true [dependencies.backon] version = "1.2" features = ["tokio-sleep"] [dependencies.base64] version = "0.22" [dependencies.bb8] version = "0.8" optional = true [dependencies.bytes] version = "1.6" [dependencies.cacache] version = "13.0" features = [ "tokio-runtime", "mmap", ] optional = true default-features = false [dependencies.chrono] version = "0.4.28" features = [ "clock", "std", ] default-features = false [dependencies.compio] version = "0.12.0" features = [ "runtime", "bytes", "polling", "dispatcher", ] optional = true [dependencies.crc32c] version = "0.6.6" optional = true [dependencies.dashmap] version = "6" optional = true [dependencies.dotenvy] version = "0.15" optional = true [dependencies.etcd-client] version = "0.14" features = ["tls"] optional = true [dependencies.fastrace] version = "0.7.1" optional = true [dependencies.flume] version = "0.11" optional = true [dependencies.foundationdb] version = "0.9.0" features = [ "embedded-fdb-include", "fdb-7_3", ] optional = true [dependencies.futures] version = "0.3" features = [ "std", "async-await", ] default-features = false [dependencies.ghac] version = "0.2.0" optional = true [dependencies.governor] version = "0.6.0" features = ["std"] optional = true [dependencies.hdfs-native] version = "0.10" optional = true [dependencies.hdrs] version = "0.3.2" features = ["async_file"] optional = true [dependencies.hmac] version = "0.12.1" optional = true [dependencies.http] version = "1.1" [dependencies.log] version = "0.4" [dependencies.md-5] version = "0.10" [dependencies.metrics] version = "0.24" optional = true [dependencies.mime_guess] version = "2.0.5" optional = true [dependencies.mini-moka] version = "0.10" optional = true [dependencies.moka] version = "0.12" features = [ "future", "sync", ] optional = true [dependencies.mongodb] version = ">=3,<3.2.0" optional = true [dependencies.mongodb-internal-macros] version = ">=3,<3.2.0" optional = true [dependencies.monoio] version = "0.2.4" features = [ "sync", "mkdirat", "unlinkat", "renameat", ] optional = true [dependencies.once_cell] version = "1" [dependencies.openssh] version = "0.11.0" optional = true [dependencies.openssh-sftp-client] version = "0.15.2" features = [ "openssh", "tracing", ] optional = true [dependencies.opentelemetry] version = "0.28" optional = true [dependencies.ouroboros] version = "0.18.4" optional = true [dependencies.percent-encoding] version = "2" [dependencies.persy] version = "1.4.6" optional = true [dependencies.probe] version = "0.5.1" optional = true [dependencies.prometheus] version = "0.13" features = ["process"] optional = true [dependencies.prometheus-client] version = "0.23.1" optional = true [dependencies.prost] version = "0.13" optional = true [dependencies.quick-xml] version = "0.36" features = [ "serialize", "overlapped-lists", ] [dependencies.rand] version = "0.8" optional = true [dependencies.redb] version = "2" optional = true [dependencies.redis] version = "0.27" features = [ "cluster-async", "tokio-comp", "connection-manager", ] optional = true [dependencies.reqsign] version = "0.16.1" optional = true default-features = false [dependencies.reqwest] version = "0.12.2" features = ["stream"] default-features = false [dependencies.rocksdb] version = "0.21.0" optional = true default-features = false [dependencies.rust-nebula] version = "^0.0.2" features = ["graph"] optional = true [dependencies.serde] version = "1" features = ["derive"] [dependencies.serde_json] version = "1" [dependencies.sha1] version = "0.10.6" optional = true [dependencies.sha2] version = "0.10" optional = true [dependencies.sled] version = "0.34.7" optional = true [dependencies.snowflaked] version = "1" features = ["sync"] optional = true [dependencies.sqlx] version = "0.8.0" features = ["runtime-tokio-rustls"] optional = true [dependencies.suppaftp] version = "6.0.3" features = [ "async-secure", "rustls", "async-rustls", ] optional = true default-features = false [dependencies.surrealdb] version = "2" features = ["protocol-http"] optional = true [dependencies.tikv-client] version = "0.3.0" optional = true default-features = false [dependencies.tokio] version = "1.27" features = [ "sync", "io-util", ] [dependencies.tracing] version = "0.1" optional = true [dependencies.uuid] version = "1" features = [ "serde", "v4", ] [dev-dependencies.criterion] version = "0.5" features = [ "async", "async_tokio", ] [dev-dependencies.dotenvy] version = "0.15" [dev-dependencies.fastrace] version = "0.7" features = ["enable"] [dev-dependencies.fastrace-jaeger] version = "0.7" [dev-dependencies.libtest-mimic] version = "0.8" [dev-dependencies.opentelemetry] version = "0.28" features = ["trace"] default-features = false [dev-dependencies.opentelemetry-otlp] version = "0.28" features = ["grpc-tonic"] [dev-dependencies.opentelemetry_sdk] version = "0.28" features = ["rt-tokio"] [dev-dependencies.pretty_assertions] version = "1" [dev-dependencies.rand] version = "0.8" [dev-dependencies.sha2] version = "0.10" [dev-dependencies.size] version = "0.4" [dev-dependencies.tokio] version = "1.27" features = [ "fs", "macros", "rt-multi-thread", ] [dev-dependencies.tracing-opentelemetry] version = "0.29.0" [dev-dependencies.tracing-subscriber] version = "0.3" features = [ "env-filter", "tracing-log", ] [target.'cfg(target_arch = "wasm32")'.dependencies.backon] version = "1.2" features = ["gloo-timers-sleep"] [target.'cfg(target_arch = "wasm32")'.dependencies.getrandom] version = "0.2" features = ["js"] [target.'cfg(target_arch = "wasm32")'.dependencies.tokio] version = "1.27" features = ["time"] [lints.clippy] unused_async = "warn" opendal-0.52.0/Cargo.toml.orig000064400000000000000000000307601046102023000142270ustar 00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. [package] categories = ["filesystem"] description = "Apache OpenDAL™: One Layer, All Storage." exclude = ["/tests/"] keywords = ["storage", "fs", "s3", "azblob", "gcs"] name = "opendal" authors = ["Apache OpenDAL "] edition = "2021" homepage = "https://opendal.apache.org/" license = "Apache-2.0" repository = "https://github.com/apache/opendal" rust-version = "1.75" version = "0.52.0" [lints.clippy] unused_async = "warn" [package.metadata.docs.rs] all-features = true [workspace] default-members = ["."] members = [".", "examples/*", "fuzz", "edge/*", "benches/vs_*"] [workspace.package] edition = "2021" license = "Apache-2.0" rust-version = "1.75" version = "0.51.1" [features] default = ["reqwest/rustls-tls", "executors-tokio", "services-memory"] # Build test utils or not. # # These features are used to control whether to build opendal's test utils. # And doesn't have any other effects. # # You should never enable this feature unless you are developing opendal. tests = [ "dep:rand", "dep:sha2", "dep:dotenvy", "layers-blocking", "services-azblob", "services-fs", "services-http", "services-memory", "internal-tokio-rt", "services-s3", ] # Enable path cache. # This is an internal feature, and should not be used by users. internal-path-cache = ["dep:moka"] # Enable tokio runtime. internal-tokio-rt = ["tokio/rt-multi-thread"] # Enable tokio executors support. executors-tokio = ["tokio/rt"] # Enable layers chaos support layers-chaos = ["dep:rand"] # Enable layers metrics support layers-metrics = ["dep:metrics"] # Enable layers mime_guess support layers-mime-guess = ["dep:mime_guess"] # Enable layers prometheus support, with tikv/prometheus-rs crate layers-prometheus = ["dep:prometheus"] # Enable layers prometheus support, with prometheus-client crate layers-prometheus-client = ["dep:prometheus-client"] # Enable layers fastrace support. layers-fastrace = ["dep:fastrace"] # Enable layers tracing support. layers-tracing = ["dep:tracing"] # Enable layers otelmetrics support. layers-otel-metrics = ["dep:opentelemetry", "opentelemetry/metrics"] # Enable layers oteltrace support. layers-otel-trace = ["dep:opentelemetry", "opentelemetry/trace"] # Enable layers throttle support. layers-throttle = ["dep:governor"] # Enable layers await-tree support. layers-await-tree = ["dep:await-tree"] # Enable layers async-backtrace support. layers-async-backtrace = ["dep:async-backtrace"] # Enable dtrace support. layers-blocking = ["internal-tokio-rt"] layers-dtrace = ["dep:probe"] services-aliyun-drive = [] services-alluxio = [] services-atomicserver = ["dep:atomic_lib"] services-azblob = [ "dep:sha2", "dep:reqsign", "reqsign?/services-azblob", "reqsign?/reqwest_request", ] services-azdls = [ "dep:reqsign", "reqsign?/services-azblob", "reqsign?/reqwest_request", ] services-azfile = [ "dep:reqsign", "reqsign?/services-azblob", "reqsign?/reqwest_request", ] services-b2 = [] services-cacache = ["dep:cacache"] services-chainsafe = [] services-cloudflare-kv = [] services-compfs = ["dep:compio"] services-cos = [ "dep:reqsign", "reqsign?/services-tencent", "reqsign?/reqwest_request", ] services-d1 = [] services-dashmap = ["dep:dashmap"] services-dbfs = [] services-dropbox = [] services-etcd = ["dep:etcd-client", "dep:bb8"] services-foundationdb = ["dep:foundationdb"] services-fs = ["tokio/fs", "internal-tokio-rt"] services-ftp = ["dep:suppaftp", "dep:bb8", "dep:async-tls"] services-gcs = [ "dep:reqsign", "reqsign?/services-google", "reqsign?/reqwest_request", ] services-gdrive = ["internal-path-cache"] services-ghac = ["dep:ghac", "dep:prost", "services-azblob"] services-github = [] services-gridfs = ["dep:mongodb", "dep:mongodb-internal-macros"] services-hdfs = ["dep:hdrs"] services-hdfs-native = ["hdfs-native"] services-http = [] services-huggingface = [] services-icloud = ["internal-path-cache"] services-ipfs = ["dep:prost"] services-ipmfs = [] services-koofr = [] services-lakefs = [] services-memcached = ["dep:bb8"] services-memory = [] services-mini-moka = ["dep:mini-moka"] services-moka = ["dep:moka"] services-mongodb = ["dep:mongodb", "dep:mongodb-internal-macros"] services-monoiofs = ["dep:monoio", "dep:flume"] services-mysql = ["dep:sqlx", "sqlx?/mysql"] services-nebula-graph = ["dep:rust-nebula", "dep:bb8", "dep:snowflaked"] services-obs = [ "dep:reqsign", "reqsign?/services-huaweicloud", "reqsign?/reqwest_request", ] services-onedrive = [] services-oss = [ "dep:reqsign", "reqsign?/services-aliyun", "reqsign?/reqwest_request", ] services-pcloud = [] services-persy = ["dep:persy", "internal-tokio-rt"] services-postgresql = ["dep:sqlx", "sqlx?/postgres"] services-redb = ["dep:redb", "internal-tokio-rt"] services-redis = ["dep:redis", "dep:bb8", "redis?/tokio-rustls-comp"] services-redis-native-tls = ["services-redis", "redis?/tokio-native-tls-comp"] services-rocksdb = ["dep:rocksdb", "internal-tokio-rt"] services-s3 = [ "dep:reqsign", "reqsign?/services-aws", "reqsign?/reqwest_request", "dep:crc32c", ] services-seafile = [] services-sftp = ["dep:openssh", "dep:openssh-sftp-client", "dep:bb8"] services-sled = ["dep:sled", "internal-tokio-rt"] services-sqlite = ["dep:sqlx", "sqlx?/sqlite", "dep:ouroboros"] services-supabase = [] services-surrealdb = ["dep:surrealdb"] services-swift = [] services-tikv = ["tikv-client"] services-upyun = ["dep:hmac", "dep:sha1"] services-vercel-artifacts = [] services-vercel-blob = [] services-webdav = [] services-webhdfs = [] services-yandex-disk = [] [lib] bench = false [[bench]] harness = false name = "ops" required-features = ["tests"] [[bench]] harness = false name = "types" required-features = ["tests"] [[test]] harness = false name = "behavior" path = "tests/behavior/main.rs" required-features = ["tests"] [dependencies] async-tls = { version = "0.13.0", optional = true } # Required dependencies anyhow = { version = "1.0.30", features = ["std"] } async-trait = "0.1.68" backon = { version = "1.2", features = ["tokio-sleep"] } base64 = "0.22" bytes = "1.6" chrono = { version = "0.4.28", default-features = false, features = [ "clock", "std", ] } futures = { version = "0.3", default-features = false, features = [ "std", "async-await", ] } http = "1.1" log = "0.4" md-5 = "0.10" # TODO: remove once_cell when lazy_lock is stable: https://doc.rust-lang.org/std/cell/struct.LazyCell.html ghac = { version = "0.2.0", optional = true } once_cell = "1" percent-encoding = "2" quick-xml = { version = "0.36", features = ["serialize", "overlapped-lists"] } reqwest = { version = "0.12.2", features = [ "stream", ], default-features = false } serde = { version = "1", features = ["derive"] } serde_json = "1" tokio = { version = "1.27", features = ["sync", "io-util"] } uuid = { version = "1", features = ["serde", "v4"] } # Test only dependencies dotenvy = { version = "0.15", optional = true } rand = { version = "0.8", optional = true } # Optional dependencies # Services # general dependencies. bb8 = { version = "0.8", optional = true } prost = { version = "0.13", optional = true } sha1 = { version = "0.10.6", optional = true } sha2 = { version = "0.10", optional = true } sqlx = { version = "0.8.0", features = [ "runtime-tokio-rustls", ], optional = true } # For http based services. reqsign = { version = "0.16.1", default-features = false, optional = true } # for self-referencing structs ouroboros = { version = "0.18.4", optional = true } # for services-atomic-server atomic_lib = { version = "0.39.0", optional = true } # for services-cacache cacache = { version = "13.0", default-features = false, features = [ "tokio-runtime", "mmap", ], optional = true } # for services-dashmap dashmap = { version = "6", optional = true } # for services-etcd etcd-client = { version = "0.14", optional = true, features = ["tls"] } # for services-foundationdb foundationdb = { version = "0.9.0", features = [ "embedded-fdb-include", "fdb-7_3", ], optional = true } # for services-hdfs hdrs = { version = "0.3.2", optional = true, features = ["async_file"] } # for services-upyun hmac = { version = "0.12.1", optional = true } # for services-mini-moka mini-moka = { version = "0.10", optional = true } # for services-moka moka = { version = "0.12", optional = true, features = ["future", "sync"] } # for services-mongodb # mongodb has known issues on 3.2.0: https://github.com/mongodb/mongo-rust-driver/issues/1287 mongodb = { version = ">=3,<3.2.0", optional = true } mongodb-internal-macros = { version = ">=3,<3.2.0", optional = true } # for services-sftp openssh = { version = "0.11.0", optional = true } openssh-sftp-client = { version = "0.15.2", optional = true, features = [ "openssh", "tracing", ] } # for services-persy persy = { version = "1.4.6", optional = true } # for services-redb redb = { version = "2", optional = true } # for services-redis redis = { version = "0.27", features = [ "cluster-async", "tokio-comp", "connection-manager", ], optional = true } # for services-rocksdb rocksdb = { version = "0.21.0", default-features = false, optional = true } # for services-sled sled = { version = "0.34.7", optional = true } # for services-ftp suppaftp = { version = "6.0.3", default-features = false, features = [ "async-secure", "rustls", "async-rustls", ], optional = true } # for services-tikv tikv-client = { version = "0.3.0", optional = true, default-features = false } # for services-hdfs-native hdfs-native = { version = "0.10", optional = true } # for services-surrealdb surrealdb = { version = "2", optional = true, features = ["protocol-http"] } # for services-compfs compio = { version = "0.12.0", optional = true, features = [ "runtime", "bytes", "polling", "dispatcher", ] } # for services-s3 crc32c = { version = "0.6.6", optional = true } # for services-nebula-graph rust-nebula = { version = "^0.0.2", optional = true, features = ["graph"] } snowflaked = { version = "1", optional = true, features = ["sync"] } # for services-monoiofs flume = { version = "0.11", optional = true } monoio = { version = "0.2.4", optional = true, features = [ "sync", "mkdirat", "unlinkat", "renameat", ] } # Layers # for layers-async-backtrace async-backtrace = { version = "0.2.6", optional = true } # for layers-await-tree await-tree = { version = "0.2", optional = true } # for layers-throttle governor = { version = "0.6.0", optional = true, features = ["std"] } # for layers-metrics metrics = { version = "0.24", optional = true } # for layers-mime-guess mime_guess = { version = "2.0.5", optional = true } # for layers-fastrace fastrace = { version = "0.7.1", optional = true } # for layers-opentelemetry opentelemetry = { version = "0.28", optional = true } # for layers-prometheus prometheus = { version = "0.13", features = ["process"], optional = true } # for layers-prometheus-client prometheus-client = { version = "0.23.1", optional = true } # for layers-tracing tracing = { version = "0.1", optional = true } # for layers-dtrace probe = { version = "0.5.1", optional = true } [target.'cfg(target_arch = "wasm32")'.dependencies] backon = { version = "1.2", features = ["gloo-timers-sleep"] } getrandom = { version = "0.2", features = ["js"] } tokio = { version = "1.27", features = ["time"] } [dev-dependencies] criterion = { version = "0.5", features = ["async", "async_tokio"] } dotenvy = "0.15" fastrace = { version = "0.7", features = ["enable"] } fastrace-jaeger = "0.7" libtest-mimic = "0.8" opentelemetry = { version = "0.28", default-features = false, features = [ "trace", ] } opentelemetry-otlp = { version = "0.28", features = ["grpc-tonic"] } opentelemetry_sdk = { version = "0.28", features = ["rt-tokio"] } pretty_assertions = "1" rand = "0.8" sha2 = "0.10" size = "0.4" tokio = { version = "1.27", features = ["fs", "macros", "rt-multi-thread"] } tracing-opentelemetry = "0.29.0" tracing-subscriber = { version = "0.3", features = [ "env-filter", "tracing-log", ] } opendal-0.52.0/DEPENDENCIES.md000064400000000000000000000001271046102023000135020ustar 00000000000000# Dependencies Refer to [DEPENDENCIES.rust.tsv](DEPENDENCIES.rust.tsv) for full list. opendal-0.52.0/DEPENDENCIES.rust.tsv000064400000000000000000000136131046102023000147160ustar 00000000000000crate 0BSD Apache-2.0 Apache-2.0 WITH LLVM-exception BSD-2-Clause BSD-3-Clause BSL-1.0 ISC MIT MPL-2.0 OpenSSL Unicode-3.0 Unlicense Zlib addr2line@0.24.2 X X adler2@2.0.0 X X X android-tzdata@0.1.1 X X android_system_properties@0.1.5 X X anyhow@1.0.95 X X async-trait@0.1.85 X X atomic-waker@1.1.2 X X autocfg@1.4.0 X X backon@1.3.0 X backtrace@0.3.74 X X base64@0.22.1 X X block-buffer@0.10.4 X X bumpalo@3.16.0 X X byteorder@1.5.0 X X bytes@1.9.0 X cc@1.2.7 X X cfg-if@1.0.0 X X chrono@0.4.39 X X const-oid@0.9.6 X X core-foundation-sys@0.8.7 X X cpufeatures@0.2.16 X X crc32c@0.6.8 X X crypto-common@0.1.6 X X digest@0.10.7 X X displaydoc@0.2.5 X X dotenvy@0.15.7 X equivalent@1.0.1 X X fastrand@2.3.0 X X fnv@1.0.7 X X form_urlencoded@1.2.1 X X futures@0.3.31 X X futures-channel@0.3.31 X X futures-core@0.3.31 X X futures-io@0.3.31 X X futures-macro@0.3.31 X X futures-sink@0.3.31 X X futures-task@0.3.31 X X futures-util@0.3.31 X X generic-array@0.14.7 X getrandom@0.2.15 X X gimli@0.31.1 X X gloo-timers@0.3.0 X X h2@0.4.7 X hashbrown@0.15.2 X X hex@0.4.3 X X hmac@0.12.1 X X home@0.5.11 X X http@1.2.0 X X http-body@1.0.1 X http-body-util@0.1.2 X httparse@1.9.5 X X httpdate@1.0.3 X X hyper@1.5.2 X hyper-rustls@0.27.5 X X X hyper-util@0.1.10 X iana-time-zone@0.1.61 X X iana-time-zone-haiku@0.1.2 X X icu_collections@1.5.0 X icu_locid@1.5.0 X icu_locid_transform@1.5.0 X icu_locid_transform_data@1.5.0 X icu_normalizer@1.5.0 X icu_normalizer_data@1.5.0 X icu_properties@1.5.1 X icu_properties_data@1.5.0 X icu_provider@1.5.0 X icu_provider_macros@1.5.0 X idna@1.0.3 X X idna_adapter@1.2.0 X X indexmap@2.7.0 X X ipnet@2.10.1 X X itoa@1.0.14 X X js-sys@0.3.76 X X libc@0.2.169 X X litemap@0.7.4 X log@0.4.22 X X md-5@0.10.6 X X memchr@2.7.4 X X mime@0.3.17 X X miniz_oxide@0.8.2 X X X mio@1.0.3 X num-traits@0.2.19 X X object@0.36.7 X X once_cell@1.20.2 X X opendal@0.52.0 X percent-encoding@2.3.1 X X pin-project-lite@0.2.16 X X pin-utils@0.1.0 X X ppv-lite86@0.2.20 X X proc-macro2@1.0.92 X X quick-xml@0.35.0 X quick-xml@0.36.2 X quote@1.0.38 X X rand@0.8.5 X X rand_chacha@0.3.1 X X rand_core@0.6.4 X X reqsign@0.16.1 X reqwest@0.12.12 X X ring@0.17.8 X rustc-demangle@0.1.24 X X rustc_version@0.4.1 X X rustls@0.23.20 X X X rustls-pemfile@2.2.0 X X X rustls-pki-types@1.10.1 X X rustls-webpki@0.102.8 X ryu@1.0.18 X X semver@1.0.24 X X serde@1.0.217 X X serde_derive@1.0.217 X X serde_json@1.0.135 X X serde_urlencoded@0.7.1 X X sha1@0.10.6 X X sha2@0.10.8 X X shlex@1.3.0 X X slab@0.4.9 X smallvec@1.13.2 X X socket2@0.5.8 X X spin@0.9.8 X stable_deref_trait@1.2.0 X X subtle@2.6.1 X syn@2.0.95 X X sync_wrapper@1.0.2 X synstructure@0.13.1 X tinystr@0.7.6 X tokio@1.42.0 X tokio-macros@2.4.0 X tokio-rustls@0.26.1 X X tokio-util@0.7.13 X tower@0.5.2 X tower-layer@0.3.3 X tower-service@0.3.3 X tracing@0.1.41 X tracing-attributes@0.1.28 X tracing-core@0.1.33 X try-lock@0.2.5 X typenum@1.17.0 X X unicode-ident@1.0.14 X X X untrusted@0.9.0 X url@2.5.4 X X utf16_iter@1.0.5 X X utf8_iter@1.0.4 X X uuid@1.12.1 X X version_check@0.9.5 X X want@0.3.1 X wasi@0.11.0+wasi-snapshot-preview1 X X X wasm-bindgen@0.2.99 X X wasm-bindgen-backend@0.2.99 X X wasm-bindgen-futures@0.4.49 X X wasm-bindgen-macro@0.2.99 X X wasm-bindgen-macro-support@0.2.99 X X wasm-bindgen-shared@0.2.99 X X wasm-streams@0.4.2 X X web-sys@0.3.76 X X webpki-roots@0.26.7 X windows-core@0.52.0 X X windows-registry@0.2.0 X X windows-result@0.2.0 X X windows-strings@0.1.0 X X windows-sys@0.52.0 X X windows-sys@0.59.0 X X windows-targets@0.52.6 X X windows_aarch64_gnullvm@0.52.6 X X windows_aarch64_msvc@0.52.6 X X windows_i686_gnu@0.52.6 X X windows_i686_gnullvm@0.52.6 X X windows_i686_msvc@0.52.6 X X windows_x86_64_gnu@0.52.6 X X windows_x86_64_gnullvm@0.52.6 X X windows_x86_64_msvc@0.52.6 X X write16@1.0.0 X X writeable@0.5.5 X yoke@0.7.5 X yoke-derive@0.7.5 X zerocopy@0.7.35 X X X zerocopy-derive@0.7.35 X X X zerofrom@0.1.5 X zerofrom-derive@0.1.5 X zeroize@1.8.1 X X zerovec@0.10.4 X zerovec-derive@0.10.3 X opendal-0.52.0/LICENSE000064400000000000000000000261351046102023000123460ustar 00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. opendal-0.52.0/README.md000064400000000000000000000325521046102023000126200ustar 00000000000000# Apache OpenDAL™: One Layer, All Storage. [![Build Status]][actions] [![Latest Version]][crates.io] [![Crate Downloads]][crates.io] [![chat]][discord] [build status]: https://img.shields.io/github/actions/workflow/status/apache/opendal/ci_core.yml?branch=main [actions]: https://github.com/apache/opendal/actions?query=branch%3Amain [latest version]: https://img.shields.io/crates/v/opendal.svg [crates.io]: https://crates.io/crates/opendal [crate downloads]: https://img.shields.io/crates/d/opendal.svg [chat]: https://img.shields.io/discord/1081052318650339399 [discord]: https://opendal.apache.org/discord Apache OpenDAL™ is an Open Data Access Layer that enables seamless interaction with diverse storage services. OpenDAL Architectural ## Useful Links - Documentation: [release](https://docs.rs/opendal/) | [dev](https://opendal.apache.org/docs/rust/opendal/) - [Examples](./examples) - [Release Notes](https://docs.rs/opendal/latest/opendal/docs/changelog/index.html) - [Upgrade Guide](https://docs.rs/opendal/latest/opendal/docs/upgrade/index.html) - [RFC List](https://docs.rs/opendal/latest/opendal/docs/rfcs/index.html) ## Services OpenDAL supports the following storage [services](https://docs.rs/opendal/latest/opendal/services/index.html): | Type | Services | |--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------| | Standard Storage Protocols | ftp http [sftp] [webdav] | | Object Storage Services | [azblob] [cos] [gcs] [obs] [oss] [s3]
[b2] [openstack_swift] [upyun] [vercel_blob] | | File Storage Services | fs [alluxio] [azdls] [azfile] [chainsafe] [compfs]
[dbfs] [gridfs] [hdfs] [hdfs_native] [ipfs] [webhdfs] | | Consumer Cloud Storage Service | [aliyun_drive] [gdrive] [onedrive] [dropbox] [icloud] [koofr]
[pcloud] [seafile] [yandex_disk] | | Key-Value Storage Services | [cacache] [cloudflare_kv] [dashmap] memory [etcd]
[foundationdb] [persy] [redis] [rocksdb] [sled]
[redb] [tikv] [atomicserver] | | Database Storage Services | [d1] [mongodb] [mysql] [postgresql] [sqlite] [surrealdb] | | Cache Storage Services | [ghac] [memcached] [mini_moka] [moka] [vercel_artifacts] | | Git Based Storage Services | [huggingface] | [sftp]: https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02 [webdav]: https://datatracker.ietf.org/doc/html/rfc4918 [azblob]: https://azure.microsoft.com/en-us/services/storage/blobs/ [cos]: https://www.tencentcloud.com/products/cos [gcs]: https://cloud.google.com/storage [obs]: https://www.huaweicloud.com/intl/en-us/product/obs.html [oss]: https://www.aliyun.com/product/oss [s3]: https://aws.amazon.com/s3/ [b2]: https://www.backblaze.com/ [openstack_swift]: https://docs.openstack.org/swift/latest/ [upyun]: https://www.upyun.com/ [vercel_blob]: https://vercel.com/docs/storage/vercel-blob [alluxio]: https://docs.alluxio.io/os/user/stable/en/api/REST-API.html [azdls]: https://azure.microsoft.com/en-us/products/storage/data-lake-storage/ [azfile]: https://learn.microsoft.com/en-us/rest/api/storageservices/file-service-rest-api [chainsafe]: https://storage.chainsafe.io/ [compfs]: https://github.com/compio-rs/compio/ [dbfs]: https://docs.databricks.com/en/dbfs/index.html [gridfs]: https://www.mongodb.com/docs/manual/core/gridfs/ [hdfs]: https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html [hdfs_native]: https://github.com/Kimahriman/hdfs-native [ipfs]: https://ipfs.tech/ [webhdfs]: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html [aliyun_drive]: https://www.aliyundrive.com/ [gdrive]: https://www.google.com/drive/ [onedrive]: https://www.microsoft.com/en-us/microsoft-365/onedrive/online-cloud-storage [dropbox]: https://www.dropbox.com/ [icloud]: https://www.icloud.com/iclouddrive [koofr]: https://koofr.eu/ [pcloud]: https://www.pcloud.com/ [seafile]: https://www.seafile.com/ [yandex_disk]: https://360.yandex.com/disk/ [cacache]: https://crates.io/crates/cacache [cloudflare_kv]: https://developers.cloudflare.com/kv/ [dashmap]: https://github.com/xacrimon/dashmap [etcd]: https://etcd.io/ [foundationdb]: https://www.foundationdb.org/ [persy]: https://crates.io/crates/persy [redis]: https://redis.io/ [rocksdb]: http://rocksdb.org/ [sled]: https://crates.io/crates/sled [redb]: https://crates.io/crates/redb [tikv]: https://tikv.org/ [atomicserver]: https://github.com/atomicdata-dev/atomic-server [d1]: https://developers.cloudflare.com/d1/ [mongodb]: https://www.mongodb.com/ [mysql]: https://www.mysql.com/ [postgresql]: https://www.postgresql.org/ [sqlite]: https://www.sqlite.org/ [surrealdb]: https://surrealdb.com/ [ghac]: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows [memcached]: https://memcached.org/ [mini_moka]: https://github.com/moka-rs/mini-moka [moka]: https://github.com/moka-rs/moka [vercel_artifacts]: https://vercel.com/docs/concepts/monorepos/remote-caching [huggingface]: https://huggingface.co/ ## Layers OpenDAL supports the following storage [layers](https://docs.rs/opendal/latest/opendal/layers/index.html) to extend the behavior: | Name | Depends | Description | |---------------------------|--------------------------|---------------------------------------------------------------------------------------| | [`AsyncBacktraceLayer`] | [async-backtrace] | Add Efficient, logical 'stack' traces of async functions for the underlying services. | | [`AwaitTreeLayer`] | [await-tree] | Add a Instrument await-tree for actor-based applications to the underlying services. | | [`BlockingLayer`] | [tokio] | Add blocking API support for non-blocking services. | | [`ChaosLayer`] | [rand] | Inject chaos into underlying services for robustness test. | | [`ConcurrentLimitLayer`] | [tokio] | Add concurrent request limit. | | [`DtraceLayer`] | [probe] | Support User Statically-Defined Tracing(aka USDT) on Linux | | [`LoggingLayer`] | [log] | Add log for every operations. | | [`MetricsLayer`] | [metrics] | Add metrics for every operations. | | [`MimeGuessLayer`] | [mime_guess] | Add `Content-Type` automatically based on the file extension in the operation path. | | [`FastraceLayer`] | [fastrace] | Add fastrace for every operations. | | [`OtelMetricsLayer`] | [opentelemetry::metrics] | Add opentelemetry::metrics for every operations. | | [`OtelTraceLayer`] | [opentelemetry::trace] | Add opentelemetry::trace for every operations. | | [`PrometheusClientLayer`] | [prometheus_client] | Add prometheus metrics for every operations. | | [`PrometheusLayer`] | [prometheus] | Add prometheus metrics for every operations. | | [`RetryLayer`] | [backon] | Add retry for temporary failed operations. | | [`ThrottleLayer`] | [governor] | Add a bandwidth rate limiter to the underlying services. | | [`TimeoutLayer`] | [tokio] | Add timeout for every operations to avoid slow or unexpected hang operations. | | [`TracingLayer`] | [tracing] | Add tracing for every operations. | [`AsyncBacktraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.AsyncBacktraceLayer.html [async-backtrace]: https://github.com/tokio-rs/async-backtrace [`AwaitTreeLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.AwaitTreeLayer.html [await-tree]: https://github.com/risingwavelabs/await-tree [`BlockingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.BlockingLayer.html [tokio]: https://github.com/tokio-rs/tokio [`ChaosLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ChaosLayer.html [rand]: https://github.com/rust-random/rand [`ConcurrentLimitLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ConcurrentLimitLayer.html [`DtraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.DtraceLayer.html [probe]: https://github.com/cuviper/probe-rs [`LoggingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.LoggingLayer.html [log]: https://github.com/rust-lang/log [`MetricsLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.MetricsLayer.html [metrics]: https://github.com/metrics-rs/metrics [`MimeGuessLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.MimeGuessLayer.html [mime_guess]: https://github.com/abonander/mime_guess [`FastraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.FastraceLayer.html [fastrace]: https://github.com/fastracelabs/fastrace [`OtelMetricsLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.OtelMetricsLayer.html [`OtelTraceLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.OtelTraceLayer.html [opentelemetry::trace]: https://docs.rs/opentelemetry/latest/opentelemetry/trace/index.html [`PrometheusClientLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusClientLayer.html [prometheus_client]: https://github.com/prometheus/client_rust [`PrometheusLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.PrometheusLayer.html [prometheus]: https://github.com/tikv/rust-prometheus [`RetryLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html [backon]: https://github.com/Xuanwo/backon [`ThrottleLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.ThrottleLayer.html [governor]: https://github.com/boinkor-net/governor [`TimeoutLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.TimeoutLayer.html [`TracingLayer`]: https://docs.rs/opendal/latest/opendal/layers/struct.TracingLayer.html [tracing]: https://github.com/tokio-rs/tracing ## Quickstart ```rust use opendal::Result; use opendal::layers::LoggingLayer; use opendal::services; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Pick a builder and configure it. let mut builder = services::S3::default(); builder.bucket("test"); // Init an operator let op = Operator::new(builder)? // Init with logging layer enabled. .layer(LoggingLayer::default()) .finish(); // Write data op.write("hello.txt", "Hello, World!").await?; // Read data let bs = op.read("hello.txt").await?; // Fetch metadata let meta = op.stat("hello.txt").await?; let mode = meta.mode(); let length = meta.content_length(); // Delete op.delete("hello.txt").await?; Ok(()) } ``` ## Examples | Name | Description | |---------------------|---------------------------------------------------------------| | [Basic] | Show how to use opendal to operate storage service. | | [Concurrent Upload] | Show how to perform upload concurrently to a storage service. | | [Multipart Upload] | Show how to perform a multipart upload to a storage service. | [Basic]: ./examples/basic [Concurrent Upload]: ./examples/concurrent-upload [Multipart Upload]: ./examples/multipart-upload ## Contributing Check out the [CONTRIBUTING](CONTRIBUTING.md) guide for more details on getting started with contributing to this project. ## Branding The first and most prominent mentions must use the full form: **Apache OpenDAL™** of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization. For more details, see the [Apache Product Name Usage Guide](https://www.apache.org/foundation/marks/guide). ## License and Trademarks Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0 Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. opendal-0.52.0/benches/README.md000064400000000000000000000005521046102023000142220ustar 00000000000000# OpenDAL Benchmarks Running benchmark ```shell OPENDAL_TEST=memory cargo bench ``` Build flamegraph: ```shell cargo flamegraph --bench io -- seek --bench ``` - `--bench io` pick the `io` target. - `-- seek --bench`: chose the benches with `seek` and `--bench` is required by `criterion` After `flamegraph.svg` has been generated, use browser to visit it. opendal-0.52.0/benches/ops/README.md000064400000000000000000000014431046102023000150230ustar 00000000000000# Ops Benchmark Tests Ops Benchmark Tests measure every operation's performance on the target platform. To support benching different backends simultaneously, we use `environment value` to carry the backend config. ## Setup Please copy `.env.example` to `.env` and change the values on need. Take `fs` for example, we need to enable bench on `fs` on `/tmp`. ```dotenv OPENDAL_FS_TEST=false OPENDAL_FS_ROOT=/path/to/dir ``` into ```dotenv OPENDAL_FS_TEST=on OPENDAL_FS_ROOT=/tmp ``` Notice: The default will skip all benches if the env is not set. ## Run Test all available backend. ```shell cargo bench --features tests ``` Test specific backend, take s3 for example, first set the corresponding environment variables of s3, then: ```shell OPENDAL_TEST=s3 cargo bench --features tests ``` opendal-0.52.0/benches/ops/main.rs000064400000000000000000000017231046102023000150370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod read; mod utils; mod write; use criterion::criterion_group; use criterion::criterion_main; criterion_group!(benches, read::bench, write::bench); criterion_main!(benches); opendal-0.52.0/benches/ops/read.rs000064400000000000000000000102561046102023000150270ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use criterion::Criterion; use futures::io; use opendal::raw::tests::init_test_service; use opendal::raw::tests::TEST_RUNTIME; use opendal::Operator; use rand::prelude::*; use size::Size; use super::utils::*; pub fn bench(c: &mut Criterion) { if let Some(op) = init_test_service().unwrap() { bench_read_full(c, op.info().scheme().into_static(), op.clone()); bench_read_parallel(c, op.info().scheme().into_static(), op.clone()); } } fn bench_read_full(c: &mut Criterion, name: &str, op: Operator) { let mut group = c.benchmark_group(format!("service_{name}_read_full")); let mut rng = thread_rng(); for size in [ Size::from_kibibytes(4), Size::from_kibibytes(256), Size::from_mebibytes(4), Size::from_mebibytes(16), ] { let content = gen_bytes(&mut rng, size.bytes() as usize); let path = uuid::Uuid::new_v4().to_string(); let temp_data = TempData::generate(op.clone(), &path, content.clone()); group.throughput(criterion::Throughput::Bytes(size.bytes() as u64)); group.bench_with_input(size.to_string(), &(op.clone(), &path), |b, (op, path)| { b.to_async(&*TEST_RUNTIME).iter(|| async { let r = op.reader_with(path).await.unwrap(); let r = r .into_futures_async_read(0..size.bytes() as u64) .await .unwrap(); io::copy(r, &mut io::sink()).await.unwrap(); }) }); drop(temp_data); } group.finish() } fn bench_read_parallel(c: &mut Criterion, name: &str, op: Operator) { let mut group = c.benchmark_group(format!("service_{name}_read_parallel")); let mut rng = thread_rng(); for size in [ Size::from_kibibytes(4), Size::from_kibibytes(256), Size::from_mebibytes(4), Size::from_mebibytes(16), ] { let content = gen_bytes(&mut rng, (size.bytes() * 2) as usize); let path = uuid::Uuid::new_v4().to_string(); let offset = (size.bytes() / 2) as u64; let buf_size = size.bytes() as usize; let temp_data = TempData::generate(op.clone(), &path, content.clone()); for parallel in [1, 2, 4, 8, 16] { group.throughput(criterion::Throughput::Bytes(parallel * size.bytes() as u64)); group.bench_with_input( format!("{}x{}", parallel, size.to_string()), &(op.clone(), &path, buf_size), |b, (op, path, buf_size)| { b.to_async(&*TEST_RUNTIME).iter(|| async { let futures = (0..parallel) .map(|_| async { let r = op.reader_with(path).await.unwrap(); let _ = r.read(offset..offset + *buf_size as u64).await.unwrap(); let mut d = 0; // mock same little cpu work for c in offset..offset + 100u64 { d += c & (0x1f1f1f1f + c % 256); } let _ = d; }) .collect::>(); futures::future::join_all(futures).await }) }, ); } drop(temp_data); } group.finish() } opendal-0.52.0/benches/ops/utils.rs000064400000000000000000000032561046102023000152560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Bytes; use opendal::raw::tests::TEST_RUNTIME; use opendal::*; use rand::prelude::*; pub fn gen_bytes(rng: &mut ThreadRng, size: usize) -> Bytes { let mut content = vec![0; size]; rng.fill_bytes(&mut content); content.into() } pub struct TempData { op: Operator, path: String, } impl TempData { pub fn existing(op: Operator, path: &str) -> Self { Self { op, path: path.to_string(), } } pub fn generate(op: Operator, path: &str, content: Bytes) -> Self { TEST_RUNTIME.block_on(async { op.write(path, content).await.expect("create test data") }); Self { op, path: path.to_string(), } } } impl Drop for TempData { fn drop(&mut self) { TEST_RUNTIME.block_on(async { self.op.delete(&self.path).await.expect("cleanup test data"); }) } } opendal-0.52.0/benches/ops/write.rs000064400000000000000000000062251046102023000152470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use criterion::Criterion; use opendal::raw::tests::init_test_service; use opendal::raw::tests::TEST_RUNTIME; use opendal::Operator; use rand::prelude::*; use size::Size; use super::utils::*; pub fn bench(c: &mut Criterion) { if let Some(op) = init_test_service().unwrap() { bench_write_once(c, op.info().scheme().into_static(), op.clone()); bench_write_with_concurrent(c, op.info().scheme().into_static(), op.clone()); } } fn bench_write_once(c: &mut Criterion, name: &str, op: Operator) { let mut group = c.benchmark_group(format!("service_{name}_write_once")); let mut rng = thread_rng(); for size in [ Size::from_kibibytes(4), Size::from_kibibytes(256), Size::from_mebibytes(4), Size::from_mebibytes(16), ] { let content = gen_bytes(&mut rng, size.bytes() as usize); let path = uuid::Uuid::new_v4().to_string(); let temp_data = TempData::existing(op.clone(), &path); group.throughput(criterion::Throughput::Bytes(size.bytes() as u64)); group.bench_with_input( size.to_string(), &(op.clone(), &path, content.clone()), |b, (op, path, content)| { b.to_async(&*TEST_RUNTIME).iter(|| async { op.write(path, content.clone()).await.unwrap(); }) }, ); std::mem::drop(temp_data); } group.finish() } fn bench_write_with_concurrent(c: &mut Criterion, name: &str, op: Operator) { let mut group = c.benchmark_group(format!("service_{name}_write_with_concurrent")); let mut rng = thread_rng(); for concurrent in [1, 2, 4, 8] { let content = gen_bytes(&mut rng, 5 * 1024 * 1024); let path = uuid::Uuid::new_v4().to_string(); group.throughput(criterion::Throughput::Bytes(16 * 5 * 1024 * 1024)); group.bench_with_input( concurrent.to_string(), &(op.clone(), &path, content.clone()), |b, (op, path, content)| { b.to_async(&*TEST_RUNTIME).iter(|| async { let mut w = op.writer_with(path).concurrent(concurrent).await.unwrap(); for _ in 0..16 { w.write(content.clone()).await.unwrap(); } w.close().await.unwrap(); }) }, ); } group.finish() } opendal-0.52.0/benches/types/README.md000064400000000000000000000112201046102023000153600ustar 00000000000000# Types Benchmark Tests This benchmark contains performance testing of critical data structures in opendal types, currently including performance testing of Buffer. ## Run ```shell cargo bench --bench types --features tests ``` The following are the test results for reference: ```shell bench_non_contiguous_buffer/bytes buf 256 KiB * 4 chunk time: [226.31 ps 227.81 ps 229.33 ps] bench_non_contiguous_buffer/buffer 256 KiB * 4 chunk time: [319.85 ps 325.66 ps 332.61 ps] bench_non_contiguous_buffer/bytes buf 256 KiB * 4 advance time: [10.241 ns 10.242 ns 10.243 ns] bench_non_contiguous_buffer/buffer 256 KiB * 4 advance time: [9.2208 ns 9.2216 ns 9.2226 ns] bench_non_contiguous_buffer/bytes buf 256 KiB * 4 truncate time: [10.248 ns 10.253 ns 10.258 ns] bench_non_contiguous_buffer/buffer 256 KiB * 4 truncate time: [8.8055 ns 8.8068 ns 8.8085 ns] bench_non_contiguous_buffer/bytes buf 256 KiB * 32 chunk time: [228.76 ps 230.04 ps 231.29 ps] bench_non_contiguous_buffer/buffer 256 KiB * 32 chunk time: [319.49 ps 323.25 ps 327.83 ps] bench_non_contiguous_buffer/bytes buf 256 KiB * 32 advance time: [10.244 ns 10.245 ns 10.246 ns] bench_non_contiguous_buffer/buffer 256 KiB * 32 advance time: [9.2172 ns 9.2184 ns 9.2197 ns] bench_non_contiguous_buffer/bytes buf 256 KiB * 32 truncate time: [10.242 ns 10.243 ns 10.246 ns] bench_non_contiguous_buffer/buffer 256 KiB * 32 truncate time: [8.8071 ns 8.8089 ns 8.8118 ns] bench_non_contiguous_buffer/bytes buf 4.00 MiB * 4 chunk time: [222.45 ps 223.42 ps 224.47 ps] bench_non_contiguous_buffer/buffer 4.00 MiB * 4 chunk time: [364.31 ps 373.49 ps 382.43 ps] bench_non_contiguous_buffer/bytes buf 4.00 MiB * 4 advance time: [10.243 ns 10.244 ns 10.245 ns] bench_non_contiguous_buffer/buffer 4.00 MiB * 4 advance time: [9.2196 ns 9.2204 ns 9.2213 ns] bench_non_contiguous_buffer/bytes buf 4.00 MiB * 4 truncate time: [10.244 ns 10.245 ns 10.246 ns] bench_non_contiguous_buffer/buffer 4.00 MiB * 4 truncate time: [8.8083 ns 8.8094 ns 8.8105 ns] bench_non_contiguous_buffer/bytes buf 4.00 MiB * 32 chunk time: [229.61 ps 230.68 ps 231.71 ps] bench_non_contiguous_buffer/buffer 4.00 MiB * 32 chunk time: [364.61 ps 369.01 ps 373.13 ps] bench_non_contiguous_buffer/bytes buf 4.00 MiB * 32 advance time: [10.239 ns 10.240 ns 10.242 ns] bench_non_contiguous_buffer/buffer 4.00 MiB * 32 advance time: [9.2188 ns 9.2238 ns 9.2336 ns] bench_non_contiguous_buffer/bytes buf 4.00 MiB * 32 truncate time: [10.249 ns 10.265 ns 10.291 ns] bench_non_contiguous_buffer/buffer 4.00 MiB * 32 truncate time: [8.8080 ns 8.8092 ns 8.8106 ns] bench_non_contiguous_buffer_with_extreme/256 KiB * 1k chunk time: [352.06 ps 361.52 ps 371.51 ps] bench_non_contiguous_buffer_with_extreme/256 KiB * 1k advance time: [378.80 ns 378.97 ns 379.18 ns] bench_non_contiguous_buffer_with_extreme/256 KiB * 1k truncate time: [8.8039 ns 8.8049 ns 8.8061 ns] bench_non_contiguous_buffer_with_extreme/256 KiB * 10k chunk time: [318.90 ps 320.87 ps 322.97 ps] bench_non_contiguous_buffer_with_extreme/256 KiB * 10k advance time: [3.6598 µs 3.6613 µs 3.6634 µs] bench_non_contiguous_buffer_with_extreme/256 KiB * 10k truncate time: [8.8065 ns 8.8074 ns 8.8083 ns] bench_non_contiguous_buffer_with_extreme/256 KiB * 100k chunk time: [319.32 ps 326.56 ps 334.76 ps] bench_non_contiguous_buffer_with_extreme/256 KiB * 100k advance time: [40.561 µs 40.623 µs 40.690 µs] bench_non_contiguous_buffer_with_extreme/256 KiB * 100k truncate time: [8.8071 ns 8.8081 ns 8.8092 ns] bench_non_contiguous_buffer_with_extreme/256 KiB * 1000k chunk time: [322.26 ps 329.02 ps 336.15 ps] bench_non_contiguous_buffer_with_extreme/256 KiB * 1000k advance time: [848.22 µs 848.97 µs 849.64 µs] bench_non_contiguous_buffer_with_extreme/256 KiB * 1000k truncate time: [8.8061 ns 8.8073 ns 8.8086 ns] ``` opendal-0.52.0/benches/types/buffer.rs000064400000000000000000000107641046102023000157340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use criterion::Criterion; use opendal::Buffer; use rand::thread_rng; use size::Size; use super::utils::*; pub fn bench_non_contiguous_buffer(c: &mut Criterion) { let mut group = c.benchmark_group("bench_non_contiguous_buffer"); let mut rng = thread_rng(); for size in [Size::from_kibibytes(256), Size::from_mebibytes(4)] { for num in [4, 32] { let bytes_buf = gen_bytes(&mut rng, size.bytes() as usize * num); let mut bytes_vec = vec![]; for _ in 0..num { bytes_vec.push(gen_bytes(&mut rng, size.bytes() as usize)); } let buffer = Buffer::from(bytes_vec); let bytes_buf_name = format!("bytes buf {} * {} ", size.to_string(), num); let buffer_name = format!("buffer {} * {}", size.to_string(), num); group.bench_function(format!("{} {}", bytes_buf_name, "chunk"), |b| { b.iter(|| bytes_buf.chunk()) }); group.bench_function(format!("{} {}", buffer_name, "chunk"), |b| { b.iter(|| buffer.chunk()) }); group.bench_function(format!("{} {}", bytes_buf_name, "advance"), |b| { b.iter(|| { let mut bytes_buf = bytes_buf.clone(); // Advance non-integer number of Bytes. bytes_buf.advance((size.bytes() as f64 * 3.5) as usize); }) }); group.bench_function(format!("{} {}", buffer_name, "advance"), |b| { b.iter(|| { let mut buffer = buffer.clone(); // Advance non-integer number of Bytes. buffer.advance((size.bytes() as f64 * 3.5) as usize); }) }); group.bench_function(format!("{} {}", bytes_buf_name, "truncate"), |b| { b.iter(|| { let mut bytes_buf = bytes_buf.clone(); // Truncate non-integer number of Bytes. bytes_buf.truncate((size.bytes() as f64 * 3.5) as usize); }) }); group.bench_function(format!("{} {}", buffer_name, "truncate"), |b| { b.iter(|| { let mut buffer = buffer.clone(); // Truncate non-integer number of Bytes. buffer.truncate((size.bytes() as f64 * 3.5) as usize); }) }); } } group.finish() } pub fn bench_non_contiguous_buffer_with_extreme(c: &mut Criterion) { let mut group: criterion::BenchmarkGroup = c.benchmark_group("bench_non_contiguous_buffer_with_extreme"); let mut rng = thread_rng(); let size = Size::from_kibibytes(256); let bytes = gen_bytes(&mut rng, size.bytes() as usize); for num in [1000, 10000, 100000, 1000000] { let repeated_bytes = RepeatedBytes { bytes: bytes.clone(), index: 0, count: num, }; let buffer = Buffer::from_iter(repeated_bytes); let buffer_name = format!("{} * {}k", size.to_string(), num / 1000); group.bench_function(format!("{} {}", buffer_name, "chunk"), |b| { b.iter(|| buffer.chunk()) }); group.bench_function(format!("{} {}", buffer_name, "advance"), |b| { b.iter(|| { let mut buffer = buffer.clone(); buffer.advance(buffer.len()); }) }); group.bench_function(format!("{} {}", buffer_name, "truncate"), |b| { b.iter(|| { let mut buffer = buffer.clone(); buffer.truncate(buffer.len()); }) }); } group.finish() } opendal-0.52.0/benches/types/concurrent_tasks.rs000064400000000000000000000044031046102023000200430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::time::Duration; use criterion::BatchSize; use criterion::Criterion; use once_cell::sync::Lazy; use opendal::raw::ConcurrentTasks; use opendal::Executor; pub static TOKIO: Lazy = Lazy::new(|| tokio::runtime::Runtime::new().expect("build tokio runtime")); pub fn bench_concurrent_tasks(c: &mut Criterion) { let mut group = c.benchmark_group("bench_concurrent_tasks"); for concurrent in [1, 2, 4, 8, 16] { group.bench_with_input( format!("concurrent {}", concurrent), &concurrent, |b, concurrent| { b.to_async(&*TOKIO).iter_batched( || { ConcurrentTasks::new(Executor::new(), *concurrent, |()| { Box::pin(async { tokio::time::sleep(Duration::from_millis(1)).await; ((), Ok(())) }) }) }, |mut tasks| async move { for _ in 0..100 { let _ = tasks.execute(()).await; } loop { if tasks.next().await.is_none() { break; } } }, BatchSize::PerIteration, ) }, ); } group.finish() } opendal-0.52.0/benches/types/main.rs000064400000000000000000000021311046102023000153740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod buffer; mod concurrent_tasks; mod utils; use criterion::criterion_group; use criterion::criterion_main; criterion_group!( benches, buffer::bench_non_contiguous_buffer, buffer::bench_non_contiguous_buffer_with_extreme, concurrent_tasks::bench_concurrent_tasks, ); criterion_main!(benches); opendal-0.52.0/benches/types/utils.rs000064400000000000000000000025461046102023000156220ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Bytes; use rand::prelude::ThreadRng; use rand::RngCore; pub struct RepeatedBytes { pub bytes: Bytes, pub index: usize, pub count: usize, } impl Iterator for RepeatedBytes { type Item = Bytes; fn next(&mut self) -> Option { if self.index < self.count { self.index += 1; Some(self.bytes.clone()) } else { None } } } pub fn gen_bytes(rng: &mut ThreadRng, size: usize) -> Bytes { let mut content = vec![0; size]; rng.fill_bytes(&mut content); content.into() } opendal-0.52.0/edge/README.md000064400000000000000000000002531046102023000135150ustar 00000000000000# OpenDAL Edge Tests OpenDAL edge tests served as edge tests for the OpenDAL project. They will have pre-set data and will test the functionality of the OpenDAL project. opendal-0.52.0/examples/README.md000064400000000000000000000007661046102023000144400ustar 00000000000000# Apache OpenDAL™ Rust Core Examples Thank you for using OpenDAL! Those examples are designed to help you to understand how to use OpenDAL Rust Core. ## Setup All examples following the same setup steps: To run this example, please copy the `.env.example`, which is at project root, to `.env` and change the values on need. Take `fs` for example, we need to change to enable behavior test on `fs` on `/tmp`. ```dotenv OPENDAL_FS_ROOT=/path/to/dir ``` into ```dotenv OPENDAL_FS_ROOT=/tmp ``` opendal-0.52.0/src/docs/comparisons/mod.rs000064400000000000000000000024121046102023000165320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Compare opendal with other projects to find out the differences and areas that opendal can improve. //! //! All documents listed should be treated as highly biased. Because: //! //! - OpenDAL's maintainers and contributors write them. //! - Writers may not be familiar with the compared projects (at least not as familiar with OpenDAL) //! //! Let's see OpenDAL: //! //! - [vs `object_store`][`vs_object_store`] #[doc = include_str!("vs_object_store.md")] pub mod vs_object_store {} opendal-0.52.0/src/docs/comparisons/vs_object_store.md000064400000000000000000000167721046102023000211370ustar 00000000000000# OpenDAL vs object_store > NOTE: This document is written by OpenDAL's maintainers and not reviewed by > object_store's maintainers. So it could not be very objective. ## About object_store [object_store](https://crates.io/crates/object_store) is > A focused, easy to use, idiomatic, high performance, `async` object store library interacting with object stores. It was initially developed for [InfluxDB IOx](https://github.com/influxdata/influxdb_iox/) and later split out and donated to [Apache Arrow](https://arrow.apache.org/). ## Similarities ### Language Yes, of course. Both `opendal` and `object_store` are developed in [Rust](https://www.rust-lang.org/), a language empowering everyone to build reliable and efficient software. ### License Both `opendal` and `object_store` are licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0). ### Owner `object_store` is a part of `Apache Arrow` which means it's hosted and maintained by the [Apache Software Foundation](https://www.apache.org/). `opendal` is now hosted by the [Apache Software Foundation](https://www.apache.org/) also. ### Domain Both `opendal` and `object_store` can be used to access data stored on object storage services. The primary users of those projects are both cloud-native databases too: - `opendal` is mainly used by: - [databend](https://github.com/datafuselabs/databend): A modern Elasticity and Performance cloud data warehouse - [GreptimeDB](https://github.com/GreptimeTeam/greptimedb): An open-source, cloud-native, distributed time-series database. - [mozilla/sccache](https://github.com/mozilla/sccache/): sccache is ccache with cloud storage - [risingwave](https://github.com/risingwavelabs/risingwave): A Distributed SQL Database for Stream Processing - [Vector](https://github.com/vectordotdev/vector): A high-performance observability data pipeline. - `object_store` is mainly used by: - [datafusion](https://github.com/apache/arrow-datafusion): Apache Arrow DataFusion SQL Query Engine - [Influxdb IOx](https://github.com/influxdata/influxdb_iox/): The new core of InfluxDB is written in Rust on top of Apache Arrow. ## Differences ### Vision `opendal` is Open Data Access Layer that accesses data freely, painlessly, and efficiently. `object_store` is more focused on async object store support. You will see the different visions lead to very different routes. ### Design `object_store` exposed a trait called [`ObjectStore`](https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html) to users. Users need to build a `dyn ObjectStore` and operate on it directly: ```rust let object_store: Arc = Arc::new(get_object_store()); let path: Path = "data/file01.parquet".try_into()?; let stream = object_store .get(&path) .await? .into_stream(); ``` `opendal` has a similar trait called [`Access`][crate::raw::Access] But `opendal` don't expose this trait to end users directly. Instead, `opendal` expose a new struct called [`Operator`][crate::Operator] and builds public API on it. ```rust let op: Operator = Operator::from_env(Scheme::S3)?; let r = op.reader("data/file01.parquet").await?; ``` ### Interception Both `object_store` and `opendal` provide a mechanism to intercept operations. `object_store` called `Adapters`: ```rust let object_store = ThrottledStore::new(get_object_store(), ThrottleConfig::default()) ``` `opendal` called [`Layer`](crate::raw::Layer): ```rust let op = op.layer(TracingLayer).layer(MetricsLayer); ``` At the time of writing: object_store (`v0.5.0`) supports: - ThrottleStore: Rate Throttling - LimitStore: Concurrent Request Limit opendal supports: - ImmutableIndexLayer: immutable in-memory index. - LoggingLayer: logging. - MetadataCacheLayer: metadata cache. - ContentCacheLayer: content data cache. - MetricsLayer: metrics - RetryLayer: retry - SubdirLayer: Allow switch directory without changing original operator. - TracingLayer: tracing ### Services `opendal` and `object_store` have different visions, so they have other services support: | service | opendal | object_store | |---------|-----------------|-----------------------------------------| | azblob | Y | Y | | fs | Y | Y | | ftp | Y | N | | gcs | Y | Y | | hdfs | Y | Y *(via [datafusion-objectstore-hdfs])* | | http | Y *(read only)* | N | | ipfs | Y *(read only)* | N | | ipmfs | Y | N | | memory | Y | Y | | obs | Y | N | | s3 | Y | Y | opendal has an idea called [`Capability`][crate::Capability], so it's services may have different capability sets. For example, opendal's `http` and `ipfs` are read only. ### Features `opendal` and `object_store` have different visions, so they have different feature sets: | opendal | object_store | notes | |-----------|----------------------|----------------------------------------------| | metadata | - | get some metadata from underlying storage | | create | put | - | | read | get | - | | read | get_range | - | | - | get_ranges | opendal doesn't support read multiple ranges | | write | put | - | | stat | head | - | | delete | delete | - | | - | list | opendal doesn't support list with prefix | | list | list_with_delimiter | - | | - | copy | - | | - | copy_if_not_exists | - | | - | rename | - | | - | rename_if_not_exists | - | | presign | - | get a presign URL of object | | multipart | multipart | both support, but API is different | | blocking | - | opendal supports blocking API | ## Demo show The most straightforward complete demo how to read a file from s3: `opendal` ```rust let mut builder = S3::default(); builder.bucket("example"); builder.access_key_id("access_key_id"); builder.secret_access_key("secret_access_key"); let store = Operator::new(builder)?.finish(); let r = store.reader("data.parquet").await?; ``` `object_store` ```rust let mut builder = AmazonS3Builder::new() .with_bucket_name("example") .with_access_key_id("access_key_id") .with_secret_access_key("secret_access_key"); let store = Arc::new(builder.build()?); let path: Path = "data.parquet".try_into().unwrap(); let stream = store.get(&path).await()?.into_stream(); ``` [datafusion-objectstore-hdfs]: https://github.com/datafusion-contrib/datafusion-objectstore-hdfs/ opendal-0.52.0/src/docs/concepts.rs000064400000000000000000000137721046102023000152470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! The core concepts of OpenDAL's public API. //! //! OpenDAL provides a unified abstraction that helps developers access all storage services. //! //! There are two core concepts in OpenDAL: //! //! - [`Builder`]: Builder accepts a series of parameters to set up an instance of underlying services. //! You can adjust the behaviour of underlying services with these parameters. //! - [`Operator`]: Developer can access underlying storage services with manipulating one Operator. //! The Operator is a delegate for underlying implementation detail, and provides one unified access interface, //! including `read`, `write`, `list` and so on. //! //! If you are interested in internal implementation details, please have a look at [`internals`][super::internals]. //! //! # Builder //! //! Let's start with [`Builder`]. //! //! A `Builder` is a trait that is implemented by the underlying services. We can use a `Builder` to configure and create a service. //! Developer can only create one service via Builder, in other words, Builder is the only public API provided by services. //! And other detailed implementation will be hidden. //! //! ```text //! ┌───────────┐ ┌───────────┐ //! │ │ build() │ │ //! │ Builder ├────────────────►│ Service │ //! │ │ │ │ //! └───────────┘ └───────────┘ //! ``` //! //! All [`Builder`] provided by OpenDAL is under [`services`][crate::services], we can refer to them like `opendal::services::S3`. //! By right the builder will be named like `OneServiceBuilder`, but usually we will export it to public with renaming it as one //! general name. For example, we will rename `S3Builder` to `S3` and developer will use `S3` finally. //! //! For example: //! //! ```no_run //! use opendal::services::S3; //! //! let mut builder = S3::default(); //! builder.bucket("example"); //! builder.root("/path/to/file"); //! ``` //! //! # Operator //! The [`Operator`] is a delegate for Service, the underlying implementation detail that implements [`Access`][crate::raw::Access], //! and it also provides one unified access interface. //! It will hold one reference of Service with its all generic types erased by OpenDAL, //! which is the reason why we say the Operator is the delegate of one Service. //! //! ```text //! ┌────────────────────┐ //! │ Operator │ //! │ │delegate │ //! ┌─────────┐ build │ ▼ │ rely on ┌─────────────────────┐ //! │ Builder ├───────┼──►┌────────────┐ │◄────────┤ business logic code │ //! └─────────┘ │ │ Service │ │ └─────────────────────┘ //! └───┴────────────┴───┘ //! ``` //! //! `Operator` can be built from `Builder`: //! //! ```no_run //! # use opendal::Result; //! use opendal::services::S3; //! use opendal::Operator; //! //! # fn test() -> Result<()> { //! let mut builder = S3::default(); //! builder.bucket("example"); //! builder.root("/path/to/file"); //! //! let op = Operator::new(builder)?.finish(); //! # Ok(()) //! # } //! ``` //! //! - `Operator` has it's internal `Arc`, so it's **cheap** to clone it. //! - `Operator` doesn't have generic parameters or lifetimes, so it's **easy** to use it everywhere. //! - `Operator` implements `Send` and `Sync`, so it's **safe** to send it between threads. //! //! After get an `Operator`, we can do operations on different paths. //! //! //! ```text //! ┌──────────────┐ //! ┌────────►│ read("abc") │ //! │ └──────────────┘ //! ┌───────────┐ │ //! │ Operator │ │ ┌──────────────┐ //! │ ┌───────┐ ├────┼────────►│ write("def") │ //! │ │Service│ │ │ └──────────────┘ //! └─┴───────┴─┘ │ //! │ ┌──────────────┐ //! └────────►│ list("ghi/") │ //! └──────────────┘ //! ``` //! //! We can read data with given path in this way: //! //! ```no_run //! # use opendal::Result; //! use opendal::services::S3; //! use opendal::Operator; //! //! # async fn test() -> Result<()> { //! let mut builder = S3::default(); //! builder.bucket("example"); //! builder.root("/path/to/file"); //! //! let op = Operator::new(builder)?.finish(); //! let bs: Vec = op.read("abc").await?; //! # Ok(()) //! # } //! ``` //! //! [`Builder`]: crate::Builder //! [`Operator`]: crate::Operator opendal-0.52.0/src/docs/internals/accessor.rs000064400000000000000000000253361046102023000172310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! The internal implementation details of [`Access`]. //! //! [`Access`] is the core trait of OpenDAL's raw API. We operate //! underlying storage services via APIs provided by [`Access`]. //! //! # Introduction //! //! [`Access`] can be split in the following parts: //! //! ```ignore //! // Attributes //! #[async_trait] //! // <----------Trait Bound--------------> //! pub trait Accessor: Send + Sync + Debug + Unpin + 'static { //! type Reader: oio::Read; // --+ //! type BlockingReader: oio::BlockingRead; // +--> Associated Type //! type Lister: oio::Lister; // + //! type BlockingLister: oio::BlockingLister; // --+ //! //! // APIs //! async fn hello(&self, path: &str, args: OpCreate) -> Result; //! async fn world(&self, path: &str, args: OpCreate) -> Result; //! } //! ``` //! //! Let's go deep into [`Access`] line by line. //! //! ## Async Trait //! //! At the first line of [`Access`], we will read: //! //! ```ignore //! #[async_trait] //! ``` //! //! This is an attribute from [`async_trait`](https://docs.rs/async-trait/latest/async_trait/). By using this attribute, we can write the following code without use nightly feature. //! //! ```ignore //! pub trait Accessor { //! async fn create_dir(&self, path: &str) -> Result<()>; //! } //! ``` //! //! `async_trait` will transform the `async fn` into: //! //! ```ignore //! pub trait Accessor { //! fn create_dir<'async>( //! &'async self, //! ) -> Pin + MaybeSend + 'async>> //! where Self: Sync + 'async; //! } //! ``` //! //! It's not zero cost, and we will improve this part once the related features are stabilised. //! //! ## Trait Bound //! //! Then we will read the declare of [`Access`] trait: //! //! ```ignore //! pub trait Accessor: Send + Sync + Debug + Unpin + 'static {} //! ``` //! //! There are many trait boundings here. For now, [`Access`] requires the following bound: //! //! - [`Send`]: Allow user to send between threads without extra wrapper. //! - [`Sync`]: Allow user to sync between threads without extra lock. //! - [`Debug`][std::fmt::Debug]: Allow users to print underlying debug information of accessor. //! - [`Unpin`]: Make sure `Accessor` can be safely moved after being pinned, so users don't need to `Pin>`. //! - `'static`: Make sure `Accessor` is not a short-time reference, allow users to use `Accessor` in closures and futures without playing with lifetime. //! //! Implementer of `Accessor` should take care of the following things: //! //! - Implement `Debug` for backend, but don't leak credentials. //! - Make sure the backend is `Send` and `Sync`, wrap the internal struct with `Arc>` if necessary. //! //! ## Associated Type //! //! The first block of [`Access`] trait is our associated types. We //! require implementers to specify the type to be returned, thus avoiding //! the additional overhead of dynamic dispatch. //! //! [`Access`] has four associated type so far: //! //! - `Reader`: reader returned by `read` operation. //! - `BlockingReader`: reader returned by `blocking_read` operation. //! - `Lister`: lister returned by `list` operation. //! - `BlockingLister`: lister returned by `blocking_scan` or `blocking_list` operation. //! //! Implementer of `Accessor` should take care the following things: //! //! - OpenDAL will erase those type at the final stage of Operator building. Please don't return dynamic trait object like `oio::Reader`. //! - Use `()` as type if the operation is not supported. //! //! ## API Style //! //! Every API of [`Access`] follows the same style: //! //! - All APIs have a unique [`Operation`] and [`Capability`] //! - All APIs are orthogonal and do not overlap with each other //! - Most APIs accept `path` and `OpXxx`, and returns `RpXxx`. //! - Most APIs have `async` and `blocking` variants, they share the same semantics but may have different underlying implementations. //! //! [`Access`] can declare their capabilities via [`AccessorInfo`]'s `set_capability`: //! //! ```ignore //! impl Access for MyBackend { //! fn metadata(&self) -> AccessorInfo { //! let mut am = AccessorInfo::default(); //! am.set_capability( //! Capability { //! read: true, //! write: true, //! ..Default::default() //! }); //! //! am.into() //! } //! } //! ``` //! //! Now that you have mastered [`Access`], let's go and implement our own backend! //! //! # Tutorial //! //! This tutorial implements a `duck` storage service that sends API //! requests to a super-powered duck. Gagaga! //! //! ## Scheme //! //! First of all, let's pick a good [`Scheme`] for our duck service. The //! scheme should be unique and easy to understand. Normally we should //! use its formal name. //! //! For example, we will use `s3` for AWS S3 Compatible Storage Service //! instead of `aws` or `awss3`. This is because there are many storage //! vendors that provide s3-like RESTful APIs, and our s3 service is //! implemented to support all of them, not just AWS S3. //! //! Obviously, we can use `duck` as scheme, let's add a new variant in [`Scheme`], and implement all required functions like `Scheme::from_str` and `Scheme::into_static`: //! //! ```ignore //! pub enum Scheme { //! Duck, //! } //! ``` //! //! ## Builder //! //! Then we can implement a builder for the duck service. The [`Builder`] //! will provide APIs for users to configure, and they will create an //! instance of a particular service. //! //! Let's create a `backend` mod under `services/duck` directory, and adding the following code. //! //! ```ignore //! use crate::raw::*; //! use crate::*; //! //! /// Duck Storage Service support. Gagaga! //! /// //! /// # Capabilities //! /// //! /// This service can be used to: //! /// //! /// - [x] read //! /// - [ ] write //! /// - [ ] list //! /// - [ ] presign //! /// - [ ] blocking //! /// //! /// # Configuration //! /// //! /// - `root`: Set the work dir for backend. //! /// //! /// ## Via Builder //! /// //! /// ```no_run //! /// use std::sync::Arc; //! /// //! /// use anyhow::Result; //! /// use opendal::services::Duck; //! /// use opendal::Operator; //! /// //! /// #[tokio::main] //! /// async fn main() -> Result<()> { //! /// // Create Duck backend builder. //! /// let mut builder = Duck::default(); //! /// // Set the root for duck, all operations will happen under this root. //! /// // //! /// // NOTE: the root must be absolute path. //! /// builder.root("/path/to/dir"); //! /// //! /// let op: Operator = Operator::new(builder)?.finish(); //! /// //! /// Ok(()) //! /// } //! /// ``` //! #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] //! #[serde(default)] //! #[non_exhaustive] //! pub struct DuckConfig { //! pub root: Option, //! } //! //! #[derive(Default, Clone)] //! pub struct DuckBuilder { //! config: DuckConfig, //! } //! ``` //! //! Note that `DuckBuilder` is part of our public API, so it needs to be //! documented. And any changes you make will directly affect users, so //! please take it seriously. Otherwise, you will be hunted down by many //! angry ducks. //! //! Then, we can implement required APIs for `DuckBuilder`: //! //! ```ignore //! impl DuckBuilder { //! /// Set root of this backend. //! /// //! /// All operations will happen under this root. //! pub fn root(&mut self, root: &str) -> &mut Self { //! self.config.root = if root.is_empty() { //! None //! } else { //! Some(root.to_string()) //! }; //! //! self //! } //! } //! //! impl Builder for DuckBuilder { //! const SCHEME: Scheme = Scheme::Duck; //! type Accessor = DuckBackend; //! type Config = DuckConfig; //! //! fn from_config(config: Self::Config) -> Self { //! DuckBuilder { config: self } //! } //! //! fn build(self) -> Result { //! debug!("backend build started: {:?}", &self); //! //! let root = normalize_root(&self.config.root.clone().unwrap_or_default()); //! debug!("backend use root {}", &root); //! //! Ok(DuckBackend { root }) //! } //! } //! ``` //! //! `DuckBuilder` is ready now, let's try to play with real ducks! //! //! ## Backend //! //! I'm sure you can see it already: `DuckBuilder` will build a //! `DuckBackend` that implements [`Access`]. The backend is what we used //! to communicate with the super-powered ducks! //! //! Let's keep adding more code under `backend.rs`: //! //! ```ignore //! /// Duck storage service backend //! #[derive(Clone, Debug)] //! pub struct DuckBackend { //! root: String, //! } //! //! #[async_trait] //! impl Access for DuckBackend { //! type Reader = DuckReader; //! type BlockingReader = (); //! type Writer = (); //! type BlockingWriter = (); //! type Lister = (); //! type BlockingLister = (); //! //! fn metadata(&self) -> AccessorInfo { //! let mut am = AccessorInfo::default(); //! am.set_scheme(Scheme::Duck) //! .set_root(&self.root) //! .set_capability( //! Capability { //! read: true, //! ..Default::default() //! }); //! //! am.into() //! } //! //! async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { //! gagaga!() //! } //! } //! ``` //! //! Congratulations, we have implemented an [`Access`] that can talk to //! Super Power Ducks! //! //! What!? There are no Super Power Ducks? So sad, but never mind, we have //! really powerful storage services [here](https://github.com/apache/opendal/issues/5). Welcome to pick one to implement. I promise you won't //! have to `gagaga!()` this time. //! //! [`Access`]: crate::raw::Access //! [`Operation`]: crate::raw::Operation //! [`Capability`]: crate::Capability //! [`AccessorInfo`]: crate::raw::AccessorInfo //! [`Scheme`]: crate::Scheme //! [`Builder`]: crate::Builder opendal-0.52.0/src/docs/internals/layer.rs000064400000000000000000000033111046102023000165300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! The internal implementation details of [`Layer`]. //! //! [`Layer`] itself is quite simple: //! //! ```ignore //! pub trait Layer { //! type LayeredAccess: Accessor; //! //! fn layer(&self, inner: A) -> Self::LayeredAccess; //! } //! ``` //! //! `XxxLayer` will wrap input [`Access`] as inner and return a new [`Access`]. So normally the implementation of [`Layer`] will be split into two parts: //! //! - `XxxLayer` will implement [`Layer`] and return `XxxAccessor` as `Self::LayeredAccess`. //! - `XxxAccessor` will implement [`Access`] and be built by `XxxLayer`. //! //! Most layer only implements part of [`Access`], so we provide //! [`LayeredAccess`] which will forward all unimplemented methods to //! `inner`. It's highly recommend to implement [`LayeredAccess`] trait //! instead. //! //! [`Layer`]: crate::raw::Layer //! [`Access`]: crate::raw::Access //! [`LayeredAccess`]: crate::raw::LayeredAccess opendal-0.52.0/src/docs/internals/mod.rs000064400000000000000000000065051046102023000162030ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! The internal implement details of OpenDAL. //! //! OpenDAL has provides unified abstraction via two-level API sets: //! //! - Public API like [`Operator`] provides user level API. //! - Raw API like [`Access`], [`Layer`] provides developer level API. //! //! OpenDAL tries it's best to keep the public API stable. But raw APIs //! may change between minor releases from time to time. So most users //! should only use the public API. And only developers need to implement //! with raw API to implement a new service [`Access`] or their own //! [`Layer`]. //! //! In this section, we will talk about the following components: //! //! - [`Access`][accessor]: to connect underlying storage services. //! - [`Layer`][layer]: middleware/interceptor between storage services. //! //! The relation between [`Access`], [`Layer`] and [`Operator`] looks like the following: //! //! ```text //! ┌─────────────────────────────────────────────────┬──────────┐ //! │ │ │ //! │ ┌──────────┐ ┌────────┐ │ │ //! │ │ │ │ ▼ │ │ //! │ s3──┐ │ │ │ Tracing Layer │ │ //! │ │ │ │ │ │ │ │ //! │ gcs──┤ │ │ │ ▼ │ │ //! │ ├──►│ Accessor ├──┘ Metrics Layer ┌───►│ Operator │ //! │ azblob──┤ │ │ │ │ │ │ //! │ │ │ │ ▼ │ │ │ //! │ hdfs──┘ │ │ Logging Layer │ │ │ //! │ │ │ │ │ │ │ //! │ └──────────┘ └──────┘ │ │ //! │ │ │ //! └─────────────────────────────────────────────────┴──────────┘ //! ``` //! //! [`Builder`]: crate::Builder //! [`Operator`]: crate::Operator //! [`Access`]: crate::raw::Access //! [`Layer`]: crate::raw::Layer pub mod accessor; pub mod layer; opendal-0.52.0/src/docs/mod.rs000064400000000000000000000025011046102023000141740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! This module holds documentation for OpenDAL. //! //! It's highly recommended that you start by reading [`concepts`] first. #![allow(rustdoc::bare_urls)] pub mod comparisons; pub mod concepts; pub mod internals; /// Changes log for all OpenDAL released versions. #[doc = include_str!("../../CHANGELOG.md")] #[cfg(not(doctest))] pub mod changelog {} #[cfg(not(doctest))] pub mod rfcs; /// Upgrade and migrate procedures while OpenDAL meets breaking changes. #[doc = include_str!("upgrade.md")] #[cfg(not(doctest))] pub mod upgrade {} opendal-0.52.0/src/docs/rfcs/0000_example.md000064400000000000000000000065041046102023000164270ustar 00000000000000- Proposal Name: (fill me in with a unique ident, `my_awesome_feature`) - Start Date: (fill me in with today's date, YYYY-MM-DD) - RFC PR: [apache/opendal#0000](https://github.com/apache/opendal/pull/0000) - Tracking Issue: [apache/opendal#0000](https://github.com/apache/opendal/issues/0000) # Summary One paragraph explanation of the proposal. # Motivation Why are we doing this? What use cases does it support? What is the expected outcome? # Guide-level explanation Explain the proposal as if it was already included in the opendal and you were teaching it to other opendal users. That generally means: - Introducing new named concepts. - Explaining the feature mainly in terms of examples. - Explaining how opendal users should *think* about the feature and how it should impact the way they use opendal. It should explain the impact as concretely as possible. - If applicable, provide sample error messages, deprecation warnings, or migration guidance. - If applicable, describe the differences between teaching this to exist opendal users and new opendal users. # Reference-level explanation This is the technical portion of the RFC. Explain the design in sufficient detail that: - Its interaction with other features is clear. - It is reasonably clear how the feature would be implemented. - Corner cases are dissected by example. The section should return to the examples given in the previous section and explain more fully how the detailed proposal makes those examples work. # Drawbacks Why should we *not* do this? # Rationale and alternatives - Why is this design the best in the space of possible designs? - What other designs have been considered, and what is the rationale for not choosing them? - What is the impact of not doing this? # Prior art Discuss prior art, both the good and the bad, in relation to this proposal. A few examples of what this can include are: - What lessons can we learn from what other communities have done here? This section is intended to encourage you as an author to think about the lessons from other communities provide readers of your RFC with a fuller picture. If there is no prior art, that is fine - your ideas are interesting to us, whether they are brand new or an adaptation from other projects. # Unresolved questions - What parts of the design do you expect to resolve through the RFC process before this gets merged? - What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? # Future possibilities Think about what the natural extension and evolution of your proposal would be and how it would affect the opendal. Try to use this section as a tool to more fully consider all possible interactions with the project in your proposal. Also, consider how this all fits into the roadmap for the project. This is also a good place to "dump ideas", if they are out of scope for the RFC, you are writing but otherwise related. If you have tried and cannot think of any future possibilities, you may state that you cannot think of anything. Note that having something written down in the future-possibilities section is not a reason to accept the current or a future RFC; such notes should be in the section on motivation or rationale in this or subsequent RFCs. The section merely provides additional information. opendal-0.52.0/src/docs/rfcs/0041_object_native_api.md000064400000000000000000000122511046102023000204420ustar 00000000000000- Proposal Name: `object_native_api` - Start Date: 2022-02-18 - RFC PR: [apache/opendal#41](https://github.com/apache/opendal/pull/41) - Tracking Issue: [apache/opendal#35](https://github.com/apache/opendal/pull/35) # Summary Refactor API in object native way to make it easier to user. # Motivation `opendal` is not easy to use. In our early adoption project `databend`, we can see a lot of code looks like: ```rust let data_accessor = self.data_accessor.clone(); let path = self.path.clone(); let reader = SeekableReader::new(data_accessor, path.as_str(), stream_len); let reader = BufReader::with_capacity(read_buffer_size as usize, reader); Self::read_column(reader, &col_meta, data_type.clone(), arrow_type.clone()).await ``` And ```rust op.stat(&path).run().await ``` ## Conclusion So in this proposal, I expect to address those problems. After implementing this proposal, we have a faster and easier-to-use `opendal`. # Guide-level explanation To operate on an object, we will use `Operator::object()` to create a new handler: ```rust let o = op.object("path/to/file"); ``` All operations that are available for `Object` for now includes: - `metadata`: get object metadata (return an error if not exist). - `delete`: delete an object. - `reader`: create a new reader to read data from this object. - `writer`: create a new writer to write data into this object. Here is an example: ```rust use anyhow::Result; use futures::AsyncReadExt; use opendal::services::fs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let op = Operator::new(fs::Backend::build().root("/tmp").finish().await?); let o = op.object("test_file"); // Write data info file; let w = o.writer(); let n = w .write_bytes("Hello, World!".to_string().into_bytes()) .await?; assert_eq!(n, 13); // Read data from file; let mut r = o.reader(); let mut buf = vec![]; let n = r.read_to_end(&mut buf).await?; assert_eq!(n, 13); assert_eq!(String::from_utf8_lossy(&buf), "Hello, World!"); // Get file's Metadata let meta = o.metadata().await?; assert_eq!(meta.content_length(), 13); // Delete file. o.delete().await?; Ok(()) } ``` # Reference-level explanation ## Native Reader support We will provide a `Reader` (which implement both `AsyncRead + AsyncSeek`) for user instead of just a `AsyncRead`. In this `Reader`, we will: - Not maintain internal buffer: caller can decide to wrap into `BufReader`. - Only rely on accessor's `read` and `stat` operations. To avoid the extra cost for `stat`, we will: - Allow user specify total_size for `Reader`. - Lazily Send `stat` while the first time `SeekFrom::End()` To avoid the extra cost for `poll_read`, we will: - Keep the underlying `BoxedAsyncRead` open, so that we can reuse the same connection/fd. With these change, we can improve the `Reader` performance both on local fs and remote storage: - fs, before ```shell Benchmarking fs/bench_read/64226295-b7a7-416e-94ce-666ac3ab037b: time: [16.060 ms 17.109 ms 18.124 ms] thrpt: [882.82 MiB/s 935.20 MiB/s 996.24 MiB/s] Benchmarking fs/bench_buf_read/64226295-b7a7-416e-94ce-666ac3ab037b: time: [14.779 ms 14.857 ms 14.938 ms] thrpt: [1.0460 GiB/s 1.0517 GiB/s 1.0572 GiB/s] ``` - fs, after ```shell Benchmarking fs/bench_read/df531bc7-54c8-43b6-b412-e4f7b9589876: time: [14.654 ms 15.452 ms 16.273 ms] thrpt: [983.20 MiB/s 1.0112 GiB/s 1.0663 GiB/s] Benchmarking fs/bench_buf_read/df531bc7-54c8-43b6-b412-e4f7b9589876: time: [5.5589 ms 5.5825 ms 5.6076 ms] thrpt: [2.7864 GiB/s 2.7989 GiB/s 2.8108 GiB/s] ``` - s3, before ```shell Benchmarking s3/bench_read/72025a81-a4b6-46dc-b485-8d875d23c3a5: time: [4.8315 ms 4.9331 ms 5.0403 ms] thrpt: [3.1000 GiB/s 3.1674 GiB/s 3.2340 GiB/s] Benchmarking s3/bench_buf_read/72025a81-a4b6-46dc-b485-8d875d23c3a5: time: [16.246 ms 16.539 ms 16.833 ms] thrpt: [950.52 MiB/s 967.39 MiB/s 984.84 MiB/s] ``` - s3, after ```shell Benchmarking s3/bench_read/6971c464-15f7-48d6-b69c-c8abc7774802: time: [4.4222 ms 4.5685 ms 4.7181 ms] thrpt: [3.3117 GiB/s 3.4202 GiB/s 3.5333 GiB/s] Benchmarking s3/bench_buf_read/6971c464-15f7-48d6-b69c-c8abc7774802: time: [5.5598 ms 5.7174 ms 5.8691 ms] thrpt: [2.6622 GiB/s 2.7329 GiB/s 2.8103 GiB/s] ``` ## Object API Other changes are just a re-order of APIs. - `Operator::read() -> BoxedAsyncRead` => `Object::reader() -> Reader` - `Operator::write(r: BoxedAsyncRead, size: u64)` => `Object::writer() -> Writer` - `Operator::stat() -> Object` => `Object::stat() -> Metadata` - `Operator::delete()` => `Object::delete()` # Drawbacks None. # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities - Implement `AsyncWrite` for `Writer` so that we can use `Writer` easier. - Implement `Operator::objects()` to return an object iterator. opendal-0.52.0/src/docs/rfcs/0044_error_handle.md000064400000000000000000000113311046102023000174420ustar 00000000000000- Proposal Name: `error_handle` - Start Date: 2022-02-23 - RFC PR: [apache/opendal#44](https://github.com/apache/opendal/pull/44) - Tracking Issue: [apache/opendal#43](https://github.com/apache/opendal/pull/43) # Summary Enhanced error handling for OpenDAL. # Motivation OpenDAL didn't handle errors correctly. ```rust fn parse_unexpected_error(_: SdkError, path: &str) -> Error { Error::Unexpected(path.to_string()) } ``` Most time, we return a path that is meaningless for debugging. There are two issues about this shortcoming: - [error: Split ErrorKind and Context for error check easier](https://github.com/apache/opendal/issues/24) - [Improvement: provides more information about the cause of DalTransportError](https://github.com/apache/opendal/issues/29) First, we can't check `ErrorKind` quickly. We have to use `matches` for the help: ```rust assert!( matches!( result.err().unwrap(), opendal::error::Error::ObjectNotExist(_) ), ); ``` Then, we didn't bring enough information for users to debug what happened inside OpenDAL. So we must handle errors correctly, so that: - We can check the `Kind` to know what error happened. - We can read `context` to know more details. - We can get the source of this error to know more details. # Guide-level explanation Now we are trying to get an object's metadata: ```rust let meta = o.metadata().await; ``` Unfortunately, the `Object` does not exist, so we can check out what happened. ```rust if let Err(e) = meta { if e.kind() == Kind::ObjectNotExist { // Handle this error } } ``` It's possible that we don't care about other errors. It's OK to log it out: ```rust if let Err(e) = meta { if e.kind() == Kind::ObjectNotExist { // Handle this error } else { error!("{e}"); } } ``` For a backend implementer, we can provide as much information as possible. For example, we can return `bucket is empty` to let the user know: ```rust return Err(Error::Backend { kind: Kind::BackendConfigurationInvalid, context: HashMap::from([("bucket".to_string(), "".to_string())]), source: anyhow!("bucket is empty"), }); ``` Or, we can return an underlying error to let users figure out: ```rust Error::Object { kind: Kind::Unexpected, op, path: path.to_string(), source: anyhow::Error::from(err), } ``` So our application users will get enough information now: ```shell Object { kind: ObjectNotExist, op: "stat", path: "/tmp/998e4dec-c84b-4164-a7a1-1f140654934f", source: No such file or directory (os error 2) } ``` # Reference-level explanation We will split `Error` into `Error` and `Kind`. `Kind` is an enum organized by different categories. Every error will map to a kind, which will be in the error message. ```rust pub enum Kind { #[error("backend not supported")] BackendNotSupported, #[error("backend configuration invalid")] BackendConfigurationInvalid, #[error("object not exist")] ObjectNotExist, #[error("object permission denied")] ObjectPermissionDenied, #[error("unexpected")] Unexpected, } ``` In `Error`, we will have different struct to carry different contexts: ```rust pub enum Error { #[error("{kind}: (context: {context:?}, source: {source})")] Backend { kind: Kind, context: HashMap, source: anyhow::Error, }, #[error("{kind}: (op: {op}, path: {path}, source: {source})")] Object { kind: Kind, op: &'static str, path: String, source: anyhow::Error, }, #[error("unexpected: (source: {0})")] Unexpected(#[from] anyhow::Error), } ``` Every one of them will carry a source: `anyhow::Error` so that users can get the complete picture of this error. We have implemented `Error::kind()`, other helper functions are possible, but they are out of this RFC's scope. ```rust pub fn kind(&self) -> Kind { match self { Error::Backend { kind, .. } => *kind, Error::Object { kind, .. } => *kind, Error::Unexpected(_) => Kind::Unexpected, } } ``` The implementer should do their best to carry as much context as possible. Such as, they should return `Error::Object` to carry the `op` and `path`, instead of just returns `Error::Unexpected(anyhow::Error::from(err))`. ```rust Error::Object { kind: Kind::Unexpected, op, path: path.to_string(), source: anyhow::Error::from(err), } ``` # Drawbacks None # Rationale and alternatives ## Why don't we implement `backtrace`? `backtrace` is not stable yet, and `OpenDAL` must be compilable on stable Rust. This proposal doesn't erase the possibility to add support once `backtrace` is stable. # Prior art None # Unresolved questions None # Future possibilities - `Backtrace` support. opendal-0.52.0/src/docs/rfcs/0057_auto_region.md000064400000000000000000000124141046102023000173200ustar 00000000000000- Proposal Name: `auto_region` - Start Date: 2022-02-24 - RFC PR: [apache/opendal#57](https://github.com/apache/opendal/pull/57) - Tracking Issue: [apache/opendal#58](https://github.com/apache/opendal/issues/58) # Summary Automatically detecting user's s3 region. # Motivation Current behavior for `region` and `endpoint` is buggy. `endpoint=https://s3.amazonaws.com` and `endpoint=""` are expected to be the same, because `endpoint=""` means take the default value `https://s3.amazonaws.com`. However, they aren't. S3 SDK has a mechanism to construct the correct API endpoint. It works like `format!("s3.{}.amazonaws.com", region)` internally. But if we specify the endpoint to `https://s3.amazonaws.com`, SDK will take this endpoint static. So users could meet errors like: ```shell attempting to access must be addressed using the specified endpoint ``` Automatically detecting the user's s3 region will help resolve this problem. Users don't need to care about the region anymore, `OpenDAL` will figure it out. Everything works regardless of whether the input is `s3.amazonaws.com` or `s3.us-east-1.amazonaws.com`. # Guide-level explanation `OpenDAL` will remove `region` option, and users only need to set the `endpoint` now. Valid input including: - `https://s3.amazonaws.com` - `https://s3.us-east-1.amazonaws.com` - `https://oss-ap-northeast-1.aliyuncs.com` - `http://127.0.0.1:9000` `OpenDAL` will handle the `region` internally and automatically. # Reference-level explanation S3 services support mechanism to indicate the correct region on itself. Sending a `HEAD` request to `/` will get a response like: ```shell :) curl -I https://s3.amazonaws.com/databend-shared HTTP/1.1 301 Moved Permanently x-amz-bucket-region: us-east-2 x-amz-request-id: NPYSWK7WXJD1KQG7 x-amz-id-2: 3FJSJ5HACKqLbeeXBUUE3GoPL1IGDjLl6SZx/fw2MS+k0GND0UwDib5YQXE6CThiQxpYBWZjgxs= Content-Type: application/xml Date: Thu, 24 Feb 2022 05:15:13 GMT Server: AmazonS3 ``` `x-amz-bucket-region: us-east-2` will be returned, and we can use this region to construct the correct endpoint for this bucket: ```shell :) curl -I https://s3.us-east-2.amazonaws.com/databend-shared HTTP/1.1 403 Forbidden x-amz-bucket-region: us-east-2 x-amz-request-id: 98CN5MYV3GQ1XMPY x-amz-id-2: Tdxy36bRRP21Oip18KMQ7FG63MTeXOpXdd5/N3izFH0oalPODVaRlpCkDU3oUN0HIE24/ezX5Dc= Content-Type: application/xml Date: Thu, 24 Feb 2022 05:16:57 GMT Server: AmazonS3 ``` It also works for S3 compilable services like minio: ```shell # Start minio with `MINIO_SITE_REGION` configured :) MINIO_SITE_REGION=test minio server . # Sending request to minio bucket :) curl -I 127.0.0.1:9900/databend HTTP/1.1 403 Forbidden Accept-Ranges: bytes Content-Length: 0 Content-Security-Policy: block-all-mixed-content Server: MinIO Strict-Transport-Security: max-age=31536000; includeSubDomains Vary: Origin Vary: Accept-Encoding X-Amz-Bucket-Region: test X-Amz-Request-Id: 16D6A12DCA57E0FA X-Content-Type-Options: nosniff X-Xss-Protection: 1; mode=block Date: Thu, 24 Feb 2022 05:18:51 GMT ``` We can use this mechanism to detect `region` automatically. The algorithm works as follows: - If `endpoint` is empty, fill it will `https://s3.amazonaws.com` and the corresponding template: `https://s3.{region}.amazonaws.com`. - Sending a `HEAD` request to `/`. - If got `200` or `403` response, the endpoint works. - Use this endpoint directly without filling the template. - Take the header `x-amz-bucket-region` as the region to fill the endpoint. - Use the fallback value `us-east-1` to make SDK happy if the header not exists. - If got a `301` response, the endpoint needs construction. - Take the header `x-amz-bucket-region` as the region to fill the endpoint. - Return an error to the user if not exist. - If got `404`, the bucket could not exist, or the endpoint is incorrect. - Return an error to the user. # Drawbacks None. # Rationale and alternatives ## Use virtual style `.`? The virtual style works too. But not all services support this kind of API endpoint. For example, using `http://testbucket.127.0.0.1` is wrong, and we need to do extra checks. Using `/` makes everything easier. ## Use `ListBuckets` API? `ListBuckets` requires higher permission than normal bucket read and write operations. It's better to finish the job without requesting more permission. ## Misbehavior S3 Compilable Services Many services didn't implement S3 API correctly. Aliyun OSS will return `404` for every bucket: ```shell :) curl -I https://aliyuncs.com/ HTTP/2 404 date: Thu, 24 Feb 2022 05:32:57 GMT content-type: text/html content-length: 690 ufe-result: A6 set-cookie: thw=cn; Path=/; Domain=.taobao.com; Expires=Fri, 24-Feb-23 05:32:57 GMT; server: Tengine/Aserver ``` QingStor Object Storage will return `307` with the `Location` header: ```shell :) curl -I https://s3.qingstor.com/community HTTP/1.1 301 Moved Permanently Server: nginx/1.13.6 Date: Thu, 24 Feb 2022 05:33:55 GMT Connection: keep-alive Location: https://pek3a.s3.qingstor.com/community X-Qs-Request-Id: 05b83b615c801a3d ``` In this proposal, we will not figure them out. It's easier for the user to fill the correct endpoint instead of automatically detecting them. # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/0069_object_stream.md000064400000000000000000000063331046102023000176340ustar 00000000000000- Proposal Name: `object_stream` - Start Date: 2022-02-25 - RFC PR: [apache/opendal#69](https://github.com/apache/opendal/pull/69) - Tracking Issue: [apache/opendal#69](https://github.com/apache/opendal/issues/69) # Summary Allow user to read dir via `ObjectStream`. # Motivation Users need `readdir` support in `OpenDAL`: [Implement List support](https://github.com/apache/opendal/issues/12). Take [databend] for example, with `List` support, we can implement copy from `s3://bucket/path/to/dir` instead of only `s3://bucket/path/to/file`. # Guide-level explanation `Operator` supports new action called `objects("path/to/dir")` which returns a `ObjectStream`, we can iterator current dir like `std::fs::ReadDir`: ```rust let mut obs = op.objects("").map(|o| o.expect("list object")); while let Some(o) = obs.next().await { // Do something upon `Object`. } ``` To better support different file modes, there is a new object meta called `ObjectMode`: ```rust let meta = o.metadata().await?; let mode = meta.mode(); if mode.contains(ObjectMode::FILE) { // Do something on a file object. } else if mode.contains(ObjectMode::DIR) { // Do something on a dir object. } ``` We will try to cache some object metadata so that users can reduce `stat` calls: ```rust let meta = o.metadata_cached().await?; ``` `o.metadata_cached()` will return local cached metadata if available. # Reference-level explanation First, we will add a new API in `Accessor`: ```rust pub type BoxedObjectStream = Box> + Unpin + Send>; async fn list(&self, args: &OpList) -> Result { let _ = args; unimplemented!() } ``` To support options in the future, we will wrap this call via `ObjectStream`: ```rust pub struct ObjectStream { acc: Arc, path: String, state: State, } enum State { Idle, Sending(BoxFuture<'static, Result>), Listing(BoxedObjectStream), } ``` So the public API to end-users will be: ```rust impl Operator { pub fn objects(&self, path: &str) -> ObjectStream { ObjectStream::new(self.inner(), path) } } ``` For cached metadata support, we will add a flag in `Metadata`: ```rust #[derive(Debug, Clone, Default)] pub struct Metadata { complete: bool, path: String, mode: Option, content_length: Option, } ``` And add new API `Objbct::metadata_cached()`: ```rust pub async fn metadata_cached(&mut self) -> Result<&Metadata> { if self.meta.complete() { return Ok(&self.meta); } let op = &OpStat::new(self.meta.path()); self.meta = self.acc.stat(op).await?; Ok(&self.meta) } ``` The backend implementer must make sure `complete` is correctly set. `Metadata` will be immutable outsides, so all `set_xxx` APIs will be set to crate public only: ```rust pub(crate) fn set_content_length(&mut self, content_length: u64) -> &mut Self { self.content_length = Some(content_length); self } ``` # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities - More precise field-level metadata cache so that user can send `stat` only when needed. [databend]: https://github.com/datafuselabs/databend opendal-0.52.0/src/docs/rfcs/0090_limited_reader.md000064400000000000000000000104411046102023000177510ustar 00000000000000- Proposal Name: `limited_reader` - Start Date: 2022-03-02 - RFC PR: [apache/opendal#0090](https://github.com/apache/opendal/pull/0090) - Tracking Issue: [apache/opendal#0090](https://github.com/apache/opendal/issues/0090) # Summary Native support for the limited reader. # Motivation In proposal [object-native-api](./0041-object-native-api.md) we introduced `Reader`, in which we will send request like: ```rust let op = OpRead { path: self.path.to_string(), offset: Some(self.current_offset()), size: None, }; ``` In this implementation, we depend on the HTTP client to drop the request when we stop reading. However, we always read too much extra data, which decreases our reading performance. Here is a benchmark around reading the whole file and only reading half: ```txt s3/read/1c741003-40ef-43a9-b23f-b6a32ed7c4c6 time: [7.2697 ms 7.3521 ms 7.4378 ms] thrpt: [2.1008 GiB/s 2.1252 GiB/s 2.1493 GiB/s] s3/read_half/1c741003-40ef-43a9-b23f-b6a32ed7c4c6 time: [7.0645 ms 7.1524 ms 7.2473 ms] thrpt: [1.0780 GiB/s 1.0923 GiB/s 1.1059 GiB/s] ``` So our current behavior is buggy, and we need more clear API to address that. # Guide-level explanation We will remove `Reader::total_size()` from public API instead of adding the following APIs for `Object`: ```rust pub fn reader(&self) -> Reader {} pub fn range_reader(&self, offset: u64, size: u64) -> Reader {} pub fn offset_reader(&self, offset: u64) -> Reader {} pub fn limited_reader(&self, size: u64) -> Reader {} ``` - `reader`: returns a new reader who can read the whole file. - `range_reader`: returns a ranged reader which read `[offset, offset+size)`. - `offset_reader`: returns a reader from offset `[offset:]` - `limited_reader`: returns a limited reader `[:size]` Take `parquet`'s actual logic as an example. We can rewrite: ```rust async fn _read_single_column_async<'b, R, F>( factory: F, meta: &ColumnChunkMetaData, ) -> Result<(&ColumnChunkMetaData, Vec)> where R: AsyncRead + AsyncSeek + Send + Unpin, F: Fn() -> BoxFuture<'b, std::io::Result>, { let mut reader = factory().await?; let (start, len) = meta.byte_range(); reader.seek(std::io::SeekFrom::Start(start)).await?; let mut chunk = vec![0; len as usize]; reader.read_exact(&mut chunk).await?; Result::Ok((meta, chunk)) } ``` into ```rust async fn _read_single_column_async<'b, R, F>( factory: F, meta: &ColumnChunkMetaData, ) -> Result<(&ColumnChunkMetaData, Vec)> where R: AsyncRead + AsyncSeek + Send + Unpin, F: Fn(usize, usize) -> BoxFuture<'b, std::io::Result>, { let (start, len) = meta.byte_range(); let mut reader = factory(start, len).await?; let mut chunk = vec![0; len as usize]; reader.read_exact(&mut chunk).await?; Result::Ok((meta, chunk)) } ``` So that: - No extra data will be read. - No extra `seek`/`stat` operation is needed. # Reference-level explanation Inside `Reader`, we will correctly maintain `offset`, `size`, and `pos`. - If `offset` is `None`, we will use `0` instead. - If `size` is `None`, we will use `meta.content_length() - self.offset.unwrap_or_default()` instead. We will calculate `Reader` current offset and size easily: ```rust fn current_offset(&self) -> u64 { self.offset.unwrap_or_default() + self.pos } fn current_size(&self) -> Option { self.size.map(|v| v - self.pos) } ``` Instead of constantly requesting the entire object content, we will set the size: ```rust let op = OpRead { path: self.path.to_string(), offset: Some(self.current_offset()), size: self.current_size(), }; ``` After this change, we will have a similar throughput for `read_all` and `read_half`: ```txt s3/read/6dd40f8d-7455-451e-b510-3b7ac23e0468 time: [4.9554 ms 5.0888 ms 5.2282 ms] thrpt: [2.9886 GiB/s 3.0704 GiB/s 3.1532 GiB/s] s3/read_half/6dd40f8d-7455-451e-b510-3b7ac23e0468 time: [3.1868 ms 3.2494 ms 3.3052 ms] thrpt: [2.3637 GiB/s 2.4043 GiB/s 2.4515 GiB/s] ``` # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities - Refactor the parquet reading logic to make the most use of `range_reader`. opendal-0.52.0/src/docs/rfcs/0112_path_normalization.md000064400000000000000000000054321046102023000207010ustar 00000000000000- Proposal Name: `path-normalization` - Start Date: 2022-03-08 - RFC PR: [apache/opendal#112](https://github.com/apache/opendal/pull/112) - Tracking Issue: [apache/opendal#112](https://github.com/apache/opendal/issues/112) # Summary Implement path normalization to enhance user experience. # Motivation OpenDAL's current path behavior makes users confused: - [operator.object("/admin/data/") error](https://github.com/apache/opendal/issues/107) - [Read /admin/data//ontime_200.csv return empty](https://github.com/apache/opendal/issues/109) They are different bugs that reflect the exact root cause: the path is not well normalized. On local fs, we can read the same path with different path: `abc/def/../def`, `abc/def`, `abc//def`, `abc/./def`. There is no magic here: our stdlib does the dirty job. For example: - [std::path::PathBuf::canonicalize](https://doc.rust-lang.org/std/path/struct.PathBuf.html#method.canonicalize): Returns the canonical, absolute form of the path with all intermediate components normalized and symbolic links resolved. - [std::path::PathBuf::components](https://doc.rust-lang.org/std/path/struct.PathBuf.html#method.components): Produces an iterator over the Components of the path. When parsing the path, there is a small amount of normalization... But for s3 alike storage system, there's no such helpers: `abc/def/../def`, `abc/def`, `abc//def`, `abc/./def` refers entirely different objects. So users may confuse why I can't get the object with this path. So OpenDAL needs to implement path normalization to enhance the user experience. # Guide-level explanation We will do path normalization automatically. The following rules will be applied (so far): - Remove `//` inside path: `op.object("abc/def")` and `op.object("abc//def")` will resolve to the same object. - Make sure path under `root`: `op.object("/abc")` and `op.object("abc")` will resolve to the same object. Other rules still need more consideration to leave them for the future. # Reference-level explanation We will build the absolute path via `{root}/{path}` and replace all `//` into `/` instead. # Drawbacks None # Rationale and alternatives ## How about the link? If we build an actual path via `{root}/{path}`, the link object may be inaccessible. I don't have good ideas so far. Maybe we can add a new flag to control the link behavior. For now, there's no feature request for link support. Let's leave for the future to resolve. ## S3 URI Clean For s3, `abc//def` is different from `abc/def` indeed. To make it possible to access not normalized path, we can provide a new flag for the builder: ```rust let builder = Backend::build().disable_path_normalization() ``` In this way, the user can control the path more precisely. # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/0191_async_streaming_io.md000064400000000000000000000300251046102023000206570ustar 00000000000000- Proposal Name: `async_streaming_io` - Start Date: 2022-03-28 - RFC PR: [apache/opendal#191](https://github.com/apache/opendal/pull/191) - Tracking Issue: [apache/opendal#190](https://github.com/apache/opendal/issues/190) **Reverted** # Summary Use `Stream`/`Sink` instead of `AsyncRead` in `Accessor`. # Motivation `Accessor` intends to be the `underlying trait of all backends for implementers`. However, it's not so underlying enough. ## Over-wrapped `Accessor` returns a `BoxedAsyncReader` for `read` operation: ```rust pub type BoxedAsyncReader = Box; pub trait Accessor { async fn read(&self, args: &OpRead) -> Result { let _ = args; unimplemented!() } } ``` And we are exposing `Reader`, which implements `AsyncRead` and `AsyncSeek` to end-users. For every call to `Reader::poll_read()`, we need: - `Reader::poll_read()` - `BoxedAsyncReader::poll_read()` - `IntoAsyncRead::poll_read()` - `ByteStream::poll_next()` If we could return a `Stream` directly, we can transform the call stack into: - `Reader::poll_read()` - `ByteStream::poll_next()` In this way, we operate on the underlying IO stream, and the caller must keep track of the reading states. ## Inconsistent OpenDAL's `read` and `write` behavior is not consistent. ```rust pub type BoxedAsyncReader = Box; pub trait Accessor: Send + Sync + Debug { async fn read(&self, args: &OpRead) -> Result { let _ = args; unimplemented!() } async fn write(&self, r: BoxedAsyncReader, args: &OpWrite) -> Result { let (_, _) = (r, args); unimplemented!() } } ``` For `read`, OpenDAL returns a `BoxedAsyncReader` which users can decide when and how to read data. But for `write`, OpenDAL accepts a `BoxedAsyncReader` instead, in which users can't control the writing logic. How large will the writing buffer size be? When to call `flush`? ## Service native optimization OpenDAL knows more about the service detail, but returning `BoxedAsyncReader` makes it can't fully use the advantage. For example, most object storage services use HTTP to transfer data which is TCP stream-based. The most efficient way is to return a full TCP buffer, but users don't know about that. First, users could have continuous small reads on stream. To overcome the poor performance, they have to use `BufReader`, which adds a new buffering between reading. Then, users don't know the correct (best) buffer size to set. Via returning a `Stream`, users could benefit from it in both ways: - Users who want underlying control can operate on the `Stream` directly. - Users who don't care about the behavior can use OpenDAL provided Reader, which always adopts the best optimization. # Guide-level explanation Within the `async_streaming_io` feature, we will add the following new APIs to `Object`: ```rust impl Object { pub async fn stream(&self, offset: Option, size: Option) -> Result {} pub async fn sink(&self, size: u64) -> Result {} } ``` Users can control the underlying logic of those bytes, streams, and sinks. For example, they can: - Read data on demand: `stream.next().await` - Write data on demand: `sink.feed(bs).await; sink.close().await;` Based on `stream` and `sink`, `Object` will provide more optimized helper functions like: - `async read(offset: Option, size: Option) -> Result` - `async write(bs: bytes::Bytes) -> Result<()>` # Reference-level explanation `read` and `write` in `Accessor` will be refactored into streaming-based: ```rust pub type BytesStream = Box; pub type BytesSink = Box; pub trait Accessor: Send + Sync + Debug { async fn read(&self, args: &OpRead) -> Result { let _ = args; unimplemented!() } async fn write(&self, args: &OpWrite) -> Result { let _ = args; unimplemented!() } } ``` All other IO functions will be adapted to fit these changes. For fs, it's simple to implement `Stream` and `Sink` for `tokio::fs::File`. We will return a `BodySinker` instead for all HTTP-based storage services. In which we maintain a `put_object` `ResponseFuture` that construct by `hyper` and a `sender` part of the channel. All data sent by users will be passed to `ResponseFuture` via the unbuffered channel. ```rust struct BodySinker { fut: ResponseFuture, sender: Sender } ``` # Drawbacks ## Performance regression on fs `fs` is not stream based backend, and convert from `Reader` to `Stream` is not zero cost. Based on benchmark over `IntoStream`, we can get nearly 70% performance drawback (pure memory): ```rust into_stream/into_stream time: [1.3046 ms 1.3056 ms 1.3068 ms] thrpt: [2.9891 GiB/s 2.9919 GiB/s 2.9942 GiB/s] into_stream/raw_reader time: [382.10 us 383.52 us 385.16 us] thrpt: [10.142 GiB/s 10.185 GiB/s 10.223 GiB/s] ``` However, real fs is not as fast as memory and most overhead will happen at disk side, so that performance regression is allowed (at least at this time). # Rationale and alternatives ## Performance for switching from Reader to Stream Before ```rust read_full/4.00 KiB time: [455.70 us 466.18 us 476.93 us] thrpt: [8.1904 MiB/s 8.3794 MiB/s 8.5719 MiB/s] read_full/256 KiB time: [530.63 us 544.30 us 557.84 us] thrpt: [448.16 MiB/s 459.30 MiB/s 471.14 MiB/s] read_full/4.00 MiB time: [1.5569 ms 1.6152 ms 1.6743 ms] thrpt: [2.3330 GiB/s 2.4184 GiB/s 2.5090 GiB/s] read_full/16.0 MiB time: [5.7337 ms 5.9087 ms 6.0813 ms] thrpt: [2.5693 GiB/s 2.6444 GiB/s 2.7251 GiB/s] ``` After ```rust read_full/4.00 KiB time: [455.67 us 466.03 us 476.21 us] thrpt: [8.2027 MiB/s 8.3819 MiB/s 8.5725 MiB/s] change: time: [-2.1168% +0.6241% +3.8735%] (p = 0.68 > 0.05) thrpt: [-3.7291% -0.6203% +2.1625%] No change in performance detected. read_full/256 KiB time: [521.04 us 535.20 us 548.74 us] thrpt: [455.59 MiB/s 467.11 MiB/s 479.81 MiB/s] change: time: [-7.8470% -4.7987% -1.4955%] (p = 0.01 < 0.05) thrpt: [+1.5182% +5.0406% +8.5152%] Performance has improved. read_full/4.00 MiB time: [1.4571 ms 1.5184 ms 1.5843 ms] thrpt: [2.4655 GiB/s 2.5725 GiB/s 2.6808 GiB/s] change: time: [-5.4403% -1.5696% +2.3719%] (p = 0.44 > 0.05) thrpt: [-2.3170% +1.5946% +5.7533%] No change in performance detected. read_full/16.0 MiB time: [5.0201 ms 5.2105 ms 5.3986 ms] thrpt: [2.8943 GiB/s 2.9988 GiB/s 3.1125 GiB/s] change: time: [-15.917% -11.816% -7.5219%] (p = 0.00 < 0.05) thrpt: [+8.1337% +13.400% +18.930%] Performance has improved. ``` ## Performance for the extra channel in `write` Based on the benchmark during research, the **unbuffered** channel does improve the performance a bit in some cases: Before: ```rust write_once/4.00 KiB time: [564.11 us 575.17 us 586.15 us] thrpt: [6.6642 MiB/s 6.7914 MiB/s 6.9246 MiB/s] write_once/256 KiB time: [1.3600 ms 1.3896 ms 1.4168 ms] thrpt: [176.46 MiB/s 179.90 MiB/s 183.82 MiB/s] write_once/4.00 MiB time: [11.394 ms 11.555 ms 11.717 ms] thrpt: [341.39 MiB/s 346.18 MiB/s 351.07 MiB/s] write_once/16.0 MiB time: [41.829 ms 42.645 ms 43.454 ms] thrpt: [368.20 MiB/s 375.19 MiB/s 382.51 MiB/s] ``` After: ```rust write_once/4.00 KiB time: [572.20 us 583.62 us 595.21 us] thrpt: [6.5628 MiB/s 6.6932 MiB/s 6.8267 MiB/s] change: time: [-6.3126% -3.8179% -1.0733%] (p = 0.00 < 0.05) thrpt: [+1.0849% +3.9695% +6.7380%] Performance has improved. write_once/256 KiB time: [1.3192 ms 1.3456 ms 1.3738 ms] thrpt: [181.98 MiB/s 185.79 MiB/s 189.50 MiB/s] change: time: [-0.5899% +1.7476% +4.1037%] (p = 0.15 > 0.05) thrpt: [-3.9420% -1.7176% +0.5934%] No change in performance detected. write_once/4.00 MiB time: [10.855 ms 11.039 ms 11.228 ms] thrpt: [356.25 MiB/s 362.34 MiB/s 368.51 MiB/s] change: time: [-6.9651% -4.8176% -2.5681%] (p = 0.00 < 0.05) thrpt: [+2.6358% +5.0614% +7.4866%] Performance has improved. write_once/16.0 MiB time: [38.706 ms 39.577 ms 40.457 ms] thrpt: [395.48 MiB/s 404.27 MiB/s 413.37 MiB/s] change: time: [-10.829% -8.3611% -5.8702%] (p = 0.00 < 0.05) thrpt: [+6.2363% +9.1240% +12.145%] Performance has improved. ``` ## Add complexity on the services side Returning `Stream` and `Sink` make it complex to implement. At first glance, it does. But in reality, it's not. Note: HTTP (especially for hyper) is stream-oriented. - Returning a `stream` is more straightforward than `reader`. - Returning `Sink` is covered by the global shared `BodySinker` struct. Other helper functions will be covered at the Object-level which services don't need to bother. # Prior art ## Returning a `Writer` The most natural extending is to return `BoxedAsyncWriter`: ```rust pub trait Accessor: Send + Sync + Debug { /// Read data from the underlying storage into input writer. async fn read(&self, args: &OpRead) -> Result { let _ = args; unimplemented!() } /// Write data from input reader to the underlying storage. async fn write(&self, args: &OpWrite) -> Result { let _ = args; unimplemented!() } } ``` But it only fixes the `Inconsistent` concern and can't help with other issues. ## Slice based API Most rust IO APIs are based on slice: ```rust pub trait Accessor: Send + Sync + Debug { /// Read data from the underlying storage into input writer. async fn read(&self, args: &OpRead, bs: &mut [u8]) -> Result { let _ = args; unimplemented!() } /// Write data from input reader to the underlying storage. async fn write(&self, args: &OpWrite, bs: &[u8]) -> Result { let _ = args; unimplemented!() } } ``` The problem is `Accessor` doesn't have states: - If we require all data must be passed at one time, we can't support large files read & write - If we allow users to call `read`/`write` multiple times, we need to implement another `Reader` and `Writer` alike logic. ## Accept `Reader` and `Writer` It's also possible to accept `Reader` and `Writer` instead. ```rust pub trait Accessor: Send + Sync + Debug { /// Read data from the underlying storage into input writer. async fn read(&self, args: &OpRead, w: BoxedAsyncWriter) -> Result { let _ = args; unimplemented!() } /// Write data from input reader to the underlying storage. async fn write(&self, args: &OpWrite, r: BoxedAsyncReader) -> Result { let _ = args; unimplemented!() } } ``` This API design addressed all concerns but made it hard for users to use. Primarily, we can't support `futures::AsyncRead` and `tokio::AsyncRead` simultaneously. For example, we can't accept a `Box::new(Vec::new())`, user can't get this vec from OpenDAL. # Unresolved questions None. # Future possibilities - Implement `Object::read_into(w: BoxedAsyncWriter)` - Implement `Object::write_from(r: BoxedAsyncReader)` opendal-0.52.0/src/docs/rfcs/0203_remove_credential.md000064400000000000000000000047021046102023000204660ustar 00000000000000- Proposal Name: `remove_credential` - Start Date: 2022-04-02 - RFC PR: [apache/opendal#203](https://github.com/apache/opendal/pull/203) - Tracking Issue: [apache/opendal#203](https://github.com/apache/opendal/issues/203) # Summary Remove the concept of credential. # Motivation `Credential` intends to carry service credentials like `access_key_id` and `secret_access_key`. At OpenDAL, we designed a global `Credential` enum for services and users to use. ```rust pub enum Credential { /// Plain refers to no credential has been provided, fallback to services' /// default logic. Plain, /// Basic refers to HTTP Basic Authentication. Basic { username: String, password: String }, /// HMAC, also known as Access Key/Secret Key authentication. HMAC { access_key_id: String, secret_access_key: String, }, /// Token refers to static API token. Token(String), } ``` However, every service only supports one kind of `Credential` with different `Credential` load methods covered by [reqsign](https://github.com/Xuanwo/reqsign). As a result, only `HMAC` is used. Both users and services need to write the same logic again and again. # Guide-level explanation `Credential` will be removed, and the services builder will provide native credential representation directly. For s3: ```rust pub fn access_key_id(&mut self, v: &str) -> &mut Self {} pub fn secret_access_key(&mut self, v: &str) -> &mut Self {} ``` For azblob: ```rust pub fn account_name(&mut self, account_name: &str) -> &mut Self {} pub fn account_key(&mut self, account_key: &str) -> &mut Self {} ``` All builders must implement `Debug` by hand and redact sensitive fields to avoid credentials being a leak. ```rust impl Debug for Builder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Builder"); ds.field("root", &self.root); ds.field("container", &self.container); ds.field("endpoint", &self.endpoint); if self.account_name.is_some() { ds.field("account_name", &""); } if self.account_key.is_some() { ds.field("account_key", &""); } ds.finish_non_exhaustive() } } ``` # Reference-level explanation Simple change without reference-level explanation needs. # Drawbacks API Breakage. # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/0221_create_dir.md000064400000000000000000000044031046102023000170760ustar 00000000000000- Proposal Name: `create-dir` - Start Date: 2022-04-06 - RFC PR: [apache/opendal#221](https://github.com/apache/opendal/pull/221) - Tracking Issue: [apache/opendal#222](https://github.com/apache/opendal/issues/222) # Summary Add creating dir support for OpenDAL. # Motivation Interoperability between OpenDAL services requires dir support. The object storage system will simulate dir operations with `/` via object ends. But we can't share the same behavior with `fs`, as `mkdir` is a separate syscall. So we need to unify the behavior about dir across different services. # Guide-level explanation After this proposal got merged, we will treat all paths that end with `/` as a dir. For example: - `read("abc/")` will return an `IsDir` error. - `write("abc/")` will return an `IsDir` error. - `stat("abc/")` will be guaranteed to return a dir or a `NotDir` error. - `delete("abc/")` will be guaranteed to delete a dir or `NotDir` / `NotEmpty` error. - `list("abc/")` will be guaranteed to list a dir or a `NotDir` error. And we will support create an empty object: ```rust // create a dir object "abc/" let _ = op.object("abc/").create().await?; // create a file object "abc" let _ = op.object("abc").create().await?; ``` # Reference-level explanation And we will add a new API called `create` to create an empty object. ```rust struct OpCreate { path: String, mode: ObjectMode, } pub trait Accessor: Send + Sync + Debug { async fn create(&self, args: &OpCreate) -> Result; } ``` `Object` will expose API like `create` which will call `Accessor::create()` internally. # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions When writing this proposal, [io_error_more](https://github.com/rust-lang/rust/issues/86442) is not stabilized yet. We can't use `NotADirectory` nor `IsADirectory` directly. Using `from_raw_os_error` is unacceptable because we can't carry our error context. ```rust use std::io; let error = io::Error::from_raw_os_error(22); assert_eq!(error.kind(), io::ErrorKind::InvalidInput); ``` So we will use `ErrorKind::Other` for now, which means our users can't check the following errors: - `IsADirectory` - `DirectoryNotEmpty` - `NotADirectory` Until they get stabilized. # Future possibilities None opendal-0.52.0/src/docs/rfcs/0247_retryable_error.md000064400000000000000000000053031046102023000202070ustar 00000000000000- Proposal Name: `retryable_error` - Start Date: 2022-04-12 - RFC PR: [apache/opendal#247](https://github.com/apache/opendal/pull/247) - Tracking Issue: [apache/opendal#248](https://github.com/apache/opendal/issues/248) # Summary Treat `io::ErrorKind::Interrupt` as retryable error. # Motivation Supports retry make our users' lives easier: > [Feature request: Custom retries for the s3 backend](https://github.com/apache/opendal/issues/196) > > While the reading/writing from/to s3, AWS occasionally returns errors that could be retried (at least 5xx?). Currently, in the databend, this will fail the whole execution of the statement (which may have been running for an extended time). Most users may need this retry feature, like `decompress`. Implementing it in OpenDAL will make users no bother, no backoff logic. # Guide-level explanation With the `retry` feature enabled: ```toml opendal = {version="0.5.2", features=["retry"]} ``` Users can configure the retry behavior easily: ```rust let backoff = ExponentialBackoff::default(); let op = op.with_backoff(backoff); ``` All requests sent by `op` will be automatically retried. # Reference-level explanation We will implement retry features via adding a new `Layer`. In the retry layer, we will support retrying all operations. To do our best to keep retrying read & write, we will implement `RetryableReader` and `RetryableWriter`, which will support retry while no actual IO happens. ## Retry operations Most operations are safe to retry, like `list`, `stat`, `delete` and `create`. We will retry those operations via input backoff. ## Retry IO operations Retry IO operations are a bit complex because IO operations have side effects, especially for HTTP-based services like s3. We can't resume an operation during the reading process without sending new requests. This proposal will do the best we can: retry the operation if no actual IO happens. If we meet an internal error before reading/writing the user's buffer, it's safe and cheap to retry it with precisely the same argument. ## Retryable Error - Operator MAY retry `io::ErrorKind::Interrupt` errors. - Services SHOULD return `io::ErrorKind::Interrupt` kind if the error is retryable. # Drawbacks ## Write operation can't be retried As we return `Writer` to users, there is no way for OpenDAL to get the input data again. # Rationale and alternatives ## Implement retry at operator level We need to implement retry logic for every operator function, and can't address the same problem: - `Reader` / `Writer` can't be retired. - Intrusive design that users cannot expand on their own # Prior art None # Unresolved questions - `read` and `write` can't be retried during IO. # Future possibilities None opendal-0.52.0/src/docs/rfcs/0293_object_id.md000064400000000000000000000033431046102023000167320ustar 00000000000000- Proposal Name: `object_id` - Start Date: 2022-05-27 - RFC PR: [apache/opendal#293](https://github.com/apache/opendal/pull/293) - Tracking Issue: [apache/opendal#294](https://github.com/apache/opendal/issues/294) # Summary Allow getting id from an object. # Motivation Allow get id from an object will make it possible to operate across different operators. Users can store objects' IDs locally and refer to them with different settings. This proposal will make tasks like backup, restore, and migration possible. # Guide-level explanation Users can fetch an object id via: ```rust let o = op.object("test_object"); let id = o.id(); ``` The id is unique and permanent inside the underlying storage. For example, if we have an s3 bucket with the root `/workdir/`, the object's id `test_object` will be `/workdir/test_object`. # Reference-level explanation `id()` and `path()` will be added as functions of `object`: ```rust impl Object { pub fn id(&self) -> String {} pub fn path(&self) -> String {} } ``` - `path` is a re-export of call to `Metadata::path()`. - `id` will be generated by Operator's root and `Metadata::path()`. # Drawbacks None # Rationale and alternatives ## Why not add a new field in `Metadata`? Adding a new field inside `Metadata` requires every service to handle the id separately. And every metadata will need to store a complete id with the operators' root. ## Why not provide a full URI like `s3://path/to/object`? Because we can't. A full and functional URI towards an object will need the operator's endpoint and credentials. It's better to provide the mechanism and allow users to construct them based on their own business. # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/0337_dir_entry.md000064400000000000000000000111251046102023000170030ustar 00000000000000- Proposal Name: `dir_entry` - Start Date: 2022-06-08 - RFC PR: [apache/opendal#337](https://github.com/apache/opendal/pull/337) - Tracking Issue: [apache/opendal#338](https://github.com/apache/opendal/issues/338) # Summary Returning `DirEntry` instead of `Object` in list. # Motivation In [Object Stream](./0069-object-stream.md), we introduce read_dir support via: ```rust pub trait ObjectStream: futures::Stream> + Unpin + Send {} impl ObjectStream for T where T: futures::Stream> + Unpin + Send {} pub struct Object { acc: Arc, meta: Metadata, } ``` However, the `meta` inside `Object` is not well-used: ```rust pub(crate) fn metadata_ref(&self) -> &Metadata {} pub(crate) fn metadata_mut(&mut self) -> &mut Metadata {} pub async fn metadata_cached(&mut self) -> Result<&Metadata> {} ``` Users can't know an object's mode after the list, so they have to send `metadata` every time they get an object: ```rust let o = op.object("path/to/dir/"); let mut obs = o.list().await?; // ObjectStream implements `futures::Stream` while let Some(o) = obs.next().await { let mut o = o?; // It's highly possible that OpenDAL already did metadata during list. // Use `Object::metadata_cached()` to get cached metadata at first. let meta = o.metadata_cached().await?; match meta.mode() { ObjectMode::FILE => { println!("Handling file") } ObjectMode::DIR => { println!("Handling dir like start a new list via meta.path()") } ObjectMode::Unknown => continue, } } ``` This behavior doesn't make sense as we already know the object's mode after the list. Introducing a separate `DirEntry` could reduce an extra call for metadata most of the time. ```rust let o = op.object("path/to/dir/"); let mut ds = o.list().await?; // ObjectStream implements `futures::Stream` while let Some(de) = ds.try_next().await { match de.mode() { ObjectMode::FILE => { println!("Handling file") } ObjectMode::DIR => { println!("Handling dir like start a new list via meta.path()") } ObjectMode::Unknown => continue, } } ``` # Guide-level explanation Within this RFC, `Object::list()` will return `DirStreamer` instead. ```rust pub trait DirStream: futures::Stream> + Unpin + Send {} pub type DirStreamer = Box; ``` `DirStreamer` will stream `DirEntry`, which carries information already known during the list. So we can: ```rust let id = de.id(); let path = de.path(); let name = de.name(); let mode = de.mode(); let meta = de.metadata().await?; ``` With `DirEntry` support, we can reduce an extra `metadata` call if we only want to know the object's mode: ```rust let o = op.object("path/to/dir/"); let mut ds = o.list().await?; // ObjectStream implements `futures::Stream` while let Some(de) = ds.try_next().await { match de.mode() { ObjectMode::FILE => { println!("Handling file") } ObjectMode::DIR => { println!("Handling dir like start a new list via meta.path()") } ObjectMode::Unknown => continue, } } ``` We can convert this `DirEntry` into `Object` without overhead: ```rust let o = de.into(); ``` # Reference-level explanation This proposal will introduce a new struct, `DirEntry`: ```rust struct DirEntry {} impl DirEntry { pub fn id() -> String {} pub fn path() -> &str {} pub fn name() -> &str {} pub fn mode() -> ObjectMode {} pub async fn metadata() -> ObjectMetadata {} } impl From for Object {} ``` And use `DirStream` to replace `ObjectStream`: ```rust pub trait DirStream: futures::Stream> + Unpin + Send {} pub type DirStreamer = Box; ``` With the addition of `DirEntry`, we will remove `meta` from `Object`: ```rust #[derive(Clone, Debug)] pub struct Object { acc: Arc, path: String, } ``` After this change, `Object` will become a thin wrapper of `Accessor` with path. And metadata related APIs like `metadata_ref()` and `metadata_mut()` will also be removed. # Drawbacks We are adding a new concept to our core logic. # Rationale and alternatives ## Rust fs API design Rust also provides abstractions like `File` and `DirEntry`: ```rust use std::fs; fn main() -> std::io::Result<()> { for entry in fs::read_dir(".")? { let dir = entry?; println!("{:?}", dir.path()); } Ok(()) } ``` Users can open a file with `entry.path()`. # Prior art None. # Unresolved questions None. # Future possibilities None. opendal-0.52.0/src/docs/rfcs/0409_accessor_capabilities.md000064400000000000000000000041141046102023000213170ustar 00000000000000- Proposal Name: `accessor_capabilities` - Start Date: 2022-06-29 - RFC PR: [apache/opendal#409](https://github.com/apache/opendal/pull/409) - Tracking Issue: [apache/opendal#410](https://github.com/apache/opendal/issues/410) # Summary Add support for accessor capabilities so that users can check if a given accessor is capable of a given ability. # Motivation Users of OpenDAL are requesting advanced features like the following: - [Support parallel upload object](https://github.com/apache/opendal/issues/256) - [Add presign url support](https://github.com/apache/opendal/issues/394) It's meaningful for OpenDAL to support them in a unified way. Of course, not all storage services have the same feature sets. OpenDAL needs to provide a way for users to check if a given accessor is capable of a given capability. # Guide-level explanation Users can check an `Accessor`'s capability via `Operator::metadata()`. ```rust let meta = op.metadata(); let _: bool = meta.can_presign(); let _: bool = meta.can_multipart(); ``` `Accessor` will return [`io::ErrorKind::Unsupported`](https://doc.rust-lang.org/stable/std/io/enum.ErrorKind.html#variant.Unsupported) for not supported operations instead of panic as `unimplemented()`. Users can check before operations or the `Unsupported` error kind after operations. # Reference-level explanation We will introduce a new enum called `AccessorCapability`, which includes `AccessorMetadata`. This enum is private and only accessible inside OpenDAL, so it's not part of our public API. We will expose the check API via `AccessorMetadata`: ```rust impl AccessorMetadata { pub fn can_presign(&self) -> bool { .. } pub fn can_multipart(&self) -> bool { .. } } ``` # Drawbacks None. # Rationale and alternatives None. # Prior art ## go-storage - [GSP-109: Redesign Features](https://github.com/beyondstorage/go-storage/blob/master/docs/rfcs/109-redesign-features.md) - [GSP-837: Support Feature Flag](https://github.com/beyondstorage/go-storage/blob/master/docs/rfcs/837-support-feature-flag.md) # Unresolved questions None. # Future possibilities None. opendal-0.52.0/src/docs/rfcs/0413_presign.md000064400000000000000000000131431046102023000164500ustar 00000000000000- Proposal Name: `presign` - Start Date: 2022-06-30 - RFC PR: [apache/opendal#0413](https://github.com/apache/opendal/pull/413) - Tracking Issue: [apache/opendal#394](https://github.com/apache/opendal/issues/394) # Summary Add presign support in OpenDAL so users can generate a pre-signed URL without leaking `serect_key`. # Motivation > By default, all S3 objects are private. Only the object owner has permission to access them. However, the object owner can optionally share objects with others by creating a presigned URL, using their own security credentials, to grant time-limited permission to download the objects. > > From [Sharing objects using presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html) We can use this presigned URL for: - Download the object within the expired time from a bucket directly - Upload content to the bucket on client-side Adding this feature in OpenDAL will make users' lives easier to generate presigned URLs across different storage services. The whole process would be: ```text ┌────────────┐ │ User ├─────────────────────┐ └──┬─────▲───┘ │ │ │ 4. Send Request to S3 Directly 1. Request Resource │ │ │ │ │ │ │ │ │ │ ▼ │ 3. Return Request ┌────────────┐ │ │ │ │ │ │ │ S3 │ ┌──▼─────┴───┐ │ │ │ │ └────────────┘ │ App │ │ ┌────────┐ │ │ │ OpenDAL│ │ │ ├────────┤ │ └─┴──┼─────┴─┘ │ ▲ └──────┘ 2. Generate Request ``` # Guide-level explanation With this feature, our users can: ## Generate presigned URL for downloading ```rust let req = op.presign_read("path/to/file")?; // req.method: GET // req.url: https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature= ``` Users can download this object directly from the s3 bucket. For example: ```shell curl -O test.txt ``` ## Generate presigned URL for uploading ```rust let req = op.presign_write("path/to/file")?; // req.method: PUT // req.url: https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature= ``` Users can upload content directly to the s3 bucket. For example: ```shell curl -X PUT -T "/tmp/test.txt" ``` # Reference-level explanation `Accessor` will add a new API `presign`: ```rust pub trait Accessor { fn presign(&self, args: &OpPresign) -> Result {..} } ``` `presign` accepts `OpPresign` and returns `Result`: ```rust struct OpPresign { path: String, op: Operation, expire: time::Duration, } struct PresignedRequest {} impl PresignedRequest { pub fn method(&self) -> &http::Method {..} pub fn url(&self) -> &http::Uri {..} } ``` We are building a new struct to avoid leaking underlying implementations like `hyper::Request` to users. This feature will be a new capability in `AccessorCapability` as described in [RFC-0409: Accessor Capabilities](./0409-accessor-capabilities.md) Based on `Accessor::presign`, we will export public APIs in `Operator`: ```rust impl Operator { fn presign_read(&self, path: &str) -> Result {} fn presign_write(&self, path: &str) -> Result {} } ``` Although it's possible to generate URLs for `create`, `delete`, `stat`, and `list`, there are no obvious use-cases. So we will not add them to this proposal. # Drawbacks None. # Rationale and alternatives ## Query Sign Support Status - s3: [Authenticating Requests: Using Query Parameters (AWS Signature Version 4)](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html) - azblob: [Delegate access with a shared access signature](https://docs.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature) - gcs: [Signed URLs](https://cloud.google.com/storage/docs/access-control/signed-urls) (Only for XML API) # Prior art ## awscli presign AWS CLI has native presign support ```shell > aws s3 presign s3://DOC-EXAMPLE-BUCKET/test2.txt https://DOC-EXAMPLE-BUCKET.s3.us-west-2.amazonaws.com/key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAEXAMPLE123456789%2F20210621%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210621T041609Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=EXAMBLE1234494d5fba3fed607f98018e1dfc62e2529ae96d844123456 ``` Refer to [AWS CLI Command Reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html) for more information. # Unresolved questions None. # Future possibilities - Add `stat`/`list`/`delete` support opendal-0.52.0/src/docs/rfcs/0423_command_line_interface.md000064400000000000000000000205131046102023000214460ustar 00000000000000- Proposal Name: `command_line_interface` - Start Date: 2022-07-08 - RFC PR: [apache/opendal#423](https://github.com/apache/opendal/pull/423) - Tracking Issue: [apache/opendal#422](https://github.com/apache/opendal/issues/422) # Summary Add command line interface for OpenDAL. # Motivation > **Q**: There are so many cli out there, why we still need a cli for OpenDAL? > > **A**: Because there are so many cli out there. To manipulate our date store in different could service, we need to install different clis: - [`aws-cli`]/[`s3cmd`]/... for AWS (S3) - [`azcopy`] for Azure Storage Service - [`gcloud`] for Google Cloud Those clis provide native and seamless experiences for their own products but also lock us and our data. However, for 80% cases, we just want to do simple jobs like `cp`, `mv` and `rm`. It's boring to figure out how to use them: - `aws --endpoint-url http://127.0.0.1:9900/ s3 cp data s3://testbucket/data --recursive` - `azcopy copy 'C:\myDirectory' 'https://mystorageaccount.blob.core.windows.net/mycontainer' --recursive` - `gsutil cp data gs://testbucket/` Can we use them in the same way? Can we let the data flow freely? Let's look back OpenDAL's slogan: **Open Data Access Layer that connect the whole world together** This is a natural extension for OpenDAL: providing a command line interface! # Guide-level explanation OpenDAL will provide a new cli called: `oli`. It's a shortcut of `OpenDAL Command Line Interface`. Users can install this cli via: ```shell cargo install oli ``` Or using they favourite package management: ```shell # Archlinux pacman -S oli # Debian / Ubuntu apt install oli # Rocky Linux / Fedora dnf install oli # macOS brew install oli ``` With `oli`, users can: - Upload files to s3: `oli cp books.csv s3://bucket/books.csv` - Download files from azblob: `oli cp azblob://bucket/books.csv /tmp/books.csv` - Move data between storage services: `oli mv s3://bucket/dir azblob://bucket/dir` - Delete all files: `oli rm -rf s3://bucket` `oli` also provide alias to make cloud data manipulating even natural: - `ocp` for `oli cp` - `ols` for `oli ls` - `omv` for `oli mv` - `orm` for `oli rm` - `ostat` for `oli stat` `oli` will provide profile management so users don't need to provide credential every time: - `oli profile add my_s3 --bucket test --access-key-id=example --secret-access-key=example` - `ocp my_s3://dir /tmp/dir` # Reference-level explanation `oli` will be a separate crate apart from `opendal` so we will not pollute the dependencies of `opendal`. But `oli` will be releases at the same time with the same version of `opendal`. That means `oli` will always use the same (latest) version of opendal. Most operations of `oli` should be trivial, we will propose new RFCs if requiring big changes. `oli` won't keep configuration. All config will go through environment, for example: - `OIL_COLOR=always` - `OIL_CONCURRENCY=16` Besides, `oil` will read profile from env like `cargo`: - `OIL_PROFILE_TEST_TYPE=s3` - `OIL_PROFILE_TEST_ENDPOINT=http://127.0.0.1:1090` - `OIL_PROFILE_TEST_BUCKET=test_bucket` - `OIL_PROFILE_TEST_ACCESS_KEPT_ID=access_key_id` - `OIL_PROFILE_TEST_SECRET_ACCESS_KEY=secret_access_key` With those environments, we can: ```shell ocp path/to/dir test://test/to/dir ``` # Drawbacks None # Rationale and alternatives ## s3cmd [s3cmd](https://s3tools.org/s3cmd) is a command line s3 client for Linux and Mac. ```shell Usage: s3cmd [options] COMMAND [parameters] S3cmd is a tool for managing objects in Amazon S3 storage. It allows for making and removing "buckets" and uploading, downloading and removing "objects" from these buckets. Commands: Make bucket s3cmd mb s3://BUCKET Remove bucket s3cmd rb s3://BUCKET List objects or buckets s3cmd ls [s3://BUCKET[/PREFIX]] List all object in all buckets s3cmd la Put file into bucket s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX] Get file from bucket s3cmd get s3://BUCKET/OBJECT LOCAL_FILE Delete file from bucket s3cmd del s3://BUCKET/OBJECT Delete file from bucket (alias for del) s3cmd rm s3://BUCKET/OBJECT Restore file from Glacier storage s3cmd restore s3://BUCKET/OBJECT Synchronize a directory tree to S3 (checks files freshness using size and md5 checksum, unless overridden by options, see below) s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR Disk usage by buckets s3cmd du [s3://BUCKET[/PREFIX]] Get various information about Buckets or Files s3cmd info s3://BUCKET[/OBJECT] Copy object s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2] Modify object metadata s3cmd modify s3://BUCKET1/OBJECT Move object s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2] Modify Access control list for Bucket or Files s3cmd setacl s3://BUCKET[/OBJECT] Modify Bucket Policy s3cmd setpolicy FILE s3://BUCKET Delete Bucket Policy s3cmd delpolicy s3://BUCKET Modify Bucket CORS s3cmd setcors FILE s3://BUCKET Delete Bucket CORS s3cmd delcors s3://BUCKET Modify Bucket Requester Pays policy s3cmd payer s3://BUCKET Show multipart uploads s3cmd multipart s3://BUCKET [Id] Abort a multipart upload s3cmd abortmp s3://BUCKET/OBJECT Id List parts of a multipart upload s3cmd listmp s3://BUCKET/OBJECT Id Enable/disable bucket access logging s3cmd accesslog s3://BUCKET Sign arbitrary string using the secret key s3cmd sign STRING-TO-SIGN Sign an S3 URL to provide limited public access with expiry s3cmd signurl s3://BUCKET/OBJECT Fix invalid file names in a bucket s3cmd fixbucket s3://BUCKET[/PREFIX] Create Website from bucket s3cmd ws-create s3://BUCKET Delete Website s3cmd ws-delete s3://BUCKET Info about Website s3cmd ws-info s3://BUCKET Set or delete expiration rule for the bucket s3cmd expire s3://BUCKET Upload a lifecycle policy for the bucket s3cmd setlifecycle FILE s3://BUCKET Get a lifecycle policy for the bucket s3cmd getlifecycle s3://BUCKET Remove a lifecycle policy for the bucket s3cmd dellifecycle s3://BUCKET List CloudFront distribution points s3cmd cflist Display CloudFront distribution point parameters s3cmd cfinfo [cf://DIST_ID] Create CloudFront distribution point s3cmd cfcreate s3://BUCKET Delete CloudFront distribution point s3cmd cfdelete cf://DIST_ID Change CloudFront distribution point parameters s3cmd cfmodify cf://DIST_ID Display CloudFront invalidation request(s) status s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID] ``` ## aws-cli [aws-cli](https://aws.amazon.com/cli/) is the official cli provided by AWS. ```shell $ aws s3 ls s3://mybucket LastWriteTime Length Name ------------ ------ ---- PRE myfolder/ 2013-09-03 10:00:00 1234 myfile.txt $ aws s3 cp myfolder s3://mybucket/myfolder --recursive upload: myfolder/file1.txt to s3://mybucket/myfolder/file1.txt upload: myfolder/subfolder/file1.txt to s3://mybucket/myfolder/subfolder/file1.txt $ aws s3 sync myfolder s3://mybucket/myfolder --exclude *.tmp upload: myfolder/newfile.txt to s3://mybucket/myfolder/newfile.txt ``` ## azcopy [azcopy](https://github.com/Azure/azure-storage-azcopy) is the new Azure Storage data transfer utility. ```shell azcopy copy 'C:\myDirectory\myTextFile.txt' 'https://mystorageaccount.blob.core.windows.net/mycontainer/myTextFile.txt' azcopy copy 'https://mystorageaccount.blob.core.windows.net/mycontainer/myTextFile.txt' 'C:\myDirectory\myTextFile.txt' azcopy sync 'C:\myDirectory' 'https://mystorageaccount.blob.core.windows.net/mycontainer' --recursive ``` ## gsutil [gsutil](https://cloud.google.com/storage/docs/gsutil) is a Python application that lets you access Cloud Storage from the command line. ```shell gsutil cp [OPTION]... src_url dst_url gsutil cp [OPTION]... src_url... dst_url gsutil cp [OPTION]... -I dst_url gsutil mv [-p] src_url dst_url gsutil mv [-p] src_url... dst_url gsutil mv [-p] -I dst_url gsutil rm [-f] [-r] url... gsutil rm [-f] [-r] -I ``` # Unresolved questions None. # Future possibilities None. [`aws-cli`]: https://github.com/aws/aws-cli [`s3cmd`]: https://s3tools.org/s3cmd [`azcopy`]: https://github.com/Azure/azure-storage-azcopy [`gcloud`]: https://cloud.google.com/sdk/docs/install opendal-0.52.0/src/docs/rfcs/0429_init_from_iter.md000064400000000000000000000060011046102023000200140ustar 00000000000000- Proposal Name: `init_from_iter` - Start Date: 2022-07-10 - RFC PR: [apache/opendal#429](https://github.com/apache/opendal/pull/429) - Tracking Issue: [apache/opendal#430](https://github.com/apache/opendal/issues/430) # Summary Allow initializing opendal operators from an iterator. # Motivation To init OpenDAL operators, users have to init an accessor first. ```rust let root = &env::var("OPENDAL_S3_ROOT").unwrap_or_else(|_| "/".to_string()); let root = format!("/{}/{}", root, uuid::Uuid::new_v4()); let mut builder = opedal::services::s3::Backend::build(); builder.root(&root); builder.bucket(&env::var("OPENDAL_S3_BUCKET").expect("OPENDAL_S3_BUCKET must set")); builder.endpoint(&env::var("OPENDAL_S3_ENDPOINT").unwrap_or_default()); builder.access_key_id(&env::var("OPENDAL_S3_ACCESS_KEY_ID").unwrap_or_default()); builder.secret_access_key(&env::var("OPENDAL_S3_SECRET_ACCESS_KEY").unwrap_or_default()); builder .server_side_encryption(&env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION").unwrap_or_default()); builder.server_side_encryption_customer_algorithm( &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM").unwrap_or_default(), ); builder.server_side_encryption_customer_key( &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY").unwrap_or_default(), ); builder.server_side_encryption_customer_key_md5( &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5").unwrap_or_default(), ); builder.server_side_encryption_aws_kms_key_id( &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID").unwrap_or_default(), ); if env::var("OPENDAL_S3_ENABLE_VIRTUAL_HOST_STYLE").unwrap_or_default() == "on" { builder.enable_virtual_host_style(); } Ok(Some(builder.finish().await?)) ``` We can simplify this logic if opendal has its native `from_iter` support. # Guide-level explanation Users can init an operator like the following: ```rust // OPENDAL_S3_BUCKET = // OPENDAL_S3_ENDPOINT = let op = Operator::from_env(Scheme::S3)?; ``` Or from a prefixed env: ```rust // OIL_PROFILE__S3_BUCKET = // OIL_PROFILE__S3_ENDPOINT = let op = Operator::from_env(Scheme::S3, "OIL_PROFILE_")?; ``` Also, we call the underlying function directly: ```rust // var it: impl Iterator let op = Operator::from_iter(Scheme::S3, it)?; ``` # Reference-level explanation Internally, every service's backend will implement the following functions: ```rust fn from_iter(it: impl Iterator) -> Backend {} ``` Note: it's not a public API of `Accessor`, and it will never be. Instead, we will use this function inside the crate to keep the ability to refactor or even remove it. # Drawbacks None. # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities ## Connection string It sounds a good idea to implement something like: ```rust let op = Operator::open("s3://bucket?region=test")? ``` But there are no valid use cases. Let's implement this in the future if needed. opendal-0.52.0/src/docs/rfcs/0438_multipart.md000064400000000000000000000102711046102023000170300ustar 00000000000000- Proposal Name: `multipart` - Start Date: 2022-07-11 - RFC PR: [apache/opendal#438](https://github.com/apache/opendal/pull/438) - Tracking Issue: [apache/opendal#439](https://github.com/apache/opendal/issues/439) # Summary Add multipart support in OpenDAL. # Motivation [Multipart Upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) APIs are widely used in object storage services to upload large files concurrently and resumable. A successful multipart upload includes the following steps: - `CreateMultipartUpload`: Start a new multipart upload. - `UploadPart`: Upload a single part with the previously uploaded id. - `CompleteMultipartUpload`: Complete a multipart upload to get a regular object. To cancel a multipart upload, users need to call `AbortMultipartUpload`. Apart from those APIs, most object services also provide a list API to get the current multipart uploads status: - `ListMultipartUploads`: List current ongoing multipart uploads - `ListParts`: List already uploaded parts. Before `CompleteMultipartUpload` has been called, users can't read already uploaded parts. After `CompleteMultipartUpload` or `AbortMultipartUpload` has been called, all uploaded parts will be removed. Object storage services commonly allow 10000 parts, and every part will allow up to 5 GiB. This way, users can upload a file up to 48.8 TiB. OpenDAL users can upload objects larger than 5 GiB via supporting multipart uploads. # Guide-level explanation Users can start a multipart upload via: ```rust let mp = op.object("path/to/file").create_multipart().await?; ``` Or build a multipart via already known upload id: ```rust let mp = op.object("path/to/file").into_multipart(""); ``` With `Multipart`, we can upload a new part: ```rust let part = mp.write(part_number, content).await?; ``` After all parts have been uploaded, we can finish this upload: ```rust let _ = mp.complete(parts).await?; ``` Or, we can abort already uploaded parts: ```rust let _ = mp.abort().await?; ``` # Reference-level explanation `Accessor` will add the following APIs: ```rust pub trait Accessor: Send + Sync + Debug { async fn create_multipart(&self, args: &OpCreateMultipart) -> Result { let _ = args; unimplemented!() } async fn write_multipart(&self, args: &OpWriteMultipart) -> Result { let _ = args; unimplemented!() } async fn complete_multipart(&self, args: &OpCompleteMultipart) -> Result<()> { let _ = args; unimplemented!() } async fn abort_multipart(&self, args: &OpAbortMultipart) -> Result<()> { let _ = args; unimplemented!() } } ``` While closing a `PartWriter`, a `Part` will be generated. `Operator` will build APIs based on `Accessor`: ```rust impl Object { async fn create_multipart(&self) -> Result {} fn into_multipart(&self, upload_id: &str) -> Multipart {} } impl Multipart { async fn write(&self, part_number: usize, bs: impl AsRef<[u8]>) -> Result {} async fn writer(&self, part_number: usize, size: u64) -> Result {} async fn complete(&self, ps: &[Part]) -> Result<()> {} async fn abort(&self) -> Result<()> {} } ``` # Drawbacks None. # Rationale and alternatives ## Why not add new object modes? It seems natural to add a new object mode like `multipart`. ```rust pub enum ObjectMode { FILE, DIR, MULTIPART, Unknown, } ``` However, to make this work, we need big API breaks that introduce `mode` in Object. And we need to change every API call to accept `mode` as args. For example: ```rust let _ = op.object("path/to/dir/").list(ObjectMODE::MULTIPART); let _ = op.object("path/to/file").stat(ObjectMODE::MULTIPART) ``` ## Why not split Object into File and Dir? We can split `Object` into `File` and `Dir` to avoid requiring `mode` in API. There is a vast API breakage too. # Prior art None. # Unresolved questions None. # Future possibilities ## Support list multipart uploads We can support listing multipart uploads to list ongoing multipart uploads so we can resume an upload or abort them. ## Support list part We can support listing parts to list already uploaded parts for an upload. opendal-0.52.0/src/docs/rfcs/0443_gateway.md000064400000000000000000000034611046102023000164470ustar 00000000000000- Proposal Name: `gateway` - Start Date: 2022-07-18 - RFC PR: [apache/opendal#443](https://github.com/apache/opendal/pull/443) - Tracking Issue: [apache/opendal#444](https://github.com/apache/opendal/issues/444) # Summary Add Gateway for OpenDAL. # Motivation Our users want features like [S3 Proxy](https://github.com/gaul/s3proxy) and [minio Gateway](https://blog.min.io/deprecation-of-the-minio-gateway/) so that they can access all their data in the same way. By providing a native gateway, we can empower users to access different storage in the same API. # Guide-level explanation OpenDAL will provide a new binary called: `oay`. It's a shortcut of `OpenDAL Gateway`. Uses can install this binary via: ```shell cargo install oay ``` Or using they favourite package management: ```shell # Archlinux pacman -S oay # Debian / Ubuntu apt install oay # Rocky Linux / Fedora dnf install oay # macOS brew install oay ``` With `oay`, users can: - Serve `fs` backend with S3 compatible API. - Serve `s3` backend with Azblob API - Serve as a s3 signing services # Reference-level explanation `oay` will be a separate crate apart from `opendal` so we will not pollute the dependencies of `opendal`. But `oay` will be releases at the same time with the same version of `opendal`. That means `oay` will always use the same (latest) version of opendal. Most operations of `oay` should be trivial, we will propose new RFCs if requiring big changes. `oay` won't keep configuration. All config will go through environment. # Drawbacks None # Rationale and alternatives None # Prior art - [S3 Proxy](https://github.com/gaul/s3proxy) - [minio Gateway](https://blog.min.io/deprecation-of-the-minio-gateway/) - [oxyno-zeta/s3-proxy](https://github.com/oxyno-zeta/s3-proxy) # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/0501_new_builder.md000064400000000000000000000055061046102023000173020ustar 00000000000000- Proposal Name: `new_builder` - Start Date: 2022-08-03 - RFC PR: [apache/opendal#501](https://github.com/apache/opendal/pull/501) - Tracking Issue: [apache/opendal#502](https://github.com/apache/opendal/issues/502) # Summary Allow users to build services without async. # Motivation Most services share a similar builder API to construct backends. ```rust impl Builder { pub async fn finish(&mut self) -> Result> {} } ``` We have `async` here so that every user who wants to build services backend must go through an async runtime. Even for `memory` backend: ```rust impl Builder { /// Consume builder to build a memory backend. pub async fn finish(&mut self) -> Result> { Ok(Arc::new(Backend::default())) } } ``` Only `s3` services need to call async functions `detect_region` to get the correct region. So, we can provide blocking `Builder` APIs and move async-related logic out for users to call out. This way, our users can build services without playing with async runtime. # Guide-level explanation After this change, all our services builder will add a new API: ```rust impl Builder { pub fn build(&mut self) -> Result {} } ``` Along with this change, our `Operator` will accept `impl Accessor + 'static` instead of `Arc` anymore: ```rust impl Operator { pub fn new(accessor: impl Accessor + 'static) -> Self {} } ``` Also, we will implement `From` for `Operator`: ```rust impl From for Operator where A: Accessor + 'static, { fn from(acc: A) -> Self { Operator::newx(acc) } } ``` We can initiate an operator quicker: ```diff - let op: Operator = Operator::new(fs::Backend::build().finish().await?); + let op: Operator = fs::Builder::new().build()?.into(); ``` # Reference-level explanation We will add the following APIs: - All builders will add `build(&mut self) -> Result` - `impl From for Operator where A: Accessor + 'static` We will deprecate the following APIs: - All builders `finish()` API (should be replaced by `build()`) - All services `build()` API (should be replaced by `Builder::new()` or `Builder::default()`) We will change the following APIs: - Operator: `new(accessor: Arc)` -> `fn new(accessor: impl dyn Accessor + 'static)` - Operator: `async fn from_iter()` -> `fn from_iter()` - Operator: `async fn from_env()` -> `fn from_env()` Most services will work the same, except for `s3`: `s3` depends on `detect_region` to check the correct region if the user doesn't input. After this change, `s3::Builder.build()` will return error if `region` is missing. Users should call `detect_region` by themselves to get the region. # Drawbacks None. # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities None. opendal-0.52.0/src/docs/rfcs/0554_write_refactor.md000064400000000000000000000042361046102023000200310ustar 00000000000000- Proposal Name: `write_refactor` - Start Date: 2022-08-22 - RFC PR: [apache/opendal#554](https://github.com/apache/opendal/pull/554) - Tracking Issue: [apache/opendal#555](https://github.com/apache/opendal/issues/555) # Summary Refactor `write` operation to accept a `BytesReader` instead. # Motivation To simulate the similar operation like POSIX fs, OpenDAL returns `BytesWriter` for users to write, flush and close: ```rust pub trait Accessor { async fn write(&self, args: &OpWrite) -> Result {} } ``` `Operator` builds the high level APIs upon this: ```rust impl Object { pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {} pub async fn writer(&self, size: u64) -> Result {} } ``` However, we are meeting the following problems: - Performance: HTTP body channel is mush slower than read from Reader directly. - Complicity: Service implementer have to deal with APIs like `new_http_channel`. - Extensibility: Current design can't be extended to multipart APIs. # Guide-level explanation Underlying `write` implementations will be replaced by: ```rust pub trait Accessor { async fn write(&self, args: &OpWrite, r: BytesReader) -> Result {} } ``` Existing API will have no changes, and we will add a new API: ```rust impl Object { pub async fn write_from(&self, size: u64, r: impl BytesRead) -> Result {} } ``` # Reference-level explanation `Accessor`'s `write` API will be changed to accept a `BytesReader`: ```rust pub trait Accessor { async fn write(&self, args: &OpWrite, r: BytesReader) -> Result {} } ``` We will provide `Operator::writer` based on this new API instead. [RFC-0438: Multipart](./0438-multipart.md) will also be updated to: ```rust pub trait Accessor { async fn write_multipart(&self, args: &OpWriteMultipart, r: BytesReader) -> Result {} } ``` In this way, we don't need to introduce a `PartWriter`. # Drawbacks ## Layer API breakage This change will introduce break changes to layers. # Rationale and alternatives None. # Prior art - [RFC-0191: Async Streaming IO](./0191-async-streaming-io.md) # Unresolved questions None. # Future possibilities None. opendal-0.52.0/src/docs/rfcs/0561_list_metadata_reuse.md000064400000000000000000000145231046102023000210260ustar 00000000000000- Proposal Name: `list_metadata_reuse` - Start Date: 2022-08-23 - RFC PR: [apache/opendal#561](https://github.com/apache/opendal/pull/561) - Tracking Issue: [apache/opendal#570](https://github.com/apache/opendal/pull/570) # Summary Reuse metadata returned during listing, by extending `DirEntry` with some metadata fields. # Motivation Users may expect to browse metadata of some directories' child files and directories. Using `walk()` of `BatchOperator` seems to be an ideal way to complete this job. Thus, they start iterating on it, but soon they realized the `DirEntry`, could only offer the name (or path, more precisely) and access mode of the object, and it's not enough. So they have to call `metadata()` for each name they extracted from the iterator. The final example looks like: ```rust let op = Operator::from_env(Scheme::Gcs)?.batch(); // here is a network request let mut dir_stream = op.walk("/dir/to/walk")?; while let Some(Ok(file)) = dir_stream.next().await { let path = file.path(); // here is another network request let size = file.metadata().await?.content_length(); println!("size of file {} is {}B", path, size); } ``` But...wait! many storage-services returns object metadata when listing, like HDFS, AWS and GCS. The rust standard library returns metadata when listing local file systems, too. In the previous versions of OpenDAL those fields were just get ignored. This wastes users' time on requesting on metadata. # Guide-level explanation The loop in main will be changed to the following code with this RFC: ```rust while let Some(Ok(file)) = dir_stream.next().await { let size = if let Some(len) = file.content_length() { len } else { file.metadata().await?.content_length(); }; let name = file.path(); println!("size of file {} is {}B", path, size); } ``` # Reference-level explanation Extend `DirEntry` with metadata fields: ```rust pub struct DirEntry { acc: Arc, mode: ObjectMode, path: String, // newly add metadata fields content_length: Option, // size of file content_md5: Option, last_modified: Option, } impl DirEntry { pub fn content_length(&self) -> Option { self.content_length } pub fn last_modified(&self) -> Option { self.last_modified } pub fn content_md5(&self) -> Option { self.content_md5 } } ``` For all services that supplies metadata during listing, like AWS, GCS and HDFS. Those optional fields will be filled up; Meanwhile for those services doesn't return metadata during listing, like in memory storages, just left them as `None`. As you can see, for those services returning metadata when listing, the operation of listing metadata will save many unnecessary requests. # Drawbacks Add complexity to `DirEntry`. To use the improved features of `DirEntry`, users have to explicitly check the existence of metadata fields. The size of `DirEntry` increased from 40 bytes to 80 bytes, a 100% percent growth requires more memory. # Rational and alternatives The largest drawback of performance usually comes from network or hard disk operations. By letting `DirEntry` storing some metadata, many redundant requests could be avoided. ## Embed a Structure Containing Metadata Define a `MetaLite` structure containing some metadata fields, and embed it in `DirEntry` ```rust struct MetaLite { pub content_length: u64, // size of file pub content_md5: String, pub last_modified: OffsetDateTime, } pub struct DirEntry { acc: Arc, mode: ObjectMode, path: String, // newly add metadata struct metadata: Option, } impl DirEntry { // get size of file pub fn content_length(&self) -> Option { self.metadata.as_ref().map(|m| m.content_length) } // get the last modified time pub fn last_modified(&self) -> Option { self.metadata.as_ref().map(|m| m.last_modified) } // get md5 message digest pub fn content_md5(&self) -> Option { self.metadata.as_ref().map(|m| m.content_md5) } } ``` The existence of those newly added metadata fields is highly correlated. If one field does not exist, the others neither. By wrapping them together in an embedded structure, 8 bytes of space for each `DirEntry` object could be saved. In the future, more metadata fields may be added to `DirEntry`, then a lot more space could be saved. This approach could be slower because some intermediate functions are involved. But it's worth sacrificing rarely used features' performance to save memory. ## Embed a `ObjectMetadata` into `DirEntry` - Embed a `ObjectMetadata` struct into `DirEntry` - Remove the `ObjectMode` field in `DirEntry` - Change `ObjectMetadata`'s `content_length` field's type to `Option`. ```rust pub struct DirEntry { acc: Arc, // - mode: ObjectMode, removed path: String, // newly add metadata struct metadata: ObjectMetadata, } impl DirEntry { pub fn mode(&self) -> ObjectMode { self.metadata.mode() } pub fn content_length(&self) -> Option { self.metadata.content_length() } pub fn content_md5(&self) -> Option<&str> { self.metadata.content_md5() } // other metadata getters... } ``` In the degree of memory layout, it's the same as proposed way in this RFC. This approach offers more metadata fields and fewer changes to code. # Prior art None. # Unresolved questions None. # Future possibilities ## Switch to Alternative Implement Approaches As the growing of metadata fields, someday the alternatives could be better. And other RFCs will be raised then. ## More Fields Add more metadata fields to DirEntry, like: - accessed: the last access timestamp of object ## Simplified Get Users have to explicitly check if those metadata fields actual present in the DirEntry. This may be done inside the getter itself. ```rust let path = file.path(); // if content_length is not exist // this getter will automatically fetch from the storage service. let size = file.content_length().await?; // the previous getter can cache metadata fetched from service // so this function could return instantly. let md5 = file.content_md5().await?; println!("size of file {} is {}B, md5 outcome of file is {}", path, size, md5); ``` opendal-0.52.0/src/docs/rfcs/0599_blocking_api.md000064400000000000000000000112061046102023000174370ustar 00000000000000- Proposal Name: `blocking_api` - Start Date: 2022-08-30 - RFC PR: [apache/opendal#599](https://github.com/apache/opendal/pull/599) - Tracking Issue: [apache/opendal#601](https://github.com/apache/opendal/issues/601) # Summary We are adding a blocking API for OpenDAL. # Motivation Blocking API is the most requested feature inside the OpenDAL community: [Opendal support sync read/write API](https://github.com/apache/opendal/discussions/68) Our users want blocking API for: - Higher performance for local IO - Using OpenDAL in a non-async environment However, supporting sync and async API in current Rust is a painful job, especially for an IO library like OpenDAL. For example: ```rust impl Object { pub async fn reader(&self) -> Result {} } ``` Supporting blocking API doesn't mean removing the `async` from the function. We should also handle the returning `Reader`: ```rust impl Object { pub fn reader(&self) -> Result {} } ``` Until now, I still don't know how to handle them correctly. But we need to have a start: not perfect, but enough for our users to have a try. So this RFC is an **experiment** try to introduce blocking API support. I expect the OpenDAL community will evaluate those APIs and keep improving them. And finally, we will pick up the best one for stabilizing. # Guide-level explanation With this RFC, we can call blocking API with the `blocking_` prefix: ```rust fn main() -> Result<()> { // Init Operator let op = Operator::from_env(Scheme::Fs)?; // Create object handler. let o = op.object("test_file"); // Write data info object; o.blocking_write("Hello, World!")?; // Read data from object; let bs = o.blocking_read()?; // Read range from the object; let bs = o.blocking_range_read(1..=11)?; // Get the object's path let name = o.name(); let path = o.path(); // Fetch more meta about the object. let meta = o.blocking_metadata()?; let mode = meta.mode(); let length = meta.content_length(); let content_md5 = meta.content_md5(); let etag = meta.etag(); // Delete object. o.blocking_delete()?; // List dir object. let o = op.object("test_dir/"); let mut ds = o.blocking_list()?; while let Some(entry) = ds.try_next()? { let path = entry.path(); let mode = entry.mode(); } Ok(()) } ``` All async public APIs of `Object` and `Operator` will have a sync version with `blocking_` prefix. And they will share precisely the same semantics. The differences are: - They will be executed and blocked on the current thread. - Input and output's `Reader` will become the blocking version like `std::io::Read`. - Output's `DirStreamer` will become the blocking version like `Iterator`. Thanks to [RFC-0501: New Builder](./0501-new-builder.md), all our builder-related APIs have been transformed into blocking APIs, so we don't change our initiation logic. # Reference-level explanation Under the hood, we will add the following APIs in `Accessor`: ```rust trait Accessor { fn blocking_create(&self, args: &OpCreate) -> Result<()>; fn blocking_read(&self, args: &OpRead) -> Result; fn blocking_write(&self, args: &OpWrite, r: BlockingBytesReader) -> Result; fn blocking_stat(&self, args: &OpStat) -> Result; fn blocking_delete(&self, args: &OpDelete) -> Result<()>; fn blocking_list(&self, args: &OpList) -> Result; } ``` Notes: - `BlockingBytesReader` is a boxed `std::io::Read`. - All blocking operations are happening on the current thread. - Blocking operation is implemented natively, no `futures::block_on`. # Drawbacks ## Two sets of APIs This RFC will add a new set of APIs, adding complicity for OpenDAL. And users may misuse them. For example: using `blocking_read` in an async context could block the entire thread. # Rationale and alternatives ## Use features to switch `async` and `sync` Some crates provide features to switch the `async` and `sync` versions of API. In this way: - We can't provide two kinds of API at the same time. - Users must decide to use `async` or `sync` at compile time. ## Use blocking IO functions in local fs services > Can we use blocking IO functions in local fs services to implement Accessor's asynchronous functions directly? What is the drawback of our current non-blocking API? We can't run blocking IO functions inside the `async` context. We need to let the local thread pool execute them and use `mio` to listen to the events. If we do so, congrats, we are building `tokio::fs` again! # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/0623_redis_service.md000064400000000000000000000235031046102023000176330ustar 00000000000000- Proposal Name: `redis_service` - Start Date: 2022-08-31 - RFC PR: [apache/opendal#0623](https://github.com/apache/opendal/pull/0623) - Tracking Issue: [apache/opendal#641](https://github.com/apache/opendal/issues/0641) # Summary Use [redis](https://redis.io) as a service of OpenDAL. # Motivation Redis is a fast, in-memory cache with persistent and distributed storage functionalities. It's widely used in production. Users also demand more backend support. Redis is a good candidate. # Guide-level explanation Users only need to provide the network address, username and password to create a Redis Operator. Then everything will act as other operators do. ```rust use opendal::services::redis::Builder; use opendal::Operator; let builder = Builder::default(); // set the endpoint of redis server builder.endpoint("tcps://domain.to.redis:2333"); // set the username of redis builder.username("example"); // set the password builder.password(&std::env::var("OPENDAL_REDIS_PASSWORD").expect("env OPENDAL_REDIS_PASSWORD not set")); // root path builder.root("/example/"); let op = Operator::new( builder.build()? // services::redis::Backend ); // congratulations, you can use `op` just like any other operators! ``` # Reference-level explanation To ease the development, [redis-rs](https://crates.io/crates/redis) will be used. Redis offers a key-value view, so the path of files could be represented as the key of the key-value pair. The content of file will be represented directly as `String`, and metadata will be encoded as [`bincode`](https://github.com/bincode-org/bincode.git) before storing as `String`. ```text +------------------------------------------+ |Object: /home/monika/ | | | SET |child: Key: v0:k:/home/monika/ -+---------> 1) /home/monika/poem0.txt | | |/* directory has no content */ | | | |metadata: Key: v0:m:/home/monika/ | +------------------------------------------+ +------------------------------------------+ |Object: /home/monika/poem0.txt | | | | /* file has no children */ | | | |content: Key: v0:c:/home/monika/poem0.txt-+--+ | | | |metadata: Key: v0:m:/home/monika/poem0.txt| | | | | | +--+---------------------------------------+ | | v +> STRING STRING +----------------------+ +--------------------+ |\x00\x00\x00\x00\xe6\a| |1JU5T3MON1K413097321| |\x00\x00\xf8\x00\a4)!V| |&JU5$T!M0N1K4$%#@#$%| |\x81&\x00\x00\x00Q\x00| |3231J)U_ST#MONIKA@#$| | ... | |1557(m0N1ka3just4M | +----------------------+ | ... | +--------------------+ ``` The [redis-rs](https://crates.io/crates/redis)'s high level APIs is preferred. ```rust const VERSION: usize = 0; /// meta_key will produce the key to object's metadata /// "/path/to/object/" -> "v{VERSION}:m:/path/to/object" fn meta_key(path: &str) -> String { format!("v{}:m:{}", VERSION, path) } /// content_key will produce the key to object's content /// "/path/to/object/" -> "v{VERSION}:c:/path/to/object" fn content_key(path: &str) -> String { format!("v{}:c:{}", VERSION, path) } fn connect() -> Result<()> { let client = redis::Client::open("redis://localhost:6379")?; let con = client.get_async_connection()?; } ``` ## Forward Compatibility All keys used will have a `v0` prefix, indicating it's using the very first version of `OpenDAL` `Redis` API. When there are changes to the layout, like refactoring the layout of storage, the version number should be updated, too. Further versions should take the compatibility with former implementations into consideration. ## Create File If user is creating a file with root `/home/monika/`, and relative path `poem0.txt`. ```rust // mode: ObjectMode // path: relative path string let path = get_abs_path(path); // /home/monika/ <> /poem.txt -> /home/monika/poem.txt let m_path = meta_key(path); // path of metadata let c_path = content_key(path); // path of content let last_modified = OffsetDatetime::now_utc().to_string(); let mut meta = ObjectMeta::default(); meta.set_mode(ObjectMode::FILE); meta.set_last_modified(OffsetDatetime::now_utc()); let encoded = bincode::encode_to_vec(meta)?; con.set(c_path, "".to_string())?; con.set(m_path, encoded.as_slice())?; ``` This will create two key-value pair. For object content, its key is `v0:c:/home/monika/poem0.txt`, the value is an empty `String`; For metadata, the key is `v0:m:/home/monika/poem0.txt`, the value is a `bincode` encoded `ObjectMetadata` structure binary string. On creating a file or directory, the backend should also create its all parent directories if not present. ```rust // create a file under `PATH` let mut path = std::path::PathBuf::new(PATH); let mut con = client.new_async_connection().await?; while let Some(parent) = path.parent() { let (p, c): (String, String) = (parent.display(), path.display()); let to_children = format!("v0:ch:{}", p); con.sadd(to_children, c).await?; path = parent; } ``` ## Read File Opendal empowers users to read with the `path` object, `offset` of the cursor and `size` to read. Redis provided a `GETRANGE` command which perfectly fit into it. ```rust // path: "poem0.txt" // offset: Option, the offset of reading // size: Option, the size of reading let path = get_abs_path(path); let c_path = content_key(path); let (mut start, mut end) = (0, -1); if let Some(offset) = offset { start = offset; } if let Some(size) = size { end = start + size; } let buf: Vec = con.getrange(c_path, start, end).await?; Box::new(buf) ``` ```redis GET v0:c:/home/monika/poem0.txt ``` ## Write File Redis ensures the writing of a single entry to be atomic, no locking is required. What needs to take care by opendal, besides the content of object, is its metadata. For example, though offering a `OBJECT IDLETIME` command, redis cannot record the last modified time of a key, so this should be done in opendal. ```rust use redis::AsyncCommands; // args: &OpWrite // r: BytesReader let mut buf = vec![]; let content_length: u64 = futures::io::copy(r, &mut buf).await?; let last_modified: String = OffsetDateTime::now().to_string(); // content md5 will not be offered let mut meta = ObjectMetadata::default(); meta.set_content_length(content_length); meta.set_last_modified(last_modified); // `ObjectMetadata` has implemented the `Serialize` and `Deserialize` trait of `Serde` // so bincode could serialize and deserialize it. let bytes = bincode::encode_to_vec(&meta)?; let path = get_abs_path(args.path()); let m_path = meta_key(path); let c_path = content_key(path); con.set(c_path, content).await?; con.set(m_path, bytes).await?; ``` ```redis SET v0:c:/home/monika/poem.txt content_string SET v0:m:/home/monika/poem.txt ``` ## Stat To get the metadata of an object, using the `GET` command and deserialize from bincode. ```rust let path = get_abs_path(args.path()); let meta = meta_key(path); let bin: Vec = con.get(meta).await?; let meta: ObjectMeta = bincode::deserialize(bin.as_slice())?; ``` ```redis GET v0:m:/home/monika/poem.txt ``` ## List For listing directories, just `SSCAN` through the child list of the directory, nice and correct. ```rust // print all sub-directories of `PATH` let s_key = format!("v0:k:{}", PATH); let mut con = client.new_async_connection().await?; let mut it = con.sscan::<&str, String>(s_key).await?; while let Some(child) = it.next_item().await { println!("get sub-dir: {}", child); } ``` ## Delete All subdirectories of path will be listed and removed. On deleting a file or directory, the backend should remove the entry from its parent's `SET`, and remove all children of entry. This could be done by postorder deleting. ```rust async fn remove_entry(con: &mut redis::aio::AsyncConnection, entry: String) { let skey = format!("v0:ch:{}", entry); let it = con.sscan::<&str, String>(skey).await?; while let Some(child) = it.next_item().await { remove_entry(&mut con, child).await; } if let Some(parent) = std::PathBuf::new(entry).parent() { let p: String = parent.display(); let parent_skey = format!("v0:ch:{}", p); let _ = con.srem(parent_skey, entry).await; } // remove metadata and content } ``` ## Blocking APIs `redis-rs` also offers a synchronous version of API, just port the functions above to its synchronous version. # Drawbacks 1. New dependencies is introduced: `redis-rs` and `bincode`; 2. Some calculations have to be done in client side, this will affect the performance; 3. Grouping atomic operations together doesn't promise transactional access, this may lead to data racing issues. 4. Writing large binary strings requiring copying all data from pipe(or `BytesReader` in opendal) to RAM, and then send to redis. # Rationale and alternatives ## RedisJSON module The [`RedisJSON`](https://redis.io/docs/stack/json/) module provides JSON support for Redis, and supports depth up to 128. Working on a JSON api could be easier than manually parsing or deserializing from `HASH`. Since `bincode` also offers the ability of deserializing and serializing, `RedisJSON` won't be used. # Prior art None # Unresolved questions None # Future possibilities The implementation proposed here is far from perfect. - The data organization could be optimized to make it acts more like a filesystem - Making a customized redis module to calculate metadata on redis side - Wait for stable of `bincode` 2.0, and bump to it. opendal-0.52.0/src/docs/rfcs/0627_split_capabilities.md000064400000000000000000000046441046102023000206620ustar 00000000000000- Proposal Name: `split-capabilities` - Start Date: 2022-09-04 - RFC PR: [apache/opendal#627](https://github.com/apache/opendal/pull/627) - Tracking Issue: [apache/opendal#628](https://github.com/apache/opendal/issues/628) # Summary Split basic operations into `read`, `write`, and `list` capabilities. # Motivation In [RFC-0409: Accessor Capabilities](./0409-accessor-capabilities.md), we introduce the ideas of `Accessor Capabilities`. Services could have different capabilities, and users can check them via: ```rust let meta = op.metadata(); let _: bool = meta.can_presign(); let _: bool = meta.can_multipart(); ``` If users call not supported capabilities, OpenDAL will return [`io::ErrorKind::Unsupported`](https://doc.rust-lang.org/stable/std/io/enum.ErrorKind.html#variant.Unsupported) instead. Along with that RFC, we also introduce an idea about `Basic Operations`: the operations that all services must support, including: - metadata - create - read - write - delete - list However, not all storage services support them. In our existing services, exception includes: - HTTP services don't support `write`, `delete`, and `list`. - IPFS HTTP gateway doesn't support `write` and `delete`. - NOTE: ipfs has a writable HTTP gateway, but there is no available instance. - fs could be read-only if mounted as `RO`. - object storage like `s3` and `gcs` could not have enough permission. - cache services may not support `list`. So in this RFC, we want to remove the idea about `Basic Operations` and convert them into different capabilities: - `read`: `read` and `stat` - `write`: `write` and `delete` - `list`: `list` # Guide-level explanation No public API changes. # Reference-level explanation This RFC will add three new capabilities: - `read`: `read` and `stat` - `write`: `write` and `delete` - `list`: `list` After this change, all services must declare the features they support. Most of this RFCs work is to refactor the tests. This RFC will refactor the behavior tests into several parts based on capabilities. # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities ## Read-only Services OpenDAL can implement read-only services after this change: - HTTP Service - IPFS HTTP Gateway ## Add new capabilities with Layers We can implement a layer that can add `list` capability for underlying storage services. For example, `IndexLayer` for HTTP services. opendal-0.52.0/src/docs/rfcs/0661_path_in_accessor.md000064400000000000000000000051061046102023000203120ustar 00000000000000- Proposal Name: `path_in_accessor` - Start Date: 2022-09-12 - RFC PR: [apache/opendal#661](https://github.com/apache/opendal/pull/661) - Tracking Issue: [apache/opendal#662](https://github.com/apache/opendal/issues/662) # Summary Move the path from `OpXxx` to `Accessor` directly. # Motivation `Accessor` uses `OpXxx` to carry `path` input: ```rust impl Accessor { async fn read(&self, args: &OpRead) -> Result { let _ = args; unimplemented!() } } #[derive(Debug, Clone, Default)] pub struct OpRead { path: String, offset: Option, size: Option, } ``` However, nearly all operation requires a `path`. And the path is represented in `String`, which means we have to clone it: ```rust impl OpRead { pub fn new(path: &str, range: impl RangeBounds) -> Result { let br = BytesRange::from(range); Ok(Self { path: path.to_string(), offset: br.offset(), size: br.size(), }) } } ``` Besides, we can't expose low-level APIs like: ```rust impl Object { pub async fn read_with(&self, op: OpRead) -> Result> { .. } } ``` Because users can't build the required `OpRead`. # Guide-level explanation With this RFC, users can use low-level APIs can control the `OpXxx` directly: ```rust impl Object { pub async fn read_with(&self, op: OpRead) -> Result> { .. } pub async fn write_with(&self, op: OpWrite, bs: impl Into>) -> Result<()> { .. } } ``` So we can add more args in requests like: ```rust o.write_with(OpWrite::new().with_content_md5("xxxxx"), bs).await; ``` # Reference-level explanation All `path` in `OpXxx` will be moved to `Accessor` directly: ```rust pub trait Accessor: Send + Sync + Debug { async fn create(&self, path: &str, args: OpCreate) -> Result<()> {} async fn read(&self, path: &str, args: OpRead) -> Result {} ... } ``` - All functions that accept `OpXxx` requires ownership instead of reference. - All `OpXxx::new()` will introduce breaking changes: ```diff - pub fn new(path: &str, range: impl RangeBounds) -> Result + pub fn new(range: impl RangeBounds) -> Self ``` # Drawbacks ## Breaking Changes This RFC may break users' code in the following ways: - Code that depends on `Accessor`: - Self-implemented Services - Self-implemented Layers - Code that depends on `OpXxx` # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities We can add more fields in `OpXxx`. opendal-0.52.0/src/docs/rfcs/0793_generic_kv_services.md000064400000000000000000000175231046102023000210410ustar 00000000000000- Proposal Name: `generic-kv-services` - Start Date: 2022-10-03 - RFC PR: [apache/opendal#793](https://github.com/apache/opendal/pull/793) - Tracking Issue: [apache/opendal#794](https://github.com/apache/opendal/issues/794) # Summary Add generic kv services support OpenDAL. # Motivation OpenDAL now has some kv services support: - memory - redis However, maintaining them is complex and very easy to be wrong. We don't want to implement similar logic for every kv service. This RFC intends to introduce a generic kv service so that we can: - Implement OpenDAL Accessor on this generic kv service - Add new kv service support via generic kv API. # Guide-level explanation No user-side changes. # Reference-level explanation OpenDAL will introduce a generic kv service: ```rust trait KeyValueAccessor { async fn get(&self, key: &[u8]) -> Result>>; async fn set(&self, key: &[u8], value: &[u8]) -> Result<()>; } ``` We will implement the OpenDAL service on `KeyValueAccessor`. To add new kv service support, users only need to implement it against `KeyValueAccessor`. ## Spec This RFC is mainly inspired by [TiFS: FUSE based on TiKV](https://github.com/Hexilee/tifs/blob/main/contribution/design.md). We will use the same `ScopedKey` idea in `TiFS`. ```rust pub enum ScopedKey { Meta, Inode(u64), Block { ino: u64, block: u64, }, Entry { parent: u64, name: String, }, } ``` We can encode a scoped key into a byte array as a key. Following is the common layout of an encoded key. ```text + 1byte +<----------------------------+ dynamic size +------------------------------------>+ | | | | | | | | | | | | | | | | | | | v v +------------------------------------------------------------------------------------------+ | | | | scope | body | | | | +-------+----------------------------------------------------------------------------------+ ``` ### Meta There is only one key in the meta scope. The meta key is designed to store metadata of our filesystem. Following is the layout of an encoded meta key. ```text + 1byte + | | | | | | | | | | | | | v +-------+ | | | 0 | | | +-------+ ``` This key will store data: ```rust pub struct Meta { inode_next: u64, } ``` The meta-structure contains only an auto-increasing counter `inode_next`, designed to generate an inode number. ### Inode Keys in the inode scope are designed to store attributes of files. Following is the layout of an encoded inode key. ```text + 1byte +<-------------------------------+ 8bytes +--------------------------------------->+ | | | | | | | | | | | | | | | | | | | v v +------------------------------------------------------------------------------------------+ | | | | 1 | inode number | | | | +-------+----------------------------------------------------------------------------------+ ``` This key will store data: ```rust pub struct Inode { meta: Metadata, blocks: HashMap, } ``` blocks is the map from `block_id` -> `size`. We will use this map to calculate the correct blocks to read. ### Block Keys in the block scope are designed to store blocks of a file. Following is the layout of an encoded block key. ```text + 1byte +<----------------- 8bytes ---------------->+<------------------- 8bytes ----------------->+ | | | | | | | | | | | | | | | | | | | | | | | | | v v v +--------------------------------------------------------------------------------------------------+ | | | | | 2 | inode number | block index | | | | | +-------+-------------------------------------------+----------------------------------------------+ ``` ### Entry Keys in the file index scope are designed to store the entry of the file. Following is the layout of an encoded file entry key. ```text + 1byte +<----------------- 8bytes ---------------->+<-------------- dynamic size ---------------->+ | | | | | | | | | | | | | | | | | | | | | | | | | v v v +--------------------------------------------------------------------------------------------------+ | | | | | 3 | inode number of parent directory | file name in utf-8 encoding | | | | | +-------+-------------------------------------------+----------------------------------------------+ ``` Store the correct inode number for this file. ```rust pub struct Index { pub ino: u64, } ``` # Drawbacks None. # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities None. opendal-0.52.0/src/docs/rfcs/0926_object_reader.md000064400000000000000000000046001046102023000176000ustar 00000000000000- Proposal Name: `object_reader` - Start Date: 2022-11-13 - RFC PR: [apache/opendal#926](https://github.com/apache/opendal/pull/926) - Tracking Issue: [apache/opendal#927](https://github.com/apache/opendal/issues/927) # Summary Returning reading related object meta in the reader. # Motivation Some services like s3 could return object meta while issuing reading requests. In `GetObject`, we could get: - Last-Modified - Content-Length - ETag - Content-Range - Content-Type - Expires We can avoid extra `HeadObject` calls by reusing that meta wisely, which could take 50ms. For example, `Content-Range` returns the content range of this read in the whole object: ` -/`. By using the content range, we can avoid `HeadObject` to get this object's total size, which means a lot for the content cache. # Guide-level explanation `reader` and all its related API will return `ObjectReader` instead: ```diff - pub async fn reader(&self) -> Result {} + pub async fn reader(&self) -> Result {} ``` `ObjectReader` impls `BytesRead` too, so existing code will keep working. And `ObjectReader` will provide similar APIs to `Entry`, for example: ```rust pub async fn content_length(&self) -> Option {} pub async fn last_modified(&self) -> Option {} pub async fn etag(&self) -> Option {} ``` Note: - All fields are optional, as services like fs could not return them. - `content_length` here is this read request's length, not the object's length. # Reference-level explanation We will change the API signature of `Accessor`: ```diff - async fn read(&self, path: &str, args: OpRead) -> Result {} + async fn read(&self, path: &str, args: OpRead) -> Result {} ``` `ObjectReader` is a wrapper of `BytesReader` and `ObjectMeta`: ```rust pub struct ObjectReader { inner: BytesReader meta: ObjectMetadata, } impl ObjectReader { pub async fn content_length(&self) -> Option {} pub async fn last_modified(&self) -> Option {} pub async fn etag(&self) -> Option {} } ``` Services can decide whether or not to fill them. # Drawbacks None. # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities ## Add content-range support We can add `content-range` in `ObjectMeta` so that users can fetch and use them. opendal-0.52.0/src/docs/rfcs/0977_refactor_error.md000064400000000000000000000104741046102023000200420ustar 00000000000000- Proposal Name: `refactor-error` - Start Date: 2022-11-21 - RFC PR: [apache/opendal#977](https://github.com/apache/opendal/pull/977) - Tracking Issue: [apache/opendal#976](https://github.com/apache/opendal/pull/976) # Summary Use a separate error instead of `std::io::Error`. # Motivation OpenDAL is used to use `std::io::Error` for all functions. This design is natural and easy to use. But there are many problems with the usage: ## Not friendly for retry `io::Error` can't carry retry-related information. In [RFC-0247: Retryable Error](./0247-retryable-error.md), we use `io::ErrorKind::Interrupt` to indicate this error is retryable. But this change will hide the real error kind from the underlying. To mark this error has been retried, we have to add another new error wrapper: ```rust #[derive(thiserror::Error, Debug)] #[error("permanent error: still failing after retry, source: {source}")] struct PermanentError { source: Error, } ``` ## ErrorKind is inaccurate `std::io::ErrorKind` is used to represent errors returned from system io, which is unsuitable for mistakes that have business semantics. For example, users can't distinguish `ObjectNotFound` or `BucketNotFound` from `ErrorKind::NotFound`. ## ErrorKind is incomplete OpenDAL has been waiting for features [`io_error_more`](https://github.com/rust-lang/rust/issues/86442) to be stabilized for a long time. But there is no progress so far, which makes it impossible to return `ErrorKind::IsADirectory` or `ErrorKind::NotADirectory` on stable rust. ## Error is not easy to carry context To carry context inside `std::io::Error`, we have to check and make sure all functions are constructed `ObjectError` or `BackendError`: ```rust #[derive(Error, Debug)] #[error("object error: (op: {op}, path: {path}, source: {source})")] pub struct ObjectError { op: Operation, path: String, source: anyhow::Error, } ``` To make everything worse, we can't prevent opendal returns raw io errors directly. For example, in `Object::range_read`: ```rust pub async fn range_read(&self, range: impl RangeBounds) -> Result> { ... io::copy(s, &mut bs).await?; Ok(bs.into_inner()) } ``` We leaked the `io::Error` without any context. # Guide-level explanation Thus, I propose to add `opendal::Error` back with everything improved. Users will have similar usage as before: ```rust if let Err(e) = op.object("test_file").metadata().await { if e.kind() == ErrorKind::ObjectNotFound { println!("object not exist") } } ``` Users can check if this error a `temporary`: ```rust if err.is_temporary() { // retry the operation } ``` Users can print error messages via `Display`: ```rust > println!("{}", err); ObjectNotFound (permanent) at read, context: { service: S3, path: path/to/file } => status code: 404, headers: {"x-amz-request-id": "GCYDTQX51YRSF4ZF", "x-amz-id-2": "EH0vV6lTwWk+lFXqCMCBSk1oovqhG4bzALU9+sUudyw7TEVrfWm2o/AFJKhYKpdGqOoBZGgMTC0=", "content-type": "application/xml", "date": "Mon, 21 Nov 2022 05:26:37 GMT", "server": "AmazonS3"}, body: "" ``` Also, users can choose to print the more verbose message via `Debug`: ```rust > println!("{:?}", err); ObjectNotFound (permanent) at read => status code: 404, headers: {"x-amz-request-id": "GCYDTQX51YRSF4ZF", "x-amz-id-2": "EH0vV6lTwWk+lFXqCMCBSk1oovqhG4bzALU9+sUudyw7TEVrfWm2o/AFJKhYKpdGqOoBZGgMTC0=", "content-type": "application/xml", "date": "Mon, 21 Nov 2022 05:26:37 GMT", "server": "AmazonS3"}, body: "" Context: service: S3 path: path/to/file Source: Backtrace: ``` # Reference-level explanation We will add new `Error` and `ErrorKind` in opendal: ```rust pub struct Error { kind: ErrorKind, message: String, status: ErrorStatus, operation: &'static str, context: Vec<(&'static str, String)>, source: Option, } ``` - status: the status of this error, which indicates if this error is temporary - operation: the operation which generates this error - context: the context related to this error - source: the underlying source error # Drawbacks ## Breaking changes This RFC will lead to a breaking at user side. # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities ## More ErrorKind We can add more error kinds to make it possible for users to check. opendal-0.52.0/src/docs/rfcs/1085_object_handler.md000064400000000000000000000046761046102023000177650ustar 00000000000000- Proposal Name: `object_handler` - Start Date: 2022-12-19 - RFC PR: [apache/opendal#1085](https://github.com/apache/opendal/pull/1085) - Tracking Issue: [apache/opendal#1085](https://github.com/apache/opendal/issues/1085) # Summary Returning a `file description` to users for native seek support. # Motivation OpenDAL's goal is to `access data freely, painlessly, and efficiently`, so we build an operation first API which means we provide operation instead of the file description. Users don't need to call `open` before `read`; OpenDAL will handle all the open and close functions. However, our users do want to control the complex behavior of that: - Some storage backends have native `seek` support, but OpenDAL can't fully use them. - Users want to improve performance by reusing the same file description without `open` and `close` for every read operation. This RFC will fill this gap. # Guide-level explanation Users can get an object handler like: ```rust let oh: ObjectHandler = op.object("path/to/file").open().await?; ``` `ObjectHandler` will implement `AsyncRead` and `AsyncSeek` so it can be used like `tokio::fs::File`. If the backend supports native seek operation, we will take the native process; otherwise, we will fall back to simulation implementations. The blocking version will be provided by: ```rust let boh: BlockingObjectHandler = op.object("path/to/file").blocking_open()?; ``` And `BlockingObjectHandler` will implement `Read` and `Seek` so it can be used like `std::fs::File`. If the backend supports native seek operation, we will take the native process; otherwise, we will fall back to simulation implementations. # Reference-level explanation This RFC will add a new API `open` in `Accessor`: ```rust pub trait Accessor { async fn open(&self, path: &str, args: OpOpen) -> Result<(RpOpen, BytesHandler)>; } ``` Only services that support native `seek` operations can implement this API, like `fs` and `hdfs`. For services that do not support native `seek` operations like `s3` and `azblob`, we will fall back to the simulation implementations: maintaining an in-memory index instead. # Drawbacks None # Rationale and alternatives ## How about writing operations? Ideally, writing on `ObjectHandler` should also be supported. But we still don't know how this API will be used. Let's apply this API for `read` first. # Prior art None # Unresolved questions None # Future possibilities - Add write support - Adopt native `pread` opendal-0.52.0/src/docs/rfcs/1391_object_metadataer.md000064400000000000000000000071061046102023000204460ustar 00000000000000- Proposal Name: `object_metadataer` - Start Date: 2023-02-21 - RFC PR: [apache/opendal#1391](https://github.com/apache/opendal/pull/1391) - Tracking Issue: [apache/opendal#1393](https://github.com/apache/opendal/issues/1393) # Summary Add object metadataer to avoid unneeded extra metadata call. # Motivation OpenDAL has native metadata cache for now: ```rust let _ = o.metadata().await?; // This call doesn't need to send a request. let _ = o.metadata().await?; ``` Also, OpenDAL can reuse metadata from `list` or `scan`: ```rust let mut ds = o.scan().await?; while let Some(de) = ds.try_next().await? { // This call doesn't need to send a request (if we are lucky enough). let _ = de.metadata().await?; } ``` By reusing metadata from `list` or `scan` we can reduce the extra `stat` call for each object. In our real use cases, we can reduce the total time to calculate the total length inside a dir with 6k files from 4 minutes to 2 seconds. However, metadata can only be cached as a whole. If services could return more metadata in `stat` than in `list`, we wouldn't be able to mark the metadata as cacheable. If services add more metadata, we could inadvertently introduce the performance degradation. This RFC aims to address this problem by hiding `ObjectMetadata` and adding `ObjectMetadataer` instead. All object metadata values will be cached separately and all user calls to object metadata will go to the cache. # Guide-level explanation This RFC will add `ObjectMetadataer` and `BlockingObjectMetadataer` for users: Users call to `o.metadata()` will return `ObjectMetadataer` instead: ```rust let om: ObjectMetadataer = o.metadata().await?; ``` And users can query more metadata over it: ```rust let content_length = om.content_length().await; let etag = om.etag().await; ``` During the whole lifetime of the corresponding `Object` or `ObjectMetadataer`, we make sure that at most one `stat` call is sent. After this change, users will never get an `ObjectMetadata` anymore. # Reference-level explanation We will introduce a bitmap to store the state of all object metadata fields separately. Everytime users call `key` on metadata, we will check as following: - If `bitmap` is set, return directly. - If `bitmap` is not set, but is complete, return directly. - If both `bitmap` is not set and not `complete`, call `stat` to get the meta. `Object` will return `ObjectMetadataer` instead of `ObjectMetadata`: ```diff - pub async fn metadata(&self) -> Result {} + pub async fn metadata(&self) -> Result {} ``` And `ObjectMetadataer` will provide the following API: ```rust impl ObjectMetadataer { pub async fn mode(&self) -> Result; pub async fn content_length(&self) -> Result; pub async fn content_md5(&self) -> Result>; pub async fn last_modified(&self) -> Result>; pub async fn etag(&self) -> Result>; } impl BlockingObjectMetadataer { pub fn mode(&self) -> Result pub fn content_length(&self) -> Result; pub fn content_md5(&self) -> Result>; pub fn last_modified(&self) -> Result>; pub fn etag(&self) -> Result>; } ``` # Drawbacks ## Breaking changes This RFC will introduce breaking changes for `Object::metadata`. And users can't do `serde::Serialize` or `serde::Deserialize` on object metadata any more. All metadata related API calls will be removed from `Object`. # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities None. opendal-0.52.0/src/docs/rfcs/1398_query_based_metadata.md000064400000000000000000000062171046102023000211650ustar 00000000000000- Proposal Name: `query_based_metadata` - Start Date: 2022-02-22 - RFC PR: [apache/opendal#1398](https://github.com/apache/opendal/pull/1398) - Tracking Issue: [apache/opendal#1398](https://github.com/apache/opendal/pull/1398) # Summary Read cached metadata based on users query. # Motivation OpenDAL has native metadata cache for now: ```rust let _ = o.metadata().await?; // This call doesn't need to send a request. let _ = o.metadata().await?; ``` Also, OpenDAL can reuse metadata from `list` or `scan`: ```rust let mut ds = o.scan().await?; while let Some(de) = ds.try_next().await? { // This call doesn't need to send a request (if we are lucky enough). let _ = de.metadata().await?; } ``` By reusing metadata from `list` or `scan` we can reduce the extra `stat` call for each object. In our real use cases, we can reduce the total time to calculate the total length inside a dir with 6k files from 4 minutes to 2 seconds. However, metadata can only be cached as a whole. If services could return more metadata in `stat` than in `list`, we wouldn't be able to mark the metadata as cacheable. If services add more metadata, we could inadvertently introduce the performance degradation. RFC [Object Metadataer](./rfc_1391_object_metadataer) intends to add `ObjectMetadataer` to address this issue. But it sooner to be proved that a failure: it's hard to design a correct API. Users have to write code like the following: ```rust let om = o.metadata().await?; let _ = om.content_length().await?; let _ = om.content_md5().await?; ``` In this RFC, we will add a query based metadata. # Guide-level explanation After this RFC, `o.metadata()` will accept a query composed by `ObjectMetadataKey`. To query already cached metadata: ```rust let meta = op.object("test").metadata(None).await?; let _ = meta.content_length(); let _ = meta.content_type(); ``` To query content length and content type: ```rust let meta = op .object("test") .metadata({ use ObjectMetadataKey::*; ContentLength | ContentType }) .await?; let _ = meta.content_length(); let _ = meta.content_type(); ``` To query all metadata about this object: ```rust let meta = op .object("test") .metadata({ ObjectMetadataKey::Complete }) .await?; let _ = meta.content_length(); let _ = meta.content_type(); ``` # Reference-level explanation We will store bits in `ObjectMetadata` to store which fields have been set. And we can compare the bits to decide whether we need to query from storage again: ```rust pub async fn metadata( &mut self, flags: impl Into>, ) -> Result> { if let Some(meta) = &self.meta { if meta.bit().contains(flags) || meta.bit().contains(ObjectMetadataKey::Complete) { return Ok(meta.clone()); } } let meta = Arc::new(self.stat().await?); self.meta = Some(meta.clone()); Ok(meta) } ``` # Drawbacks ## Breaking changes After this RFC, `Object::metadata()` will accept a query. And all existing users need to adapt their code for that. # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/1420_object_writer.md000064400000000000000000000077751046102023000176600ustar 00000000000000- Proposal Name: `object_writer` - Start Date: 2023-02-27 - RFC PR: [apache/opendal#1420](https://github.com/apache/opendal/pull/1420) - Tracking Issue: [apache/opendal#1421](https://github.com/apache/opendal/issues/1421) # Summary Adding `ObjectWriter` to improve support for multipart uploads, as well as enable retry options for write operations. # Motivation OpenDAL works well for `read` operations: - OpenDAL can seek over content even on services like S3. - OpenDAL can retry read from the failing point without extra read cost. However, OpenDAL is not good at `write`: ## Complex multipart operations OpenDAL supports multipart operations but it's very hard to use: ```ignore let object_multipart = o.create_multipart().await?; let part_0 = object_multipart.write(0, content_0).await?; ... let part_x = object_multipart.write(x, content_x).await?; let new_object = object_multipart.complete(vec![part_0,...,part_x]).await?; ``` Users should possess the knowledge of the multipart API to effectively use it. To exacerbate the situation, the multipart API is not standardized and only some object storage services offer support for it. Unfortunately, we cannot even provide support for it on the local file system. ## Lack of retry support OpenDAL can't retry `write` operations because we accept an `Box`. Once we pass this read into other functions, we consumed it. ```rust async fn write(&self, path: &str, args: OpWrite, r: input::Reader) -> Result { self.inner.write(path, args, r).await } ``` By introducing the `ObjectWriter` feature, we anticipate resolving all the associated inquiries simultaneously. # Guide-level explanation `ObjectWriter` will provide the following APIs: ```rust impl ObjectWriter { pub async write(&mut self, bs: Bytes) -> Result<()>; pub async close(&mut self) -> Result<()>; } ``` After `ObjectWriter` has been constructed, users can use it as a normal writer: ```rust let mut w = o.writer().await?; w.write(bs1).await?; w.write(bs2).await?; w.close().await?; ``` `ObjectWriter` also implements `AsyncWrite` trait which will allow users to use `io::copy` as well: ```rust let mut w = o.writer().await?; let _ = io::copy_buf(r, o).await?; ``` # Reference-level explanation OpenDAL will add a new trait called `output::Writer`: ```rust pub trait Write: Unpin + Send + Sync { pub async write(&mut self, bs: Bytes) -> Result<()>; pub async initiate(&mut self) -> Result<()>; pub async append(&mut self, bs: Bytes) -> Result<()>; pub async close(&mut self) -> Result<()>; } ``` - `write` is used to write full content. - `initiate` is used to initiate a multipart writer. - `append` is used to append more content into this writer. - `close` is used to close and construct the final file. And `Accessor` will change the `write` API into: ```diff pub trait Accessor { + type Writer: output::Write; - async fn write(&self, path: &str, args: OpWrite, r: input::Reader) -> Result; + async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> } ``` # Drawbacks More heavy work for service implementers. # Rationale and alternatives ## Why accept `Bytes`? OpenDAL's write is similar to `io::Write::write_all` which will always consume the whole input and return errors if something is wrong. By accepting `Bytes`, we can reduce the extra `Clone` between user land to OpenDAL's services/layers. # Prior art None. # Unresolved questions None. # Future possibilities ## Vectored Write We can add `write_vectored` support in the future: ```rust pub trait Write: Unpin + Send + Sync { pub async write_vectored(&mut self, bs: &[Bytes]) -> Result<()>; } ``` Take `s3` services as an example, we can upload different parts at the same time. ## Write From Stream We can add `write_from` support in the future: ```rust pub trait Write: Unpin + Send + Sync { pub async write_from(&mut self, r: BytesStream, size: u64) -> Result<()>; } ``` By implementing this feature, users don't need to hold a large buffer inside memory. opendal-0.52.0/src/docs/rfcs/1477_remove_object_concept.md000064400000000000000000000104201046102023000213450ustar 00000000000000- Proposal Name: `remove_object_concept` - Start Date: `2023-03-05` - RFC PR: [apache/opendal#1477](https://github.com/apache/opendal/pull/1477) - Tracking Issue: [apache/opendal#0000](https://github.com/apache/opendal/issues/0000) # Summary Eliminating the Object concept to enhance the readability of OpenDAL. # Motivation OpenDAL introduces [Object Native API][crate::docs::rfcs::rfc_0041_object_native_api] to resolve the problem of not being easy to use: ```diff - let reader = SeekableReader::new(op, path, stream_len); + let reader = op.object(&path).reader().await?; - op.stat(&path).run().await + op.object(&path).stat().await ``` However, times are changing. After the list operation has been moved to the `Object` level, `Object` is now more like a wrapper for `Operator`. The only meaningful API of `Operator` now is `Operator::object`. Writing `op.object(&path)` repeatedly is boring. Let's take real example from databend as an example: ```rust if let Some(dir) = dir_path { match op.object(&dir).stat().await { Ok(_) => { let mut ds = op.object(&dir).scan().await?; while let Some(de) = ds.try_next().await? { if let Some(fi) = stat_file(de).await? { files.push(fi) } } } Err(e) => warn!("ignore listing {path}/, because: {:?}", e), }; } ``` We designed `Object` to make users can reuse the same `Object`. However, nearly no users use our API this way. Most users just build a new `Object` every time. There are two problems: ## Extra cost `Object::new()` is not zero cost: ```rust pub(crate) fn with(op: Operator, path: &str, meta: Option) -> Self { Self { acc: op.inner(), path: Arc::new(normalize_path(path)), meta: meta.map(Arc::new), } } ``` The `Object` must contain an `Operator` and an `Arc` of strings. With the introduction of [Query Based Metadata][crate::docs::rfcs::rfc_1398_query_based_metadata], there is no longer a need to perform operations on the object. ## Complex concepts The term `Object` is applied in various fields, making it difficult to provide a concise definition of `opendal::Object`. Moreover, this could potentially confuse our users who may assume that `opendal::Object` is primarily intended for object storage services. I propose eliminating the intermediate API layer of `Object` and enabling users to directly utilize `Operator`. # Guide-level explanation After this RFC is implemented, our users can: ```rust # read all content of the file op.read("file").await?; # read part content of the file op.range_read("file", 0..1024).await?; # create a reader op.reader("file").await?; # write all content into file op.write("file", bs).await?; # create a writer op.writer("file").await?; # get metadata of a path op.stat("path").await?; # delete a path op.delete("path").await?; # remove paths op.remove(vec!["path_a"]).await?; # remove path recursively op.remove_all("path").await?; # create a dir op.create_dir("dir/").await?; # list a dir op.list("dir/").await?; # scan a dir op.scan("dir/").await?; ``` We will include the `BlockingOperator` for enhanced ease of use while performing blocking operations. ```rust # this is a cheap call without allocation let bop = op.blocking(); # read all content bop.read("file")?; # write all content bop.write("file", bs)?; ``` The scan/list result will be an `Entry` or `BlockingEntry` that contains the same fields as Object, but is only used for scan/list entries. The public API should look like: ```rust impl Entry { pub fn mode(&self) -> EntryType; pub fn path(&self) -> &str; pub async fn stat(&self) -> Result; pub async fn metadata(&self, key: impl Into) -> Result; ... } ``` # Reference-level explanation We will remove `Object` entirely and move all `Object` APIs to `Operator` instead: ```rust - op.object("path").read().await + op.read("path").await ``` Along with this change, we should also rename the `ObjectXxx` structs, such as `ObjectReader` to `Reader`. # Drawbacks ## Breaking Changes This RFC proposes a major breaking change that will require almost all current usage to be rewritten. # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/1735_operation_extension.md000064400000000000000000000101001046102023000210730ustar 00000000000000- Proposal Name: `operation_extension` - Start Date: 2023-03-23 - RFC PR: [apache/opendal#1735](https://github.com/apache/opendal/pull/1735) - Tracking Issue: [apache/opendal#1738](https://github.com/apache/opendal/issues/1738) # Summary Extend operation capabilities to support additional native features. # Motivation OpenDAL only supports a limited set of capabilities for operations. - `read`/`stat`: only supports `range` - `write`: only supports `content_type` and `content_disposition` Our community has a strong need for more additional native features. For example: - [opendal#892](https://github.com/apache/opendal/issues/892) wants [Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control): Allow users to specify the cache control headers for the uploaded files. - [opendal#825](https://github.com/apache/opendal/issues/825) wants [If-Match](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Match) and [If-None-Match](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match): Allow users to makes a request conditional. - [opendal#1726](https://github.com/apache/opendal/issues/1726) wants [response-content-disposition](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html): Allow users to specify the content disposition for the downloaded files. All of these feature requests are essentially asking for the same thing: the capability to define supplementary arguments for operations, particularly about HTTP services. # Guide-level explanation In this RFC, we will allow users to specify the standard HTTP headers like cache_control/if_match: ```rust let op = OpRead::default(). with_cache_control("max-age=3600"). with_if_match("\"bfc13a64729c4290ef5b2c2730249c88ca92d82d\""); let bs = o.read_with(op).await?; ``` Also, we will support some non-standard but widely used features like `response-content-disposition`: ```rust let op = OpRead::default(). with_override_content_disposition("attachment; filename=\"foo.txt\""); let req = op.presign_read_with("filename", Duration::hours(1), op)?; ``` And, finally, we will support allow specify the default headers for services. Take `s3`'s `storage_class` as an example: ```rust let mut builder = S3::default(); builder.default_storage_class("STANDARD_IA"); builder.default_cache_control("max-age=3600"); ``` In general, we will support the following features: - Allow users to specify the (non-)standard HTTP headers for operations. - Allow users to specify the default HTTP headers for services. # Reference-level explanation We will make the following changes in OpenDAL: For `OpRead` & `OpStat`: ```diff pub struct OpRead { br: BytesRange, + cache_control: Option, + if_match: Option, + if_none_match: Option, + override_content_disposition: Option, } ``` For `OpWrite`: ```diff pub struct OpWrite { append: bool, content_type: Option, content_disposition: Option, + cache_control: Option, } ``` We will provide different default options for each service. For example, we can add `default_storage_class` for `s3`. # Drawbacks None # Rationale and alternatives ## Why using `override_content_disposition` instead of `response_content_disposition`? `response_content_disposition` is not a part of HTTP standard, it's the private API provided by `s3`. - `azblob` will use `x-ms-blob-content-disposition` header - `ocios` will use `httpResponseContentDisposition` query OpenDAL does not accept the query as is. Instead, we have created a more readable name `override_content_disposition` to clarify its purpose. # Prior art None # Unresolved questions None # Future possibilities ## Strict Mode Additionally, we have implemented a `strict` option for the `Operator`. If users enable this option, OpenDAL will return an error message for unsupported options. Otherwise, it will ignore them. For instance, if users rely on the `if_match` behavior but services like `fs` and `hdfs` do not support it natively, they can use the `op.with_strict()` function to prompt OpenDAL to return an error. opendal-0.52.0/src/docs/rfcs/2083_writer_sink_api.md000064400000000000000000000060741046102023000202040ustar 00000000000000- Proposal Name: `writer_sink_api` - Start Date: 2023-04-23 - RFC PR: [apache/opendal#2083](https://github.com/apache/opendal/pull/2083) - Tracking Issue: [apache/opendal#2084](https://github.com/apache/opendal/issues/2084) # Summary Include a `sink` API within the `Writer` to enable streaming writing. # Motivation OpenDAL does not support streaming data uploads. Users must first load the data into memory and then send it to the `writer`. ```rust let bs = balabala(); w.write(bs).await?; let bs = daladala(); w.write(bs).await?; ... w.close().await?; ``` There are two main drawbacks to OpenDAL: - high memory usage, as reported in issue #1821 on GitHub - low performance due to the need to buffer user data before sending it over the network. To address this issue, it would be beneficial for OpenDAL to provide an API that allows users to pass a stream or reader directly into the writer. # Guide-level explanation I propose to add the following API to `Writer`: ```rust impl Writer { pub async fn copy_from(&mut self, size: u64, r: R) -> Result<()> where R: futures::AsyncRead + Send + Sync + 'static; pub async fn pipe_from(&mut self, size: u64, s: S) -> Result<()> where S: futures::TryStream + Send + Sync + 'static Bytes: From; } ``` Users can now upload data in a streaming way: ```rust // Start writing the 5 TiB file. let w = op.writer_with( OpWrite::default() .with_content_length(5 * 1024 * 1024 * 1024 * 1024)); let r = balabala(); // Send to network directly without in-memory buffer. w.copy_from(size, r).await?; // repeat... ... // Close the write once we are ready! w.close().await?; ``` The underlying services will handle this stream in the most efficient way possible. # Reference-level explanation To support `Wrtier::copy_from` and `Writer::pipe_from`, we will add a new API called `sink` inside `oio::Writer`: ```rust #[async_trait] pub trait Write: Unpin + Send + Sync { async fn sink(&mut self, size: u64, s: Box + Send + Sync>) -> Result<()>; } ``` OpenDAL converts the user input reader and stream into a byte stream for `oio::Write`. Services that support streaming upload natively can directly pass the stream. If not, they can use `write` repeatedly to write the entire stream. # Drawbacks None. # Rationale and alternatives ## What's the different of `OpWrite::content_length` and `sink` size? The `OpWrite::content_length` parameter specifies the total length of the file to be written, while the `size` argument in the `sink` API indicates the size of the reader or stream provided. Certain services may optimize by writing all content in a single request if `content_length` is the same with given `size`. # Prior art None # Unresolved questions None # Future possibilities ## Retry for the `sink` API It's impossible to retry the `sink` API itself, but we can provide a wrapper to retry the stream's call of `next`. If we met a retryable error, we can call `next` again by crate like `backon`. ## Blocking support for sink We will add async support first. opendal-0.52.0/src/docs/rfcs/2133_append_api.md000064400000000000000000000051271046102023000171050ustar 00000000000000- Proposal Name: `append_api` - Start Date: 2023-04-26 - RFC PR: [apache/opendal#2133](https://github.com/apache/opendal/pull/2133) - Tracking Issue: [apache/opendal#2163](https://github.com/apache/opendal/issues/2163) # Summary Introduce append operations for OpenDAL which allow users to add data to a file. # Motivation OpenDAL has the write operation used to create a file and upload in parts. This is implemented based on multipart API. However, current approach has some limitations: - Data could be lost and not readable before w.close() returned Ok(()) - File can't be appended again after w.close() returned Ok(()) To address these issues, I propose adding an append operation. Users can create an appender that provides a reentrant append operation. Each append operation will add data to the end of the file, which can be read immediately after the operation. # Guide-level explanation The files created by the append operation can be appended via append API. ```rust async fn append_test(op: Operation) -> Result<()> { // create writer let append = op.append("path_to_file").await?; let bs = read_from_file(); append.append(bs).await?; let bs = read_from_another_file(); append.append(bs).await?; // close the file append.close().await?; } ``` Difference between the write and append operation: - write: Always create a new file, not readable until close. - append: Can append existing appendable file, readable after append return. # Reference-level explanation For underlay API, we will make these changes in Accessor: ```rust trait Accessor { type Appender: oio::Append; async fn append(&self, path: &str, args: OpAppend) -> Result; } ``` To implement this feature, we need to add a new API `append` into `oio::Append`. ```rust #[async_trait] pub trait Append: Unpin + Send + Sync { /// Append data to the end of file. /// Users will call `append` multiple times. Please make sure `append` is safe to re-enter. async fn append(&mut self, bs: Bytes) -> Result<()>; /// Seal the file to mark it as unmodifiable. async fn close(&mut self) -> Result<()>; } ``` # Drawbacks None. # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities We can use append API to implement for services that natively support append, such as [Azure blob](https://learn.microsoft.com/en-us/rest/api/storageservices/append-block?tabs=azure-ad) and [Alibaba cloud OSS](https://www.alibabacloud.com/help/en/object-storage-service/latest/appendobject). This will improve the performance and reliability of append operation. opendal-0.52.0/src/docs/rfcs/2299_chain_based_operator_api.md000064400000000000000000000047521046102023000220110ustar 00000000000000- Proposal Name: `chain_based_operator_api` - Start Date: 2023-05-23 - RFC PR: [apache/opendal#2299](https://github.com/apache/opendal/pull/2299) - Tracking Issue: [apache/opendal#2300](https://github.com/apache/opendal/issues/2300) # Summary Add chain based Operator API to replace `OpXxx`. # Motivation OpenDAL provides `xxx_with` API for users to add more options for requests: ```rust let bs = op.read_with("path/to/file", OpRead::new().with_range(0..=1024)).await?; ``` However, the API's usability is hindered as users are required to create a new `OpXxx` struct. The API call can be excessively verbose: ```rust let bs = op.read_with( "path/to/file", OpRead::new() .with_range(0..=1024) .with_if_match("") .with_if_none_match("") .with_override_cache_control("") .with_override_content_disposition("") ).await?; ``` # Guide-level explanation In this proposal, I plan to introduce chain based `Operator` API to make them more friendly to use: ```rust let bs = op.read_with("path/to/file") .range(0..=1024) .if_match("") .if_none_match("") .override_cache_control("") .override_content_disposition("") .await?; ``` By eliminating the usage of `OpXxx`, our users can write code that is more readable. # Reference-level explanation To implement chain based API, we will change `read_with` as following: ```diff - pub async fn read_with(&self, path: &str, args: OpRead) -> Result> + pub fn read_with(&self, path: &str) -> FutureRead ``` `FutureRead` will implement `Future>>`, so that users can still call `read_with` like the following: ```rust let bs = op.read_with("path/to/file").await?; ``` For blocking operations, we will change `read_with` as following: ```diff - pub fn read_with(&self, path: &str, args: OpRead) -> Result> + pub fn read_with(&self, path: &str) -> FunctionRead ``` `FunctionRead` will implement `call(self) -> Result>`, so that users can call `read_with` like the following: ```rust let bs = op.read_with("path/to/file").call()?; ``` After this change, all `OpXxx` will be moved as raw API. # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities ## Change API after fn_traits stabilized After [fn_traits](https://github.com/rust-lang/rust/issues/29625) get stabilized, we will implement `FnOnce` for `FunctionXxx` instead of `call`. opendal-0.52.0/src/docs/rfcs/2602_object_versioning.md000064400000000000000000000135771046102023000205270ustar 00000000000000- Proposal Name: object_versioning - Start Date: 2023-07-06 - RFC PR: [apache/opendal#2602](https://github.com/apache/opendal/pull/2602) - Tracking Issue: [apache/opendal#2611](https://github.com/apache/opendal/issues/2611) # Summary This proposal describes the object versioning (or object version control) feature of OpenDAL. # Motivation There is a kind of storage service, which is called object storage service, provides a simple and scalable way to store, organize, and access unstructured data. These services store data as objects within buckets. And an object is a file and any metadata that describes that file, a bucket is a container for objects. The object versioning provided by these services is a very useful feature. It allows users to keep multiple versions of an object in the same bucket. If users enable object versioning, each object will have a history of versions. Each version will have a unique version ID, which is a string that is unique for each version of an object. (The object, bucket, and version ID mentioned here are all concepts of object storage services, they could be called differently in different services, but they are the same thing.) OpenDAL provides support for some of those services, such as S3, GCS, Azure Blob Storage, etc. Now we want to add support for object versioning to OpenDAL. # Guide-level explanation When object versioning is enabled, the following operations will be supported: - `stat`: Get the metadata of an object with specific version ID. - `read`: Read a specific version of an object. - `delete`: Delete a specific version of an object. Code example: ```rust // To get the current version ID of a file let meta = op.stat("path/to/file").await?; let version_id = meta.version().expect("just for example"); // To fetch the metadata of specific version of a file let meta = op.stat_with("path/to/file").version("version_id").await?; let version_id = meta.version().expect("just for example"); // get the version ID // To read an file with specific version ID let content = op.read_with("path/to/file").version("version_id").await?; // To delete an file with specific version ID op.delete_with("path/to/file").version("version_id").await?; ``` # Reference-level explanation Those operations with object version are different from the normal operations: - `stat`: when getting the metadata of a file, it will always get the metadata of the latest version of the file if no version ID is specified. And there will be a new field `version` in the metadata to indicate the version ID of the file. - `read`: when reading a file, it will always read the latest version of the file if no version ID is specified. - `delete`: when deleting a file, it will always delete the latest version of the file if no version ID is specified. And users will not be able to read this file without specifying the version ID, unless they specify a version not be deleted. And with object versioning, when writing an object, it will always create a new version of the object than overwrite the old version. But here it is imperceptible to the user. Because the version id is generated by the service itself, it cannot be specified by the user and user cannot override the historical version. To implement object versioning, we will do the following: - Add a new field `version` to `OpStat`, `OpRead` and `OpDelete` struct. - Add a new field `version` to `ObjectMetadata` struct. - Add a new property(setter) `version` to the return value of `stat_with`, `read_with` method. - Add a new method `delete_with` and add a new property(setter) `version` to the return value of `delete_with` method. For service backend, it should support the following operations: - `stat`: Get the metadata of an object with specific version ID. - `read`: Read a specific version of an object. - `delete`: Delete a specific version of an object. # Drawbacks None. # Rationale and alternatives ## What is object versioning? Object versioning is a feature that allows users to keep multiple versions of an object in the same bucket. It's a way to preserve, retrieve, and restore every version of every object stored in a bucket. With object versioning, users can easily recover from both unintended user actions and application failures. ## How does object versioning work? When object versioning is enabled, each object will have a history of versions. Each version will have a unique version ID, which is a string that is unique for each version of an object. The version ID is not a timestamp. It is not guaranteed to be sequential. Many object storage services produce object version IDs by themselves, using their own algorithms. Users cannot specify the version ID when writing an object. ## Will object versioning affect the existing code? There is no difference between whether object versioning is enabled or not when writing an object. The storage service will always create a new version of the object than overwrite the old version when writing an object. But here it is imperceptible to the user. ## What are the benefits of object versioning? With object versioning, users can: - Track the history of a file. - Implement optimistic concurrency control. - Implement a simple backup system. ## reference - [AWS S3 Object Versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html) - [How does AWS S3 object versioning work?](https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning-workflows.html) - [How to enable object versioning for a bucket in AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html) - [Google Cloud Storage Object Versioning](https://cloud.google.com/storage/docs/object-versioning) - [Azure Blob Storage Object Versioning](https://docs.microsoft.com/en-us/azure/storage/blobs/versioning-overview) # Prior art None. # Unresolved questions None. # Future possibilities Impl a new method `list_versions`(list all versions of an object). opendal-0.52.0/src/docs/rfcs/2758_merge_append_into_write.md000064400000000000000000000040271046102023000217110ustar 00000000000000- Proposal Name: `merge_append_into_write` - Start Date: 2023-08-02 - RFC PR: [apache/opendal#2758](https://github.com/apache/opendal/pull/2758) - Tracking Issue: [apache/opendal#2760](https://github.com/apache/opendal/issues/2760) # Summary Merge the `appender` API into `writer` by introducing a new `writer_with.append(true)` method to enable append mode. # Motivation Currently OpenDAL has separate `appender` and `writer` APIs: ```rust let mut appender = op.appender_with("file.txt").await?; appender.append(bs).await?; appender.append(bs).await?; ``` This duplication forces users to learn two different APIs for writing data. By adding this change, we can: - Simpler API surface - users only need to learn one writing API. - Reduce code duplication between append and write implementations. - Atomic append semantics are handled internally in `writer`. - Reuse the `sink` api for both `overwrite` and `append` mode. # Guide-level explanation The new approach is to enable append mode on `writer`: ```rust let mut writer = op.writer_with("file.txt").append(true).await?; writer.write(bs).await?; // appends to file writer.write(bs).await?; // appends again ``` Calling `writer_with.append(true)` will start the writer in append mode. Subsequent `write()` calls will append rather than overwrite. There is no longer a separate `appender` API. # Reference-level explanation We will add an `append` flag into `OpWrite`: ```rust impl OpWrite { pub fn with_append(mut self, append: bool) -> Self { self.append = append; self } } ``` All services need to check `append` flag and handle append mode accordingly. Services that not support append should return an `Unsupported` error instead. # Drawbacks - `writer` API is more complex with the append mode flag. - Internal implementation must handle both overwrite and append logic. # Rationale and alternatives None # Prior art Python's file open() supports an `"a"` mode flag to enable append-only writing. # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/2774_lister_api.md000064400000000000000000000023511046102023000171470ustar 00000000000000- Proposal Name: `lister_api` - Start Date: 2023-08-04 - RFC PR: [apache/opendal#2774](https://github.com/apache/opendal/pull/2774) - Tracking Issue: [apache/opendal#2775](https://github.com/apache/opendal/issues/2775) # Summary Add `lister` API to align with other OpenDAL APIs like `read`/`reader`. # Motivation Currently OpenDAL has `list` APIs like: ```rust let lister = op.list().await?; ``` This is inconsistent with APIs like `read`/`reader` and can confuse users. We should add a new `lister` API and change the `list` to: - Align with other OpenDAL APIs - Simplify usage # Guide-level explanation The new APIs will be: ```rust let entries = op.list().await?; // Get entries directly let lister = op.lister().await?; // Get lister ``` - `op.list()` returns entries directly. - `op.lister()` returns a lister that users can list entries on demand. # Reference-level explanation We will: - Rename existing `list` to `lister` - Add new `list` method to call `lister` and return all entries - Merge `scan` into `list_with` with `delimiter("")` This keeps the pagination logic encapsulated in `lister`. # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/2779_list_with_metakey.md000064400000000000000000000110241046102023000205430ustar 00000000000000- Proposal Name: `list_with_metakey` - Start Date: 2023-08-04 - RFC PR: [apache/opendal#2779](https://github.com/apache/opendal/pull/2779) - Tracking Issue: [apache/opendal#2802](https://github.com/apache/opendal/issues/2802) # Summary Move `Operator` `metadata` API to `list_with().metakey()` to simplify the usage. # Motivation The current `Entry` metadata API is: ```rust use opendal::Entry; use opendal::Metakey; let meta = op .metadata(&entry, Metakey::ContentLength | Metakey::ContentType) .await?; ``` This API is difficult to understand and rarely used correctly. And in reality, users always fetch the same set of metadata during listing. Take one of our users code as an example: ```rust let stream = self .inner .scan(&path) .await .map_err(|err| format_object_store_error(err, &path))?; let stream = stream.then(|res| async { let entry = res.map_err(|err| format_object_store_error(err, ""))?; let meta = self .inner .metadata(&entry, Metakey::ContentLength | Metakey::LastModified) .await .map_err(|err| format_object_store_error(err, entry.path()))?; Ok(format_object_meta(entry.path(), &meta)) }); Ok(stream.boxed()) ``` By moving metadata to `lister`, our user code can be simplified to: ```rust let stream = self .inner .scan_with(&path) .metakey(Metakey::ContentLength | Metakey::LastModified) .await .map_err(|err| format_object_store_error(err, &path))?; let stream = stream.then(|res| async { let entry = res.map_err(|err| format_object_store_error(err, ""))?; let meta = entry.into_metadata() Ok(format_object_meta(entry.path(), &meta)) }); Ok(stream.boxed()) ``` By introducing this change: - Users don't need to capture `Operator` in the closure. - Users don't need to do async call like `metadata()` again. If we don't have this change: - every place that could receive a `fn()` must use `Fn()` instead which enforce users to have a generic parameter in their code. - It's harder for other languages binding to implement `op.metadata()` right. # Guide-level explanation The new API will be: ```rust let entries: Vec = op .list_with("dir") .metakey(Metakey::ContentLength | Metakey::ContentType).await?; let meta: &Metadata = entries[0].metadata(); ``` Metadata can be queried directly when listing entries via `metadata()`, and later extracted via `into_parts()`. # Reference-level explanation ## How metakey works For every services, `stat` will return the full set of it's metadata. For example, `s3` will return `ContentLength | ContentType | LastModified | ...`, and `fs` will return `ContentLength | LastModified`. And most services will return part of those metadata during `list`. `s3` will return `ContentLength`, `LastModified`, but `fs` returns none of them. So when users use `list` to list entries, they will get a list of entries with incomplete metadata. The metadata could be in three states: - Filled: the metadata is returned in `list` - NotExist: the metadata is not supported by service. - Unknown: the metadata is supported by service but not returned in `list`. By accept `metakey`, we can compare the returning entry's metadata with metakey: - Return the entry if metakey already met by `Filled` and `NotExist`. - Send `stat` call to fetch the metadata if metadata is `Unknown`. ## Changes We will add `metakey` into `OpList`. Underlying services can use those information to try their best to fetch the metadata. There are following possibilities: - The entry metadata is met: `Lister` return the entry directly - The entry metadata is not met and not fully filled: `Lister` will try to send `stat` call to fetch the metadata - The entry metadata is not met and fully filled: `Lister` will return the entry directly. To make sure we can handle all metadata correctly, we will add a new capability called `stat_complete_metakey`. This capability will be used to indicate the complete set of metadata that can be fetched via `stat` call. For example, `s3` can set this capability to `ContentLength | ContentType | LastModified | ...`, and `fs` only have `ContentLength | LastModified`. `Lister` can use this capability to decide whether to send `stat` call or not. Services' lister implementation should not changed. # Drawbacks None # Rationale and alternatives Keeping the complex standalone API has limited benefit given low usage. # Prior art None # Unresolved questions None # Future possibilities ## Add glob and regex support for Lister We can add `glob` and `regex` support for `Lister` to make it more powerful. opendal-0.52.0/src/docs/rfcs/2852_native_capability.md000064400000000000000000000050531046102023000205020ustar 00000000000000- Proposal Name: `native_capability` - Start Date: 2023-08-11 - RFC PR: [apache/opendal#2852](https://github.com/apache/opendal/pull/2852) - Tracking Issue: [apache/opendal#2859](https://github.com/apache/opendal/issues/2859) # Summary Add `native_capability` and `full_capability` to `Operator` so that users can make more informed decisions. # Motivation OpenDAL adds `Capability` to inform users whether a service supports a specific feature. However, this is not enough for users to make decisions. OpenDAL doesn't simply expose the services' API directly; instead, it simulates the behavior to make it more useful. For example, `s3` doesn't support seek operations like a local file system. But it's a quite common operation for users. So OpenDAL will try to simulate the behavior by calculating the correct offset and reading the data from that offset instead. After this simulation, the `s3` service has the `read_can_seek` capability now. As another example, most services like `s3` don't support blocking operations. OpenDAL implements a `BlockingLayer` to make it possible. After this implementation, the `s3` service has the `blocking` capability now. However, these capabilities alone are insufficient for users to make informed decisions. Take the `s3` service's `blocking` capability as an example. Users are unable to determine whether it is a native capability or not, which may result in them unknowingly utilizing this feature in performance-sensitive scenarios, leading to significantly poor performance. So this proposal intends to address this issue by adding `native_capability` and `full_capability` to `OperatorInfo`. Users can use `native_capability` to determine whether a capability is native or not. # Guide-level explanation We will add two new APIs `native_capability()` and `full_capability()` in `OperatorInfo`, and remove the `capability()` and related `can_xxx()` API. ```diff + pub fn native_capability(&self) -> Capability + pub fn full_capability(&self) -> Capability - pub fn capability(&self) -> Capability ``` # Reference-level explanation We will add two new fields `native_capability` and `full_capability` in `AccessorInfo`: - Services SHOULD only set `native_capability`, and `full_capability` will be the same as `native_capability`. - Layers MAY change `full_capability` and MUST NOT modify `native_capability`. - `OperatorInfo` should forward `native_capability()` and `full_capability()` to `AccessorInfo`. # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/2884_merge_range_read_into_read.md000064400000000000000000000037611046102023000223160ustar 00000000000000- Proposal Name: `merge_range_read_into_read` - Start Date: 2023-08-20 - RFC PR: [apache/opendal#2884](https://github.com/apache/opendal/pull/2884) - Tracking Issue: [apache/opendal#2885](https://github.com/apache/opendal/issues/2885) # Summary Merge the `range_read` API into `read` by deleting the `op.range_reader(path, range)` and `op.range_read(path, range)` method. # Motivation Currently OpenDAL has separate `range_read` and `read` APIs: ```rust let bs = op.range_read("path/to/file", 1024..2048).await?; let bs = op.read("path/to/file").await?; ``` As same as `range_reader` and `reader` APIs: ```rust let reader = op.range_reader("path/to/file", 1024..2048).await?; let reader = op.reader("path/to/file").await?; ``` This duplication forces users to learn two different APIs for reading data. By adding this change, we can: - Simpler API surface - users only need to learn one writing API. - Reduce code duplication between read and range_read implementations. - Atomic read semantics are handled internally in `reader`. # Guide-level explanation There is no new approach to read data from file. The `read` and `reader` API supported range read by default. Calling `read_with("path/to/file").range(range)` will return a `reader` that supports range read. ```rust let bs = op.read_with("path/to/file").range(1024..2048).await?; ``` Calling `reader_with("path/to/file").range(range)` will return a `reader` that supports range read. ```rust let rs = op.reader_with("path/to/file").range(1024..2048).await?; ``` There is no longer a separate `range_read` and `range_reader` API. # Reference-level explanation None # Drawbacks None ## Breaking Changes This RFC has removed the `range_read` and `range_reader` APIs. If you have been using these APIs, you will need to reimplement your code by `op.read("path/to/file").range(1024..2048)` or `op.reader("path/to/file").range(1024..2048)`. # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/3017_remove_write_copy_from.md000064400000000000000000000057311046102023000215740ustar 00000000000000- Proposal Name: `remove_write_copy_from` - Start Date: 2023-09-06 - RFC PR: [apache/opendal#3017](https://github.com/apache/opendal/pull/3017) - Tracking Issue: [apache/opendal#3017](https://github.com/apache/opendal/issues/3017) # Summary Remove the `oio::Write::copy_from()` API pending a more thoughtful design. # Motivation In [RFC-2083: Writer Sink API](./2083_writer_sink_api.md), we launched an API, initially named `sink` and changed to `copy_from`, that enables data writing from a `Reader` to a `Writer` object. The current API signature is: ```rust pub trait Write: Unpin + Send + Sync { /// Copies data from the given reader to the writer. /// /// # Behavior /// /// - `Ok(n)` indicates successful writing of `n` bytes. /// - `Err(err)` indicates a failure, resulting in zero bytes written. /// /// A situation where `n < size` may arise; the caller should then transmit the remaining bytes until the full amount is written. async fn copy_from(&mut self, size: u64, src: oio::Reader) -> Result; } ``` The API has the following limitations: - Incompatibility with existing buffering and retry mechanisms. - Imposes ownership requirements on the reader, complicating its use. The reader must be recreated for every write operation. Due to restrictions in both Rust and Hyper's APIs, the following ideal implementation is currently unattainable: ```rust pub trait Write: Unpin + Send + Sync { async fn copy_from(&mut self, size: u64, src: &mut impl oio::Read) -> Result; } ``` - Rust doesn't allow us to have `impl oio::Read` in trait method if we want object safe. - hyper doesn't allow us to use reference here, it requires `impl Stream + 'static`. Given these constraints, the proposal is to remove `oio::Write::copy_from` until a more fitting design becomes feasible. # Guide-level explanation The `Writer::sink()` and `Writer::copy()` methods will be kept, but it's internal implementation will be changed to use `AsyncWrite` instead. For example: ```diff pub async fn copy(&mut self, size: u64, read_from: R) -> Result where R: futures::AsyncRead + Send + Sync + Unpin + 'static, { if let State::Idle(Some(w)) = &mut self.state { let r = Box::new(oio::into_streamable_read( oio::into_read_from_file(read_from, 0, size), 64 * 1024, )); - w.copy_from(size, r).await + futures::io::copy(&mut r, w).await } else { unreachable!( "writer state invalid while copy, expect Idle, actual {}", self.state ); } } ``` # Reference-level explanation The method `oio::Write::copy_from` will be removed. # Drawbacks The deprecation eliminates the ability to stream data uploads. A viable alternative is to directly use `AsyncWrite` offered by `Writer`. # Rationale and Alternatives N/A # Prior Art N/A # Unresolved Questions N/A # Future Possibilities Introduce utility functions such as `Writer::copy_from(r: &dyn AsyncRead)` when possible. opendal-0.52.0/src/docs/rfcs/3197_config.md000064400000000000000000000170001046102023000162560ustar 00000000000000- Proposal Name: `config` - Start Date: 2023-09-27 - RFC PR: [apache/opendal#3197](https://github.com/apache/opendal/pull/3197) - Tracking Issue: [apache/opendal#3240](https://github.com/apache/opendal/issues/3240) # Summary Expose services config to the user. # Motivation OpenDAL provides two ways to configure services: through a builder pattern and via a map. The `Builder` allows users to configure services using the builder pattern: ```rust // Create fs backend builder. let mut builder = Fs::default(); // Set the root for fs, all operations will happen under this root. builder.root("/tmp"); // Build an `Operator` to start operating the storage. let op: Operator = Operator::new(builder)?.finish(); ``` The benefit of builder is that it is type safe and easy to use. However, it is not flexible enough to configure services. Users must create a new builder for each service they wish to configure, translating user input into the API calls for each respective builder. Consider the following real-world example from one of our users: ```rust let mut builder = services::S3::default(); // Credential. builder.access_key_id(&cfg.access_key_id); builder.secret_access_key(&cfg.secret_access_key); builder.session_token(&cfg.session_token); builder.role_arn(&cfg.role_arn); builder.external_id(&cfg.external_id); // Root. builder.root(&cfg.root); // Disable credential loader if cfg.disable_credential_loader { builder.disable_config_load(); builder.disable_ec2_metadata(); } // Enable virtual host style if cfg.enable_virtual_host_style { builder.enable_virtual_host_style(); } ``` The `Map` approach allows users to configure services using a string-based HashMap: ```rust let map = HashMap::from([ // Set the root for fs, all operations will happen under this root. ("root".to_string(), "/tmp".to_string()), ]); // Build an `Operator` to start operating the storage. let op: Operator = Operator::via_map(Scheme::Fs, map)?; ``` This approach is simpler since it allows users to configure all services within a single map. However, it is not type safe and not easy to use. Users will need to convert their input to string and make sure the key is correct. And breaking changes could happen silently. This is one of our limitations: We need a way to configure services that is type safe, easy to use and flexible. The other one is that there is no way for users to fetch the config of a service after it's built. This limitation complicates the dynamic modification of a service's root path for the user. Our users have to wrap all our configs into an enum and store it in their own struct: ```rust pub enum StorageParams { Azblob(StorageAzblobConfig), Fs(StorageFsConfig), Ftp(StorageFtpConfig), Gcs(StorageGcsConfig), Hdfs(StorageHdfsConfig), Http(StorageHttpConfig), Ipfs(StorageIpfsConfig), Memory, Moka(StorageMokaConfig), Obs(StorageObsConfig), Oss(StorageOssConfig), S3(StorageS3Config), Redis(StorageRedisConfig), Webhdfs(StorageWebhdfsConfig), Cos(StorageCosConfig), } ``` So I propose to expose services config to the users, allowing them to work on config structs directly and fetch the config at runtime. # Guide-level explanation First of all, we will add config struct for each service. For example, `Fs` will have a `FsConfig` struct and `S3` will have a `S3Config`. The fields within the config struct are public and marked as non-exhaustive. ```rust #[non_exhaustive] pub struct S3Config { pub root: Option, pub bucket: String, pub endpoint: Option, pub region: Option, ... } ``` Then, we will add a `Config` enum that contains all the config structs. The enum is public and non-exhaustive too. ```rust #[non_exhaustive] pub enum Config { Fs(FsConfig) S3(S3Config), Custom(&'static str, HashMap), } ``` Notably, a `Custom` variant will be added to the enum. This variant aligns with `Scheme::Custom(name)` and allows users to configure custom services. At `Operator` level, we will add `from_config` and `via_config` methods. ```rust impl Operator { pub fn via_config(cfg: impl Into) -> Result {} } ``` Additionally, `OperatorInfo` will introduce a new API method, `config()`: ```rust impl OperatorInfo { pub fn config(&self) -> Config {} } ``` Users can use `config()` to fetch the config of a service at runtime and construct a new operator based on needs. # Reference-level explanation Every services will have a `XxxConfig` struct. `XxxConfig` will implement the following things: - `Default` trait: All config fields will have a default value. - `Deserialize` trait: Allow users to deserialize a config from a string. - `Into` trait: All service config can be converted to `Config` enum. Internally, `XxxConfig` will have the following traits: - `FromMap` trait: Allow users to build a config via hashmap which will replace existing `from_map` API in `Builder`. - `Into` trait: Config can convert into corresponding builder with zero cost. All config fields will be public and non-exhaustive, allowing users to build config this way: ```rust let s3 = S3Config { bucket: "test".to_string(), endpoint: Some("http://localhost:9000".to_string()), ..Default::default() } ``` The public API of existing builders will remain unchanged, although their internal implementations will be modified to utilize `XxxConfig`. Type that can't be represents as `String` like `Box` and `HttpClient` will be kept in `Builder` as before. For example: ```rust #[non_exhaustive] pub struct S3Config { pub root: Option, pub bucket: String, pub endpoint: Option, pub region: Option, ... } pub struct S3Builder { config: S3Config, customized_credential_load: Option>, http_client: Option, } ``` # Drawbacks ## API Surface This modification will significantly expand OpenDAL's public API surface, makes it harder to maintain and increases the risk of breaking changes. Also, this change will add much more work for bindings which need to implement `XxxConfig` for each service. ## Secrets Leakage After our config supports `Serialize`, it's possible that users will serialize the config and log it. This will lead to secrets leakage. We should add a warning in the docs to prevent this. And we should also encourage uses of services like AWS IAM instead of static secrets. # Rationale and alternatives ## Move `root` out of service config to operator level There is another way to solve the problem: [Move `root` out of service config to operator level](https://github.com/apache/opendal/issues/3151). We can move `root` out of the service config and put it in `Operator` level. This way, users can configure `root` for all services in one place. However, this is a large breaking changes and users will need to maintain the `root` logic everywhere. # Prior art None. # Unresolved questions None. # Future possibilities ## Implement `FromStr` for `Config` We can implement `FromStr` for `Config` so that users can parse a config from a string. ```rust let cfg = Config::from_str("s3://bucket/path/to/file?access_key_id=xxx&secret_access_key=xxx")?; ``` ## Implement `Serialize` for `Config` We can implement `Serialize` for `Config` so that users can serialize a config. ```rust // Serialize let bs = serde_json::to_vec(&cfg)?; // Deserialize let cfg: Config = serde_json::from_slice(&bs)?; ``` ## Implement `check` for `Config` Implement check for config so that users can check if a config is valid before `build`. opendal-0.52.0/src/docs/rfcs/3232_align_list_api.md000064400000000000000000000033061046102023000177610ustar 00000000000000- Proposal Name: `align_list_api` - Start Date: 2023-10-07 - RFC PR: [apache/opendal#3232](https://github.com/apache/opendal/pull/3232) - Tracking Issue: [apache/opendal#3236](https://github.com/apache/opendal/issues/3236) # Summary Refactor internal `Page` API to `List` API. # Motivation OpenDAL's `Lister` is implemented by `Page`: ```rust #[async_trait] pub trait Page: Send + Sync + 'static { /// Fetch a new page of [`Entry`] /// /// `Ok(None)` means all pages have been returned. Any following call /// to `next` will always get the same result. async fn next(&mut self) -> Result>>; } ``` Each call to `next` will retrieve a page of `Entry` objects. This design is modeled after the `list_object` API used in object storage services. However, this design has several drawbacks: - Services like `fs`, `hdfs` needs to buffer the whole page in memory before returning it to the caller. - `Page` is not aligned with `opendal::Lister` make it hard to understand the code. - `Page` is not aligned with `Read` & `Write` which is poll based. # Guide-level explanation No user-facing changes. # Reference-level explanation We will rename `Page` to `List` and change the API to: ```rust pub trait List: Send + Sync + 'static { /// Fetch a new [`Entry`] /// /// `Ok(None)` means all entries have been returned. Any following call /// to `next` will always get the same result. fn poll_next(&mut self, cx: &mut Context<'_>) -> Result>; } ``` All `page` related code will be replaced by `list`. # Drawbacks Breaking changes for raw API. # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/3243_list_prefix.md000064400000000000000000000150161046102023000173360ustar 00000000000000- Proposal Name: `list_prefix` - Start Date: 2023-10-08 - RFC PR: [apache/opendal#3243](https://github.com/apache/opendal/pull/3243) - Tracking Issue: [apache/opendal#3247](https://github.com/apache/opendal/issues/3247) # Summary Allow users to specify a prefix and remove the requirement that the path must end with `/`. # Motivation OpenDAL uses `/` to distinguish between a file and a directory. This design is necessary for object storage services such as S3 and GCS, where both `abc` (file) and `abc/` (directory) can coexist. We require users to provide the correct path to the API. For instance, when using `read("abc/")`, it returns `IsADirectory`, whereas with `list("abc/")` it returns `NotADirectory`. This behavior may be perplexing for users. As a side-effect of this design, OpenDAL always return exist for `stat("not_exist/")` since there is no way for OpenDAL to check if `not_exist/file_example` is exist via `HeadObject` call. There are some issues and pull requests related to those issues. - [Invalid metadata for dir objects in s3](https://github.com/apache/opendal/issues/3199) - [`is_exist` always return true for key end with '/', in S3 service](https://github.com/apache/opendal/issues/2086) POSIX-like file systems also have their own issues, as they lack native support for listing a prefix. Give file tree like the following: ```shell abc/ abc/def abc/xyz/ ``` Calling `list("ab")` will return `NotFound` after we removing the requirement that the path must end with `/`. So I propose the following changes of OpenDAL API behaviors: - Remove the requirement that the path for `list` must end with `/`. - Object storage services will use `list_object` API to check if a dir is exist. - Simulate the list prefix behavior for POSIX-like file systems. # Guide-level explanation Given the following file tree: ```shell abc/ abc/def_file abc/def_dir/ abc/def_dir/xyz_file abc/def_dir/xyz_dir/ ``` While listing a path: | Case | Path | Result | Description | |-------------------------|-----------------|-------------------------------------|-----------------------------------------| | list dir | `abc/` | `abc/def_file`
`abc/def_dir/` | children that matches the dir | | list prefix | `abc/def` | `abc/def_file`
`abc/def_dir/` | children that matches the prefix | | list file | `abc/def_file` | `abc/def_file` | the only children that matches the path | | list dir without `/` | `abc/def_dir` | `abc/def_dir/` | the only children that matches the path | | list file ends with `/` | `abc/def_file/` | EMPTY | no children matches the dir | | list not exist dir | `def/` | EMPTY | no children found matches the dir | | list not exist file | `def` | EMPTY | no children found matches the prefix | While listing a path with `delimiter` set to `""`: | Case | Path | Result | Description | |-------------------------|-----------------|-----------------------------------------------------------------------------------------------|-----------------------------------------| | list dir | `abc/` | `abc/def_file`
`abc/def_dir/`
`abc/def_dir/xyz_file`
`abc/def_dir/xyz_dir/` | children that matches the dir | | list prefix | `abc/def` | `abc/def_file`
`abc/def_dir/`
`abc/def_dir/xyz_file`
`abc/def_dir/xyz_dir/` | children that matches the prefix | | list file | `abc/def_file` | `abc/def_file` | the only children that matches the path | | list dir without `/` | `abc/def_dir` | `abc/def_dir/`
`abc/def_dir/xyz_file`
`abc/def_dir/xyz_dir/` | children that matches the path | | list file ends with `/` | `abc/def_file/` | EMPTY | no children matches the dir | | list not exist dir | `def/` | EMPTY | no children found matches the dir | | list not exist file | `def` | EMPTY | no children found matches the prefix | While stat a path: | Case | Path | Result | |------------------------|-----------------|--------------------------------------------| | stat existing dir | `abc/` | Metadata with dir mode | | stat existing file | `abc/def_file` | Metadata with file mode | | stat dir without `/` | `abc/def_dir` | Error `NotFound` or metadata with dir mode | | stat file with `/` | `abc/def_file/` | Error `NotFound` | | stat not existing path | `xyz` | Error `NotFound` | While create dir on a path: | Case | Path | Result | |-----------------------------|--------|----------------------------| | create dir on existing dir | `abc/` | Ok | | create dir on existing file | `abc` | Error with `NotADirectory` | | create dir with `/` | `xyz/` | Ok | | create dir without `/` | `xyz` | Ok with `xyz/` created | # Reference-level explanation For POSIX-like services, we will: - Simulate the list prefix behavior by listing the parent dir and filter the children that matches the prefix. - Return `NotFound` while stat an existing file with `/` For object storage services, we will: - Use `list_object` API while stat a path ends with `/`. - Return dir metadata if the dir is exist or there is at least a children. - Return `NotFound` if the dir is not exist and there is no children. - Check path before create dir with a path not ends with `/`. - Return `NotADirectory` if the path is exist. - Create the dir with `/` appended. # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/3356_lazy_reader.md000064400000000000000000000070621046102023000173160ustar 00000000000000- Proposal Name: `lazy_reader` - Start Date: 2023-10-22 - RFC PR: [apache/opendal#3356](https://github.com/apache/opendal/pull/3356) - Tracking Issue: [apache/opendal#3359](https://github.com/apache/opendal/issues/3359) # Summary Doing read IO in a lazy way. # Motivation The aim is to minimize IO cost. OpenDAL sends an actual IO request to the storage when `Accessor::read()` is invoked. For storage services such as S3, this equates to an IO request. However, in practical scenarios, users typically create a reader and use `seek` to navigate to the correct position. Take [parquet2 read_metadata](https://docs.rs/parquet2/latest/src/parquet2/read/metadata.rs.html) as an example: ```rust /// Reads a [`FileMetaData`] from the reader, located at the end of the file. pub fn read_metadata(reader: &mut R) -> Result { // check file is large enough to hold footer let file_size = stream_len(reader)?; if file_size < HEADER_SIZE + FOOTER_SIZE { return Err(Error::oos( "A parquet file must contain a header and footer with at least 12 bytes", )); } // read and cache up to DEFAULT_FOOTER_READ_SIZE bytes from the end and process the footer let default_end_len = min(DEFAULT_FOOTER_READ_SIZE, file_size) as usize; reader.seek(SeekFrom::End(-(default_end_len as i64)))?; ... deserialize_metadata(reader, max_size) } ``` In `read_metadata`, we initiate a seek as soon as the reader is invoked. This action, when performed on non-seekable storage services such as s3, results in an immediate HTTP request and cancellation. By postponing the IO request until the first `read` call, we can significantly reduce the number of IO requests. The expense of initiating and immediately aborting an HTTP request is significant. Here are the benchmark results, using a stat call as our baseline: On minio server that setup locally: ```rust service_s3_read_stat/4.00 MiB time: [315.23 µs 328.23 µs 341.42 µs] service_s3_read_abort/4.00 MiB time: [961.69 µs 980.68 µs 999.50 µs] ``` On remote storage services with high latency: ```rust service_s3_read_stat/4.00 MiB time: [407.85 ms 409.61 ms 411.39 ms] service_s3_read_abort/4.00 MiB time: [1.5282 s 1.5554 s 1.5828 s] ``` # Guide-level explanation There have been no changes to the API. The only modification is that the IO request has been deferred until the first `read` call, meaning no errors will be returned when calling `op.reader()`. For instance, users won't encounter a `file not found` error when invoking `op.reader()`. # Reference-level explanation Most changes will happen inside `CompleteLayer`. In the past, we will call `Accessor::read()` directly in `complete_reader`: ```rust async fn complete_reader( &self, path: &str, args: OpRead, ) -> Result<(RpRead, CompleteReader)> { .. let seekable = capability.read_can_seek; let streamable = capability.read_can_next; let range = args.range(); let (rp, r) = self.inner.read(path, args).await?; let content_length = rp.metadata().content_length(); ... } ``` In the future, we will postpone the `Accessor::read()` request until the first `read` call. # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities ## Add `read_at` for `oio::Reader` After `oio::Reader` becomes zero cost, we can add `read_at` to `oio::Reader` to support read data by range. opendal-0.52.0/src/docs/rfcs/3526_list_recursive.md000064400000000000000000000040421046102023000200510ustar 00000000000000- Proposal Name: `list_recursive` - Start Date: 2023-11-08 - RFC PR: [apache/opendal#3526](https://github.com/apache/opendal/pull/3526) - Tracking Issue: [apache/opendal#0000](https://github.com/apache/opendal/issues/0000) # Summary Use `recursive` to replace `delimiter`. # Motivation OpenDAL add `delimiter` in `list` to allow users to control the list behavior: - `delimiter == "/"` means use `/` as delimiter of path, it behaves like list current dir. - `delimiter == ""` means don't set delimiter of path, it behaves like list current dir and all it's children. Ideally, we should allow users to input any delimiter such as `|`, `-`, and `+`. The `delimiter` concept can be challenging for users unfamiliar with object storage services. Currently, only `/` and empty spaces are accepted as delimiters, despite not being fully implemented across all services. We need to inform users that `delimiter == "/"` is used to list the current directory, while `delimiter == ""` is used for recursive listing. This may not be immediately clear. So, why not use `recursive` directly for more clear API behavior? # Guide-level explanation OpenDAL will use `recursive` to replace `delimiter`. Default behavior is not changed, so users that using `op.list()` is not affected. For users who is using `op.list_with(path).delimiter(delimiter)`: - `op.list_with(path).delimiter("")` -> `op.list_with(path).recursive(true)` - `op.list_with(path).delimiter("/")` -> `op.list_with(path).recursive(false)` - `op.list_with(path).delimiter(other_value)`: not supported anymore. # Reference-level explanation We will add `recursive` as a new arg in `OpList` and remove all fields related to `delimiter`. # Drawbacks ## Can't support to use `|`, `-`, and `+` as delimiter We never support this feature before. # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities ## Add delete with recursive support Some services have native support for delete with recursive, such as azfile. We can add this feature in the future if needed. opendal-0.52.0/src/docs/rfcs/3574_concurrent_stat_in_list.md000064400000000000000000000100501046102023000217440ustar 00000000000000- Proposal Name: `concurrent_stat_in_list` - Start Date: 2023-11-13 - RFC PR: [apache/opendal#3574](https://github.com/apache/opendal/pull/3574) - Tracking Issue: [apache/opendal#3575](https://github.com/apache/opendal/issues/3575) # Summary Add concurrent stat in list operation. # Motivation [RFC-2779](https://github.com/apache/opendal/pull/2779) allows user to list with metakey. However, the stat inside list could make the list process much slower. We should allow concurrent stat during list so that stat could be sent concurrently. # Guide-level explanation For users who want to concurrently run statistics in list operations, they will call the new API `concurrent`. The `concurrent` function will take a number as input, and this number will represent the maximum concurrent stat handlers. The default behavior remains unchanged, so users using `op.list_with()` are not affected. And this implementation should be zero cost to users who don't want to do concurrent stat. ```rust op.lister_with(path).metakey(Metakey::ContentLength).concurrent(10).await? ``` # Reference-level explanation When `concurrent` is set and `list_with` is called, the list operation will be split into two parts: list and stat. The list part will iterate through the entries inside the buffer, and if its `metakey` is unknown, it will send a stat request to the storage service. We will add a new field `concurrent` to `OpList`. The type of `concurrent` is `Option`. If `concurrent` is `None`, it means the default behavior. If `concurrent` is `Some(n)`, it means the maximum concurrent stat handlers are `n`. Then we could use a sized `VecDeque` to limit the maximum concurrent stat handlers. Additionally, we could use handlers `JoinHandle` inside `VecDeque` to spawn and queue the stat tasks. While iterating through the entries, we should check if the `metakey` is unknown and if the `VecDeque` is full. If the `metakey` is unknown and the `VecDeque` is full, we should wait and join the handle once it’s finished, since we need to keep the entry order. If the metakey is unknown and the handlers are full, we should break the loop and wait for the spawned tasks inside handlers to finish. After the spawned tasks finish, we should iterate through the handlers and return the result. If the metakey is known, we should check if the handlers are empty. If true, return the result immediately; otherwise, we should wait for the spawned tasks to finish. # Drawbacks 1. More memory usage 2. More complex code 3. More complex testing # Rationale and alternatives ## Why not `VecDeque>`? To maintain the order of returned entries, we need to pre-run future entries before returning the current one to address the slowness issue. Although we could use `VecDeque>` to store the spawned tasks, using it here would prevent us from executing the async block concurrently when we only have one `cx: &mut Context<'_>`. ## Do we need `Semaphore`? No, we can control the concurrent number by limiting the length of the `VecDeque`. Using a `semaphore` will introduce more cost and memory. ## Why not using `JoinSet`? The main reason is that `JoinSet` can't maintain the order of entries. The other reason is that `JoinSet` requires mutability to spawn or join the next task, and `tokio::spawn()` requires the async block to be `'static`. This implies that we need to use `Arc` to wrap our `JoinSet`. However, to change the value inside `Arc`, we need to introduce a `Mutex`. Since it's inside an async block, we need to use Tokio's `Mutex` to satisfy the `Sync` bound. Therefore, for every operation on the `JoinSet`, there will be an `.await` on the lock outside the async block, making concurrency impossible inside `poll_next()`. # Prior art None # Unresolved questions - How to implement a similar logic to `blocking` API? - Quoted from [oowl](https://github.com/oowl): It seems these features can be implemented in blocking mode, but it may require breaking something in OpenDAL, such as using some pthread API in blocking mode. # Future possibilities None opendal-0.52.0/src/docs/rfcs/3734_buffered_reader.md000064400000000000000000000060311046102023000201140ustar 00000000000000- Proposal Name: `buffered_reader` - Start Date: 2023-12-10 - RFC PR: [apache/opendal#3574](https://github.com/apache/opendal/pull/3734) - Tracking Issue: [apache/opendal#3575](https://github.com/apache/opendal/issues/3735) # Summary Allowing the underlying reader to fetch data at the buffer's size to amortize the IO's overhead. # Motivation The objective is to mitigate the IO overhead. In certain scenarios, the reader processes the data incrementally, meaning that it utilizes the `seek()` function to navigate to a specific position within the file. Subsequently, it invokes the `read()` to reads `limit` bytes into memory and performs the decoding process. OpenDAL triggers an IO request upon invoking `read()` if the `seek()` has reset the inner state. For storage services like S3, [research](https://www.vldb.org/pvldb/vol16/p2769-durner.pdf) suggests that an optimal IO size falls within the range of 8MiB to 16MiB. If the IO size is too small, the Time To First Byte (TTFB) dominates the overall runtime, resulting in inefficiency. Therefore, this RFC proposes the implementation of a buffered reader to amortize the overhead of IO. # Guide-level explanation For users who want to buffered reader, they will call the new API `buffer`. And the default behavior remains unchanged, so users using `op.reader_with()` are not affected. The `buffer` function will take a number as input, and this number will represent the maximum buffer capability the reader is able to use. ```rust op.reader_with(path).buffer(32 * 1024 * 1024).await ``` # Reference-level explanation This feature will be implemented in the `CompleteLayer`, with the addition of a `BufferReader` struct in `raw/oio/reader/buffer_reader.rs`. The `BufferReader` employs a `tokio::io::ReadBuf` as the inner buffer and uses `offset: Option` to track the buffered range start of the file, the buffered data should always be `file[offset..offset + buf.len()]`. ```rust ... async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { BufferReader::new(self.complete_read(path, args).await) } ... fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { BufferReader::new(self.complete_blocking_read(path, args)) } ... ``` A `buffer` field of type `Option` will be introduced to `OpRead`. If `buffer` is set to `None`, it functions with default behavior. However, if buffer is set to `Some(n)`, it denotes the maximum buffer capability that the `BufferReader` can utilize. The behavior is similar to [std::io::BufReader](https://doc.rust-lang.org/std/io/struct.BufReader.html), with the difference being that our implementation always provides the `seek_relative` (without discarding the inner buffer) if it's available; And it doesn't buffer trailing reads when the read range is smaller than the buffer capability. # Drawbacks None # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities - Concurrent fetching. - Tailing buffering. opendal-0.52.0/src/docs/rfcs/3898_concurrent_writer.md000064400000000000000000000067561046102023000206170ustar 00000000000000- Proposal Name: `concurrent_writer` - Start Date: 2024-01-02 - RFC PR: [apache/opendal#3898](https://github.com/apache/opendal/pull/3898) - Tracking Issue: [apache/opendal#3899](https://github.com/apache/opendal/issues/3899) # Summary Enhance the `Writer` by adding concurrent write capabilities. # Motivation Certain services, such as S3, GCS, and AzBlob, offer the `multi_write` functionality, allowing users to perform multiple write operations for uploading of large files. If a service support `multi_write`, the [Capability::write_can_multi](https://opendal.apache.org/docs/rust/opendal/struct.Capability.html#structfield.write_can_multi) metadata should be set to `true`. ```rust let mut writer = op.writer("path/to").await?; // a writers supports the `multi_write`. writer.write(part0).await?; writer.write(part1).await?; // It starts to upload after the `part0` is finished. writer.close().await?; ``` Currently, when invoking a `Writer` that supports the `multi_write` functionality, multiple writes are proceed serially, without fully leveraging the potential for improved throughput through concurrent uploads. We should enhance support to allow concurrent processing of multiple write operations. # Guide-level explanation For users who want to concurrent writer, they will call the new API `concurrent`. And the default behavior remains unchanged, so users using `op.writer_with()` are not affected. The `concurrent` function will take a number as input, and this number will represent the maximum concurrent write task amount the writer can perform. - If `concurrent` is set to 0 or 1, it functions with default behavior(writes serially). - However, if `concurrent` is set to number larger than 1. It enables concurrent uploading of up to `concurrent` write tasks and allows users to initiate additional write tasks without waiting to complete the previous write operation, as long as the inner task queue still has available slots. The concurrent write feature operate independently of other features. ```rust let mut w = op.writer_with(path).concurrent(8).await; w.write(part0).await?; w.write(part1).await?; // `write` won't wait for part0. w.close().await?; // `close` will make sure all parts are finished. ``` # Reference-level explanation The S3 and similar services use `MultipartUploadWriter`, while GCS uses `RangeWriter`. We can enhance these services by adding concurrent write features to them. A `concurrent` field of type `usize` will be introduced to `OpWrite` to allow the user to set the maximum concurrent write task amount. For other services that don't support `multi_write`, setting the concurrent parameter will have no effect, maintaining the default behavior. This feature will be implemented in the `MultipartUploadWriter` and `RangeWriter`, which will utilize a `ConcurrentFutures` as a task queue to store concurrent write tasks. When the upper layer invokes `poll_write`, the `Writer` pushes write to the task queue (`ConcurrentFutures`) if there are available slots, and then relinquishes control back to the upper layer. This allows for up to `concurrent` write tasks to uploaded concurrently without waiting. If the task queue is full, the `Writer` waits for the first task to yield results. # Drawbacks - More memory usage - More concurrent connections # Rationale and alternatives None # Prior art None # Unresolved questions None # Future possibilities - Adding `write_at` for `fs`. - Use `ConcurrentFutureUnordered` instead of `ConcurrentFutures.` opendal-0.52.0/src/docs/rfcs/3911_deleter_api.md000064400000000000000000000112201046102023000172560ustar 00000000000000- Proposal Name: `deleter_api` - Start Date: 2024-01-04 - RFC PR: [apache/opendal#3911](https://github.com/apache/opendal/pull/3911) - Tracking Issue: [apache/opendal#3922](https://github.com/apache/opendal/issues/3922) # Summary Introduce the `Deleter` API to enhance batch and recursive deletion capabilities. # Motivation All OpenDAL's public API follow the same design: - `read`: Execute a read operation. - `read_with`: Execute a read operation with additional options, like range and if_match. - `reader`: Create a reader for streaming data, enabling flexible access. - `reader_with`: Create a reader with advanced options. However, `delete` operations vary. OpenDAL offers several methods for file deletion: - `delete`: Delete a single file or an empty dir. - `remove`: Remove a list of files. - `remove_via`: Remove files produced by a stream. - `remove_all`: Remove all files under a path. This design is not consistent with the other APIs, and it is not easy to use. So I propose `Deleter` to address them all at once. # Guide-level explanation The following new API will be added to `Operator`: ```diff impl Operator { pub async fn detele(&self, path: &str) -> Result<()>; + pub fn delete_with(&self, path: &str) -> FutureDelete; + pub async fn deleter(&self) -> Result; + pub fn deleter_with(&self) -> FutureDeleter; } ``` - `delete` is the existing API, which deletes a single file or an empty dir. - `delete_with` is an extension of the existing `delete` API, which supports additional options, such as `version`. - `deleter` is a new API that returns a `Deleter` instance. - `deleter_with` is an extension of the existing `deleter` API, which supports additional options, such as `concurrent`. The following new options will be available for `delete_with` and `deleter_with`: - `concurrent`: How many delete tasks can be performed concurrently? - `buffer`: How many files can be buffered for send in a single batch? Users can delete multiple files in this way: ```rust let deleter = op.deleter().await?; // Add a single file to the deleter. deleter.delete(path).await?; // Add a stream of files to the deleter. deleter.delete_all(&mut lister).await?; // Close deleter, make sure all input files are deleted. deleter.close().await?; ``` `Deleter` also implements [`Sink`](https://docs.rs/futures/latest/futures/sink/trait.Sink.html), so all the methods of `Sink` are available for `Deleter`. For example, users can use [`forward`](https://docs.rs/futures/latest/futures/stream/trait.StreamExt.html#method.forward) to forward a stream of files to `Deleter`: ```rust // Init a deleter to start batch delete tasks. let deleter = op.deleter().await?; // List all files that ends with tmp let lister = op.lister(path).await? .filter(|x|future::ready(x.ends_with(".tmp"))); // Forward all paths into deleter. lister.forward(deleter).await?; // Send all from a stream into deleter. deleter.send_all(&mut lister).await?; // Close the deleter. deleter.close().await?; ``` Users can control the behavior of `Deleter` by setting the options: ```rust let deleter = op.deleter_with() // Allow up to 8 concurrent delete tasks, default to 1. .concurrent(8) // Configure the buffer size to 1000, default value provided by services. .buffer(1000) .await?; // Add a single file to the deleter. deleter.delete(path).await?; // Add a stream of files to the deleter. deleter.delete_all(&mut lister).await?; // Close deleter, make sure all input files are deleted. deleter.close().await?; ``` In response to `Deleter` API, we will remove APIs like `remove`, `remove_via` and `remove_all`. - `remove` and `remove_via` could be replaced by `Deleter` directly. - `remove_all` could be replaced by `delete_with(path).recursive(true)`. # Reference-level explanation To provide those public APIs, we will add a new associated type in `Accessor`: ```rust trait Accessor { ... type Deleter = oio::Delete; type BlockingDeleter = oio::BlockingDelete; } ``` And the `delete` API will be changed to return a `oio::Delete` instead: ```diff trait Accessor { - async fn delete(&self) -> Result<(RpDelete, Self::Deleter)>; + async fn delete(&self, args: OpDelete) -> Result<(RpDelete, Self::Deleter)>; } ``` Along with this change, we will remove the `batch` API from `Accessor`: ```rust trait Accessor { - async fn batch(&self, args: OpBatch) -> Result; } ``` # Drawbacks - Big breaking changes. # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities ## Add API that accepts `IntoIterator` It's possible to add a new API that accepts `IntoIterator` so users can input `Vec` or `Iter` into `Deleter`. opendal-0.52.0/src/docs/rfcs/4382_range_based_read.md000064400000000000000000000174011046102023000202400ustar 00000000000000- Proposal Name: `range_based_read` - Start Date: 2024-03-20 - RFC PR: [apache/opendal#4382](https://github.com/apache/opendal/pull/4382) - Tracking Issue: [apache/opendal#4383](https://github.com/apache/opendal/issues/4383) # Summary Convert `oio::Read` into a stateless, range-based reading pattern. # Motivation The current `oio::Read` API is stateful: ```rust pub trait Read: Unpin + Send + Sync { fn read(&mut self, limit: usize) -> impl Future> + Send; fn seek(&mut self, pos: io::SeekFrom) -> impl Future> + Send; } ``` Users use `read` to retrieve data from storage and can use `seek` to navigate to specific positions. OpenDAL manages the underlying state. This design is good for users from `std::io::Read`, `futures::AsyncRead` and `tokio::io::AsyncRead`. OpenDAL also provides `range` option at the `Operator` level for users to read a specific range of data. The most common usage will be like: ```rust let r: Reader = op.reader_with(path).range(1024..2048).await?; ``` However, after observing our users, we found that: - `AsyncSeek` in `Reader` is prone to misuse. - `Reader` does not support concurrent reading. - `Reader` can't adopt Completion-based IO ## Misuse of `AsyncSeek` When designing `Reader`, I expected users to check the `read_can_seek` capability to determine if the underlying storage services natively support `seek`. However, many users are unaware of this and directly use `seek`, leading to suboptimal performance. For example, `s3` storage does not support `seek` natively. When users call `seek`, opendal will drop current reader and sending a new request. This behavior is hidden from users and can lead to unexpected performance issues like [What's going on in my parquet stream](https://github.com/apache/opendal/issues/3725). ## Lack of concurrent reading `oio::Read` complicates supporting concurrent reading. Users must implement a feature similar to merge IO, as discussed in [support merge io read api by settings](https://github.com/apache/opendal/issues/3675). There is no way for opendal to support this feature. ## Can't adopt Completion-based IO Completion-based IO requires take the buffer's owner ship. But API that take `&mut [u8]` can't do that. # Guide-level explanation So I propose to convert `Reader` into a stateless, range-based reading pattern. We will remove the following `impl` from `Reader`: - `futures::AsyncRead` - `futures::AsyncSeek` - `futures::Stream` - `tokio::AsyncRead` - `tokio::AsyncSeek` We will add the following new APIs to `Reader`: ```rust impl Reader { /// Read data from the storage at the specified offset. pub async fn read(&self, buf: &mut impl BufMut, offset: u64, limit: usize) -> Result; /// Read data from the storage at the specified range. pub async fn read_range( &self, buf: &mut impl BufMut, range: impl RangeBounds, ) -> Result; /// Read all data from the storage into given buf. pub async fn read_to_end(&self, buf: &mut impl BufMut) -> Result; /// Copy data from the storage into given writer. pub async fn copy(&mut self, write_into: &mut impl futures::AsyncWrite) -> Result; /// Sink date from the storage into given sink. pub async fn sink(&mut self, sink_from: &mut S) -> Result where S: futures::Sink, T: Into, } ``` Apart from `Reader`'s own API, we will also provide convert to existing IO APIs like: ```rust impl Reader { /// Convert Reader into `futures::AsyncRead` pub fn into_futures_io_async_read(self, range: Range) -> FuturesIoAsyncReader; /// Convert Reader into `futures::Stream` pub fn into_futures_bytes_stream(self, range: Range) -> FuturesBytesStream; } ``` After this change, users will be able to use `Reader` to read data from storage in a stateless, range-based pattern. Users can also convert `Reader` into `futures::AsyncRead`, `futures::AsyncSeek` and `futures::Stream` as needed. # Reference-level explanation The new raw API will be: ```rust pub trait Read: Unpin + Send + Sync { fn read_at( &self, offset: u64, limit: usize, ) -> impl Future> + Send; } ``` The API is similar to [`ReadAt`](https://doc.rust-lang.org/std/fs/struct.File.html#method.read_at), but with following changes: ```diff - async fn read_at(&self, buf: &mut [u8], offset: u64) -> Result + async fn read_at(&self, offset: u64, limit: usize) -> Result ``` - opendal chooses to use `oio::Buffer` instead of `&mut [u8]` to avoid lifetime issues. - opendal chooses to return `oio::Buffer` to let services itself manage the buffer. For example, http based storage services like `s3` is a stream that generating data on the fly. # Drawbacks ## Breaking changes to `Reader` This change will break the existing `Reader` API. Users will need to update their code to use the new `Reader` API. Users wishing to migrate to the new range-based API will need to update their code. Those who simply want to use `futures::AsyncRead` can instead utilize `Reader::into_futures_read`. # Rationale and alternatives None. # Prior art ## `object_store`'s API design Current API design inspired from `object_store`'s `ObjectStore` a lot: ```rust #[async_trait] pub trait ObjectStore: std::fmt::Display + Send + Sync + Debug + 'static { /// Return the bytes that are stored at the specified location. async fn get(&self, location: &Path) -> Result { self.get_opts(location, GetOptions::default()).await } /// Perform a get request with options async fn get_opts(&self, location: &Path, options: GetOptions) -> Result; /// Return the bytes that are stored at the specified location /// in the given byte range. /// /// See [`GetRange::Bounded`] for more details on how `range` gets interpreted async fn get_range(&self, location: &Path, range: Range) -> Result { let options = GetOptions { range: Some(range.into()), ..Default::default() }; self.get_opts(location, options).await?.bytes().await } /// Return the bytes that are stored at the specified location /// in the given byte ranges async fn get_ranges(&self, location: &Path, ranges: &[Range]) -> Result> { coalesce_ranges( ranges, |range| self.get_range(location, range), OBJECT_STORE_COALESCE_DEFAULT, ) .await } } ``` We can add support that similar to `get_ranges` in the future. OpenDAL opts to return a `Reader` rather than directly implementing `read` to allow for optimization with storage services like `fs` to reduce the extra `open` syscall. # Unresolved questions ## Buffer After switching to range-based reading, we can no longer keep a buffer within the reader. As of writing this proposal, users should use `into_async_buf_read` instead. # Future possibilities ## Read Ranges We can implement `read_ranges` support in the future. This will allow users to read multiple ranges of data in less requests. ## Native `read_at` for fs and hdfs We can reduce unnecessary `open` and `seek` syscalls by using the `read_at` API across different platforms. ## Auto Range Read We can implement [Auto ranged read support](https://github.com/apache/opendal/issues/1105) like AWS S3 Crt Client. For examples, split the range into multiple ranges and read them concurrently. Services can define the preferred io size as default, and users can override it. For example, s3 can use `8 MiB` as preferred io size, while fs can use `4 KiB` instead. ## Completion-based IO `oio::Read` is designed with Completion-based IO in mind. We can add IOCP/io_uring support in the future. opendal-0.52.0/src/docs/rfcs/4638_executor.md000064400000000000000000000157541046102023000166660ustar 00000000000000- Proposal Name: `executor` - Start Date: 2024-05-23 - RFC PR: [apache/opendal#4638](https://github.com/apache/opendal/pull/4638) - Tracking Issue: [apache/opendal#4639](https://github.com/apache/opendal/issues/4639) # Summary Add executor in opendal to allow running tasks concurrently in background. # Motivation OpenDAL offers top-tier support for concurrent execution, allowing tasks to run simultaneously in the background. Users can easily enable concurrent file read/write operations with just one line of code: ```diff let mut w = op .writer_with(path) .chunk(8 * 1024 * 1024) // 8 MiB per chunk + .concurrent(16) // 16 concurrent tasks .await?; w.write(bs).await?; w.write(bs).await?; // The submitted tasks only be executed while user calling `write`. ... sleep(Duration::from_secs(10)).await; // The submitted tasks make no progress during `sleep`. ... w.close().await?; ``` However, the execution of those tasks relies on users continuously calling `write`. They cannot run tasks concurrently in the background. (I explained the technical details in the `Rationale and alternatives` section.) This can result in the following issues: - Task latency may increase as tasks are not executed until the task queue is full. - Memory usage may be high because all chunks must be held in memory until the task is completed. I propose introducing an executor abstraction in OpenDAL to enable concurrent background task execution. The executor will automatically manage the tasks in the background without requiring users to drive the progress manually. # Guide-level explanation OpenDAL will add a new `Executor` struct to manage concurrent tasks. ```rust pub struct Executor { ... } pub struct Task { ... } impl Executor { /// Create a new tokio based executor. pub fn new() -> Self { ... } /// Create a new executor with given execute impl. pub fn with(exec: Arc) -> Self { ... } /// Run given future in background immediately. pub fn execute(&self, f: F) -> Task where F: Future + Send + 'static, { ... } } ``` The `Executor` uses the `tokio` runtime by default but users can also provide their own runtime by: ```rust pub trait Execute { fn execute(&self, f: BoxedFuture<()>) -> Result<()>; } ``` Users can set executor in `OpWrite` / `OpRead` to enable concurrent background task execution: ```rust + let exec = Executor::new(); let w = op .writer_with(path) .chunk(8 * 1024 * 1024) // 8 MiB per chunk .concurrent(16) // 16 concurrent tasks + .executor(exec) // Use specified executor .await?; ``` Specifying an executor every time is cumbersome. Users can also set a global executor for given operator: ```rust + let exec = Executor::new(); + let op = op.with_default_executor(exec); let w = op .writer_with(path) .chunk(8 * 1024 * 1024) // 8 MiB per chunk .concurrent(16) // 16 concurrent tasks .await?; ``` # Reference-level explanation As mentioned in the `Guide-level explanation`, the `Executor` struct will manage concurrent tasks in the background. `Executor` will be powered by trait `Execute` to support different underlying runtimes. To make trait `Execute` object safe, we only accept `BoxedFuture<()>` as input. `Executor` will handle the future output and return the result to the caller. Operations that supporting concurrent execution will add a new field: ```rust pub struct OpXxx { ... executor: Option, } ``` Operator will add a new field to store the default executor: ```rust pub struct Operator { ... default_executor: Option, } ``` The `Task` spawned by `Executor` will be a future that can be awaited to fetch the result: ```rust let res = task.await; ``` The task will be executed immediately after calling `execute`. Users can also cancel the task by dropping the `Task` object. Users don't need to poll those `Task` object to make progress. # Drawbacks ## Complexity To support concurrent execution, we need to introduce: - a new `Executor` struct - a new `Task` struct - a new `Execute` trait This may increase the complexity of the codebase. # Rationale and alternatives ## Why introducing so many new abstractions? We need to introduce new abstractions to support concurrent execution across different runtimes. Unfortunately, this is the current reality of async rust. Supporting just one or two runtimes by adding features is much easier. Supporting only Tokio is extremely simple, requiring about 10 lines of changes. However, this violates our vision of free data access. Firstly, we don't want to force our users to use Tokio. We aim to support all runtimes, including async-std, smol, and others. Secondly, OpenDAL should be capable of running in any environment, including embedded systems. We don’t want to restrict our users to a specific runtime. Finally, users may have their own preferences for observability and performance in their runtime. We intend to accommodate these needs effortlessly. ## Why `ConcurrentFutures` doesn't work? `ConcurrentFutures` is a `Vec`, users need to keep calling `poll_next` to make progress. This is not suitable for our use case. We need a way to run tasks in the background without user intervention. > I've heard that futures will wake up when they're ready, and it's the runtime's job to poll them, right? No, it's partially correct. The runtime will wake up the future when it's ready, but it's the user's job to poll the future. The runtime will not poll the future automatically unless it's managed by the runtime. For tokio, that means all futures provided by tokio, like `tokio::time::Sleep`, will be polled by tokio runtime. However, if you create a future by yourself, you need to poll it manually. I have an example to explain this: *Try it at [playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=628e67adef90128151e175d22c87808e)* ```rust use futures::stream::FuturesUnordered; use futures::StreamExt; use std::time::Duration; use tokio::time::{sleep, Instant}; #[tokio::main] async fn main() { let now = Instant::now(); let mut cf = FuturesUnordered::new(); // cf.push(Box::pin(sleep(Duration::from_secs(3)))); cf.push(Box::pin(async move { sleep(Duration::from_secs(3)).await; println!("async task finished at {}s", now.elapsed().as_secs_f64()); })); sleep(Duration::from_secs(4)).await; println!("outer sleep finished at {}s", now.elapsed().as_secs_f64()); let _: Vec<()> = cf.collect().await; println!("consumed: {}s", now.elapsed().as_secs_f64()) } ``` # Prior art None. # Unresolved questions None. # Future possibilities ## Blocking Executor This proposal mainly focuses on async tasks. However, we can also consider adding blocking support to `Executor`. Users can use concurrent tasks in blocking context too: ```rust let w = op .writer_with(path) .chunk(8 * 1024 * 1024) // 8 MiB per chunk + .concurrent(16) // 16 concurrent tasks .do()?; ``` opendal-0.52.0/src/docs/rfcs/5314_remove_metakey.md000064400000000000000000000070141046102023000200220ustar 00000000000000- Proposal Name: `remove_metakey` - Start Date: 2024-11-12 - RFC PR: [apache/opendal#5313](https://github.com/apache/opendal/pull/5313) - Tracking Issue: [apache/opendal#5314](https://github.com/apache/opendal/issues/5314) # Summary Remove the `Metakey` concept from OpenDAL and replace it with a simpler and more predictable metadata handling mechanism. # Motivation The current `Metakey` design has several issues: 1. Performance Impact: Users often initiate costly operations unintentionally, such as using `Metakey::Full`, which results in extra stat calls 2. Usability Issues: Users often try to access metadata that hasn't been explicitly requested 3. API Confusion: There's a conflict between `Metakey::Version` and the new `version(bool)` parameter 4. Implementation Complexity: Service developers struggle to implement `Metakey` correctly The goal is a simpler, more intuitive API that prevents common mistakes and improves performance as standard. # Guide-level explanation Instead of using `Metakey` to specify which metadata fields to fetch, services will now declare their metadata capabilities upfront through a new `MetadataCapability` struct: ```rust let entries = op.list("path").await?; for entry in entries { if op.metadata_capability().content_type { println!("Content-Type: {}", entry.metadata().content_type()); } } ``` If users need additional metadata not provided by `list`: ```rust let entries = op.list("path").await?; for entry in entries { let mut meta = entry.metadata(); if !op.metadata_capability().etag { meta = op.stat(&entry.path()).await?; } println!("Content-Type: {}", meta.etag()); } ``` For existing OpenDAL users, the main changes are: - Remove all `metakey()` calls from their code - Use `metadata_capability()` to check available metadata - Explicitly call `stat()` when needed # Reference-level explanation The implementation involves: 1. Remove the `Metakey` enum 2. Add new `MetadataCapability` struct: ```rust pub struct MetadataCapability { pub content_length: bool, pub content_type: bool, pub last_modified: bool, pub etag: bool, pub mode: bool, pub version: bool, ... } ``` 3. Add method to Operator to query capabilities: ```rust impl Operator { pub fn metadata_capability(&self) -> MetadataCapability; } ``` 4. Modify list operation to avoid implicit stat calls 5. Update all service implementations to declare their metadata capabilities Each service implementation will need to: - Remove `Metakey` handling logic - Implement `metadata_capability()` to accurately indicate the metadata provided by default - Ensure list operations return metadata that's always available without extra API calls # Drawbacks - Breaking change for existing users - Loss of fine-grained control over metadata fetching - Potential increased API calls if users need multiple metadata fields # Rationale and alternatives This design is superior because: - Prevents performance pitfalls by default - Makes metadata availability explicitly - Simplifies service implementation - Provides clearer mental model Alternatives considered: 1. Keep `Metakey` but make it more restrictive 2. Add warnings for potentially costly operations 3. Make stat calls async/lazy Not making this change would continue the current issues of performance problems and API misuse. # Prior art None # Unresolved questions None # Future possibilities - Add metadata prefetching optimization - Add metadata caching layer - Support for custom metadata fields - Automated capability detection opendal-0.52.0/src/docs/rfcs/5444_operator_from_uri.md000064400000000000000000000114571046102023000205550ustar 00000000000000- Proposal Name: `operator_from_uri` - Start Date: 2024-12-23 - RFC PR: [apache/opendal#5444](https://github.com/apache/opendal/pull/5444) - Tracking Issue: [apache/opendal#5445](https://github.com/apache/opendal/issues/5445) # Summary This RFC proposes adding URI-based configuration support to OpenDAL, allowing users to create operators directly from URIs. The proposal introduces a new `from_uri` API in both the `Operator` and `Configurator` traits, along with an `OperatorRegistry` to manage operator factories. # Motivation Currently, creating an operator in OpenDAL requires explicit configuration through builder patterns. While this approach provides type safety and clear documentation, it can be verbose and inflexible for simple use cases. Many storage systems are naturally identified by URIs (e.g., `s3://bucket/path`, `fs:///path/to/dir`). Adding URI-based configuration would: - Simplify operator creation for common use cases - Enable configuration via connection strings (common in many applications) - Make OpenDAL more approachable for new users - Allow dynamic operator creation based on runtime configuration # Guide-level explanation The new API allows creating operators directly from URIs: ```rust // Create an operator using URI let op = Operator::from_uri("s3://my-bucket/path", vec![ ("endpoint".to_string(), "http://localhost:8080"to_string()), ])?; // Users can pass options through the URI along with additional key-value pairs // The extra options will override identical options specified in the URI let op = Operator::from_uri("s3://my-bucket/path?region=us-east-1", vec![ ("endpoint".to_string(), "http://localhost:8080"to_string()), ])?; // Create a file system operator let op = Operator::from_uri("fs:///tmp/test", vec![])?; ``` OpenDAL will, by default, register services enabled by features in a global `OperatorRegistry`. Users can also create custom operator registries to support their own schemes or additional options. ``` // Using with custom registry let registry = OperatorRegistry::new(); registry.register("custom", my_factory); let op = registry.parse("custom://endpoint", options)?; ``` # Reference-level explanation The implementation consists of three main components: 1. The `OperatorFactory` and `OperatorRegistry`: `OperatorFactory` is a function type that takes a URI and a map of options and returns an `Operator`. `OperatorRegistry` manages operator factories for different schemes. ```rust type OperatorFactory = fn(http::Uri, HashMap) -> Result; pub struct OperatorRegistry { ... } impl OperatorRegistry { fn register(&self, scheme: &str, factory: OperatorFactory) { ... } fn parse(&self, uri: &str, options: impl IntoIterator) -> Result { ... } } ``` 2. The `Configurator` trait extension: `Configurator` will add a new API to create a configuration from a URI and options. OpenDAL will provide default implementations for common configurations. But services can override this method to support their own special needs. For example, S3 might need to extract the `bucket` and `region` from the URI when possible. ```rust impl Configurator for S3Config { fn from_uri(uri: &str, options: impl IntoIterator) -> Result { ... } } ``` 3. The `Operator` `from_uri` method: The `Operator` trait will add a new `from_uri` method to create an operator from a URI and options. This method will use the global `OperatorRegistry` to find the appropriate factory for the scheme. ```rust impl Operator { pub fn from_uri( uri: &str, options: impl IntoIterator, ) -> Result { ... } } ``` We are intentionally using `&str` instead of `Scheme` here to simplify working with external components outside this crate. Additionally, we plan to remove `Scheme` from our public API soon to enable splitting OpenDAL into multiple crates. # Drawbacks - Increases API surface area - Less type safety compared to builder patterns - Potential for confusing error messages with invalid URIs - Need to maintain backwards compatibility # Rationale and alternatives Alternatives considered: 1. Connection string format instead of URIs 2. Builder pattern with URI parsing 3. Macro-based configuration URI-based configuration was chosen because: - URIs are widely understood - Natural fit for storage locations - Extensible through custom schemes - Common in similar tools # Prior art Similar patterns exist in: - Database connection strings (PostgreSQL, MongoDB) - [`object_store::parse_url`](https://docs.rs/object_store/latest/object_store/fn.parse_url.html) # Unresolved questions None # Future possibilities - Support for connection string format. - Configuration presets like `r2` and `s3` with directory bucket enabled. opendal-0.52.0/src/docs/rfcs/5479_context.md000064400000000000000000000110101046102023000164750ustar 00000000000000- Proposal Name: `context` - Start Date: 2024-12-30 - RFC PR: [apache/opendal#5480](https://github.com/apache/opendal/pull/5480) - Tracking Issue: [apache/opendal#5479](https://github.com/apache/opendal/issues/5479) # Summary Add `Context` in opendal to distribute global resources like http client, runtime, etc. # Motivation OpenDAL now includes two global resources, the `http client` and `runtime`, which are utilized by the specified service across all enabled layers. However, it's a bit challenging for layers to interact with these global resources. ## For http client Layers cannot directly access the HTTP client. The only way to interact with the HTTP client is through the service builder, such as [`S3::http_client()`](https://docs.rs/opendal/latest/opendal/services/struct.S3.html#method.http_client). Layers like logging and metrics do not have direct access to the HTTP client. Users need to implement the `HttpFetcher` trait to interact with the HTTP client. However, the drawback is that users lack context for the given requests; they do not know which service the request originates from or which operation it is performing. ## For runtime OpenDAL has the [`Execute`](https://docs.rs/opendal/latest/opendal/trait.Execute.html) for users to implement so that they can interact with the runtime. However, the API is difficult to use, as layers need to extract and construct the `Executor` for every request. For example: ```rust async fn read(&self, path: &str, mut args: OpRead) -> Result<(RpRead, Self::Reader)> { if let Some(exec) = args.executor().cloned() { args = args.with_executor(Executor::with(TimeoutExecutor::new( exec.into_inner(), self.io_timeout, ))); } self.io_timeout(Operation::Read, self.inner.read(path, args)) .await .map(|(rp, r)| (rp, TimeoutWrapper::new(r, self.io_timeout))) } ``` # Guide-level explanation So I propose to add a `Context` to OpenDAL to distribute global resources like the HTTP client and runtime. The `Context` is a struct that contains the global resources, such as the HTTP client and runtime. It is passed to the service builder and layers so that they can interact with the global resources. ```rust let mut ctx = Context::default(); ctx.set_http_client(my_http_client); ctx.set_executor(my_executor); let op = op.with_context(ctx); ``` The following API will be added: - new struct `Context` - `Context::default()` - `Context::load_http_client(&self) -> HttpClient` - `Context::load_executor(&self) -> Executor` - `Context::update_http_client(&self, f: impl FnOnce(HttpClient) -> HttpClient)` - `Context::update_executor(&self, f: impl FnOnce(Executor) -> Executor)` - `Operator::with_context(ctx: Context) -> Operator` The following API will be deprecated: - `Operator::default_executor` - `Operator::with_default_executor` - `OpRead::with_executor` - `OpRead::executor` - `OpWrite::with_executor` - `OpWrite::executor` - All services builders' `http_client` API # Reference-level explanation We will add `Context` struct in `AccessInfo`. Every service must use `Context::default()` for `AccessInfo` and stores the same instance of `Context` in the service core. All the following usage of http client or runtime should be through the `Context` instead. The `Context` itself is a struct wrapped by something like `ArcSwap`, allowing us to update it atomically. The layers will switch to `Context` to get the global resources instead of `OpRead`. We no longer need to hijack the read operation. ```rust - async fn read(&self, path: &str, mut args: OpRead) -> Result<(RpRead, Self::Reader)> { - if let Some(exec) = args.executor().cloned() { - args = args.with_executor(Executor::with(TimeoutExecutor::new( - exec.into_inner(), - self.io_timeout, - ))); - } - - ... - } ``` Instead, we can directly get the executor from the `Context` during `layer`. ```rust impl Layer for TimeoutLayer { type LayeredAccess = TimeoutAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { inner .info() .context() .update_executor(|exec| Executor::with(TimeoutExecutor::new(exec, self.io_timeout))); TimeoutAccessor { inner, timeout: self.timeout, io_timeout: self.io_timeout, } } } ``` # Drawbacks A bit cost (`50ns`) for every operation that `load_http_client`. # Rationale and alternatives None. # Prior art None. # Unresolved questions None. # Future possibilities None.opendal-0.52.0/src/docs/rfcs/5485_conditional_reader.md000064400000000000000000000053561046102023000206530ustar 00000000000000- Proposal Name: `conditional_reader` - Start Date: 2024-12-31 - RFC PR: [apache/opendal#5485](https://github.com/apache/opendal/pull/5485) - Tracking Issue: [apache/opendal#5486](https://github.com/apache/opendal/issues/5486) # Summary Add `if_match`, `if_none_match`, `if_modified_since` and `if_unmodified_since` options to OpenDAL's `reader_with` API. # Motivation OpenDAL currently supports conditional `reader_with` operations based only on `version`. However, many storage services also support conditional operations based on Etag and/or modification time. Adding these options will: - Provide more granular control over read operations. - Align OpenDAL with features provided by modern storage services, meeting broader use cases. # Guide-level explanation Four new options will be added to the `reader_with` API: ## `if_match` Return the content only if its Etag matches the specified Etag; otherwise, an error kind `ErrorKind::ConditionNotMatch` will be returned: ```rust let reader = op.reader_with("path/to/file") .if_match(etag) .await?; ``` ## `if_none_match` Return the content only if its Etag does NOT match the specified Etag; otherwise, an error kind `ErrorKind::ConditionNotMatch` will be returned: ```rust let reader = op.reader_with("path/to/file") .if_none_match(etag) .await?; ``` ## `if_modified_since` Return the content if it has been modified since the specified time; otherwise, an error kind `ErrorKind::ConditionNotMatch` will be returned: ```rust use chrono::{Duration, Utc}; let last_check = Utc::now() - Duration::seconds(3600); // 1 hour ago let reader = op.reader_with("path/to/file") .if_modified_since(last_check) .await?; ``` ## `if_unmodified_since` Return the content if it has NOT been modified since the specified time; otherwise, an error kind `ErrorKind::ConditionNotMatch` will be returned: ```rust use chrono::{Duration, Utc}; let timestamp = Utc::now() - Duration::seconds(86400); // 24 hours ago let reader = op.reader_with("path/to/file") .if_unmodified_since(timestamp) .await?; ``` # Reference-level explanation The main implementation will include: 1. Add new fields(`if_modified_since`, `if_unmodified_since`) and related functions to `OpRead`. 2. Add the related functions to `FutureReader` 3. Add new capability flags: ```rust pub struct Capability { // ... other fields pub read_with_if_modified_since: bool, pub read_with_if_unmodified_since: bool, } ``` 4. implement `if_modified_since`, `if_unmodified_since` for the underlying storage service. # Drawbacks - Add complexity to the API # Rationale and alternatives - Follows existing OpenDAL patterns for conditional operations # Prior art None # Unresolved questions None # Future possibilities None opendal-0.52.0/src/docs/rfcs/5495_list_with_deleted.md000064400000000000000000000140371046102023000205170ustar 00000000000000- Proposal Name: `list_with_deleted` - Start Date: 2025-01-02 - RFC PR: [apache/opendal#5495](https://github.com/apache/opendal/pull/0000) - Tracking Issue: [apache/opendal#5496](https://github.com/apache/opendal/issues/5496) # Summary Add `list_with(path).deleted(true)` to enable users to list deleted files from storage services. # Motivation OpenDAL is currently working on adding support for file versions, allowing users to read, list, and delete them. ```rust // Read given version op.read_with(path).version(version_id).await; // Fetch the metadata of given version. op.stat_with(path).version(version_id).await; // Delete the given version. op.delete_with(path).version(version_id).await; // List the path's versions. op.list_with(path).versions().await; ``` However, to implement the complete data recovery workflow, we should also include support for recovering deleted files from storage services. This feature is referred to as `DeleteMarker` in S3 and `Soft Deleted` in Azure Blob Storage or Google Cloud Storage. Users can utilize these deleted files (or versions) to restore files that may have been accidentally deleted. # Guide-level explanation I suggest adding `list_with(path).deleted(true)` to allow users to list deleted files from storage services. ```rust let entries = op.list_with(path).deleted(true).await; ``` Please note that `deleted` here means "including deleted files" rather than "only deleted files." Therefore, `list_with(path).deleted(true)` will list both current files and deleted ones. At the same time, we will add an `is_deleted` field to the `Metadata` struct to indicate whether the file has been deleted. Together with the existing `is_current` field, we will have the following matrix: | `is_current` | `is_deleted` | Description | |---------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `Some(true)` | `false` | **The metadata's associated version is the latest, current version.** This is the normal state, indicating that this version is the most up-to-date and accessible version. | | `Some(true)` | `true` | **The metadata's associated version is the latest, deleted version (Latest Delete Marker or Soft Deleted).** This is particularly important in object storage systems like S3. It signifies that this version is the **most recent delete marker**, indicating the object has been deleted. Subsequent GET requests will return 404 errors unless a specific version ID is provided. | | `Some(false)` | `false` | **The metadata's associated version is neither the latest version nor deleted.** This indicates that this version is a previous version, still accessible by specifying its version ID. | | `Some(false)` | `true` | **The metadata's associated version is not the latest version and is deleted.** This represents a historical version that has been marked for deletion. Users will need to specify the version ID to access it, and accessing it may be subject to specific delete marker behavior (e.g., in S3, it might not return actual data but a specific delete marker response). | | `None` | `false` | **The metadata's associated file is not deleted, but its version status is either unknown or it is not the latest version.** This likely indicates that versioning is not enabled for this file, or versioning information is unavailable. | | `None` | `true` | **The metadata's associated file is deleted, but its version status is either unknown or it is not the latest version.** This typically means the file was deleted without versioning enabled, or its versioning information is unavailable. This may represent an actual data deletion operation rather than an S3 delete marker. | # Reference-level explanation - Implement the `list_with(path).deleted(true)` API for the `Operator`. - Add an `is_deleted` field to `Metadata`. - Integrate logic for including deleted files into the `list` method of the storage service. # Drawbacks None. # Rationale and alternatives ## Why "including deleted files" rather than "only deleted files"? Most storage services are designed to list files along with deleted files, rather than exclusively listing deleted files. For example: - S3's [ListObjectVersions](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html) API lists all versions of an object, including delete markers. - GCS's [list](https://cloud.google.com/storage/docs/json_api/v1/objects/list) API includes a parameter `softDeleted` to display soft-deleted files. - AzBlob's [List Blobs](https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs) API supports the parameter `include=deleted` to list soft-deleted blobs. So, it is more natural to list files along with deleted files, rather than only listing deleted files. # Prior art None. # Unresolved questions None. # Future possibilities None.opendal-0.52.0/src/docs/rfcs/5556_write_returns_metadata.md000064400000000000000000000072321046102023000215740ustar 00000000000000- Proposal Name: `write_returns_metadata` - Start Date: 2025-01-16 - RFC PR: [apache/opendal#5556](https://github.com/apache/opendal/pull/5556) - Tracking Issue: [apache/opendal#5557](https://github.com/apache/opendal/issues/5557) # Summary Enhance write operations by returning metadata after successful writes. # Motivation Currently, write operations (`write`, `write_with`, `writer`, `writer_with`) only return `Result<()>` or `Result`. Users who need metadata after writing (like `ETag` or `version_id`) must make an additional `stat()` call. This is inefficient and can lead to race conditions if the file is modified between the write and stat operations. Many storage services (like S3, GCS, Azure Blob) return metadata in their write responses. We should expose this information to users directly after write operations. # Guide-level explanation The write operations will be enhanced to return metadata: ```rust // Before op.write("path/to/file", data).await?; let meta = op.stat("path/to/file").await?; if Some(etag) = meta.etag() { println!("File ETag: {}", etag); } // After let meta = op.write("path/to/file", data).await?; if Some(etag) = meta.etag() { println!("File ETag: {}", etag); } ``` For writer operations: ```rust // Before let mut writer = op.writer("path/to/file").await?; writer.write(data).await?; writer.close().await?; let meta = op.stat("path/to/file").await?; if Some(etag) = meta.etag() { println!("File ETag: {}", etag); } // After let mut writer = op.writer("path/to/file").await?; writer.write(data).await?; let meta = writer.close().await?; if Some(etag) = meta.etag() { println!("File ETag: {}", etag); } ``` The behavior remains unchanged if users don't need the metadata - they can simply ignore the return value. # Reference-level explanation ## Changes to `Operator` API The following functions will be modified to return `Result` instead of `Result<()>`: - `write()` - `write_with()` The `writer()` and `writer_with()` return types remain unchanged as they return `Result`. ## Changes to struct `Writer` The `Writer` struct will be modified to return `Result` instead of `Result<()>` for the `close()` function. ## Changes to trait `oio::Write` and trait `oio::MultipartWrite` The `Write` trait will be modified to return `Result` instead of `Result<()>` for the `close()` function. The `MultipartWrite` trait will be modified to return `Result` instead of `Result<()>` for the `complete_part()` and `write_once` functions. ## Implementation Details For services that return metadata in their write responses: - The metadata will be captured from the service response - All available fields (etag, version_id, etc.) will be populated For services that don't return metadata in write responses: - for `fs`: we can use `stat` to retrieve the metadata before returning. since the metadata is cached by the kernel, this won't cause a performance issue. - for other services: A default metadata object will be returned. # Drawbacks - Minor breaking change for users who explicitly type the return value of write operations - Additional complexity in the Writer implementation # Rationale and alternatives - Provides a clean, consistent API - Maintains backward compatibility for users who ignore the return value - Improves performance by avoiding additional stat calls when possible # Prior art Similar patterns exist in other storage SDKs: - `object_store` crate returns metadata in `PutResult` after calling `put_opts` - AWS SDK returns metadata in `PutObjectOutput` - Azure SDK returns `UploadFileResponse` after uploads # Unresolved questions - None # Future possibilities - Noneopendal-0.52.0/src/docs/rfcs/README.md000064400000000000000000000113251046102023000152670ustar 00000000000000# RFCs - OpenDAL Active RFC List RFCs power OpenDAL's development. The "RFC" (request for comments) process is intended to provide a consistent and controlled path for changes to OpenDAL (such as new features) so that all stakeholders can be confident about the direction of the project. Many changes, including bug fixes and documentation improvements, can be implemented and reviewed via the normal GitHub pull request workflow. Some changes, though, are "substantial" and we ask that these be put through a bit of a design process and produce a consensus among the OpenDAL community. ### Which kinds of changes require an RFC? Any substantial change or addition to the project that would require a significant amount of work to implement should generally be an RFC. Some examples include: - A new feature that creates a new public API or raw API. - The removal of features that already shipped as part of the release. - A big refactor of existing code or reorganization of code into new modules. Those are just a few examples. Ultimately, the judgment call of what constitutes a big enough change to warrant an RFC is left to the project maintainers. If you submit a pull request to implement a new feature without going through the RFC process, it may be closed with a polite request to submit an RFC first. ## Before creating the RFC Preparing in advance before submitting an RFC hastily can increase its chances of being accepted. If you have proposals to make, it is advisable to engage in some preliminary groundwork to facilitate a smoother process. It is great to seek feedback from other project developers first, as this can help validate the viability of the RFC. To ensure a sustained impact on the project, it is important to work together and reach a consensus. Common preparatory steps include presenting your idea on platforms such as GitHub [issues](https://github.com/apache/opendal/issues/) or [discussions](https://github.com/apache/opendal/discussions/categories/ideas), or engaging in discussions through our [email list](https://opendal.apache.org/community/#mailing-list) or [Discord server](https://opendal.apache.org/discord). ## The RFC process - Fork the [OpenDAL repo](https://github.com/apache/opendal) and create your branch from `main`. - Copy [`0000_example.md`] to `0000-my-feature.md` (where "my-feature" is descriptive). Don't assign an RFC number yet; This is going to be the PR number, and we'll rename the file accordingly if the RFC is accepted. - Submit a pull request. As a pull request, the RFC will receive design feedback from the larger community, and the author should be prepared to revise it in response. - Now that your RFC has an open pull request, use the issue number of this PR to update your `0000-` prefix to that number. - Build consensus and integrate feedback. RFCs that have broad support are much more likely to make progress than those that don't receive any comments. Feel free to reach OpenDAL maintainers for help. - RFCs rarely go through this process unchanged, especially as alternatives and drawbacks are shown. You can make edits, big and small, to the RFC to clarify or change the design, but make changes as new commits to the pull request, and leave a comment on the pull request explaining your changes. Specifically, do not squash or rebase commits after they are visible on the pull request. - The RFC pull request lasts for three days after the last update. After that, the RFC will be accepted or declined based on the consensus reached in the discussion. - For the accepting of an RFC, we will require approval from at least three maintainers. - Once the RFC is accepted, please create a tracking issue and update links in RFC. And then the PR will be merged and the RFC will become 'active' status. ## Implementing an RFC An active RFC does not indicate the priority assigned to its implementation, nor does it imply that a developer has been specifically assigned the task of implementing the feature. The RFC author is encouraged to submit an implementation after the RFC has been accepted. Nevertheless, it is not obligatory for them to do so. Accepted RFCs may represent features that can wait until a developer chooses to work on them. Each accepted RFC is associated with an issue in the OpenDAL repository, which tracks its implementation. If you are interested in implementing an RFC but are unsure if someone else is already working on it, feel free to inquire by leaving a comment on the associated issue. ## Some useful tips - The author of an RFC may not be the same one as the implementer. Therefore, when submitting an RFC, it is advisable to include sufficient information. - If modifications are needed for an accepted RFC, please submit a new pull request or create a new RFC to propose changes. opendal-0.52.0/src/docs/rfcs/mod.rs000064400000000000000000000161441046102023000151410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #![doc = include_str!("README.md")] /// RFC example #[doc = include_str!("0000_example.md")] pub mod rfc_0000_example {} /// Object native API #[doc = include_str!("0041_object_native_api.md")] pub mod rfc_0041_object_native_api {} /// Error handle #[doc = include_str!("0044_error_handle.md")] pub mod rfc_0044_error_handle {} /// Auto region #[doc = include_str!("0057_auto_region.md")] pub mod rfc_0057_auto_region {} /// Object stream #[doc = include_str!("0069_object_stream.md")] pub mod rfc_0069_object_stream {} /// Limited reader #[doc = include_str!("0090_limited_reader.md")] pub mod rfc_0090_limited_reader {} /// Path normalization #[doc = include_str!("0112_path_normalization.md")] pub mod rfc_0112_path_normalization {} /// Async streaming IO #[doc = include_str!("0191_async_streaming_io.md")] pub mod rfc_0191_async_streaming_io {} /// Remove credential #[doc = include_str!("0203_remove_credential.md")] pub mod rfc_0203_remove_credential {} /// Create dir #[doc = include_str!("0221_create_dir.md")] pub mod rfc_0221_create_dir {} /// Retryable error #[doc = include_str!("0247_retryable_error.md")] pub mod rfc_0247_retryable_error {} /// Object ID #[doc = include_str!("0293_object_id.md")] pub mod rfc_0293_object_id {} /// Dir entry #[doc = include_str!("0337_dir_entry.md")] pub mod rfc_0337_dir_entry {} /// Accessor capabilities #[doc = include_str!("0409_accessor_capabilities.md")] pub mod rfc_0409_accessor_capabilities {} /// Presign #[doc = include_str!("0413_presign.md")] pub mod rfc_0413_presign {} /// Command line interface #[doc = include_str!("0423_command_line_interface.md")] pub mod rfc_0423_command_line_interface {} /// Init from iter #[doc = include_str!("0429_init_from_iter.md")] pub mod rfc_0429_init_from_iter {} /// Multipart #[doc = include_str!("0438_multipart.md")] pub mod rfc_0438_multipart {} /// Gateway #[doc = include_str!("0443_gateway.md")] pub mod rfc_0443_gateway {} /// New builder #[doc = include_str!("0501_new_builder.md")] pub mod rfc_0501_new_builder {} /// Write refactor #[doc = include_str!("0554_write_refactor.md")] pub mod rfc_0554_write_refactor {} /// List metadata reuse #[doc = include_str!("0561_list_metadata_reuse.md")] pub mod rfc_0561_list_metadata_reuse {} /// Blocking API #[doc = include_str!("0599_blocking_api.md")] pub mod rfc_0599_blocking_api {} /// Redis service #[doc = include_str!("0623_redis_service.md")] pub mod rfc_0623_redis_service {} /// Split capabilities #[doc = include_str!("0627_split_capabilities.md")] pub mod rfc_0627_split_capabilities {} /// Path in accessor #[doc = include_str!("0661_path_in_accessor.md")] pub mod rfc_0661_path_in_accessor {} /// Generic KV services #[doc = include_str!("0793_generic_kv_services.md")] pub mod rfc_0793_generic_kv_services {} /// Object reader #[doc = include_str!("0926_object_reader.md")] pub mod rfc_0926_object_reader {} /// Refactor error #[doc = include_str!("0977_refactor_error.md")] pub mod rfc_0977_refactor_error {} /// Object handler #[doc = include_str!("1085_object_handler.md")] pub mod rfc_1085_object_handler {} /// Object metadataer #[doc = include_str!("1391_object_metadataer.md")] pub mod rfc_1391_object_metadataer {} /// Query based metadata #[doc = include_str!("1398_query_based_metadata.md")] pub mod rfc_1398_query_based_metadata {} /// Object writer #[doc = include_str!("1420_object_writer.md")] pub mod rfc_1420_object_writer {} /// Remove object concept #[doc = include_str!("1477_remove_object_concept.md")] pub mod rfc_1477_remove_object_concept {} /// Operation extension #[doc = include_str!("1735_operation_extension.md")] pub mod rfc_1735_operation_extension {} /// Writer sink API #[doc = include_str!("2083_writer_sink_api.md")] pub mod rfc_2083_writer_sink_api {} /// Append API #[doc = include_str!("2133_append_api.md")] pub mod rfc_2133_append_api {} /// Chain based operator API #[doc = include_str!("2299_chain_based_operator_api.md")] pub mod rfc_2299_chain_based_operator_api {} /// Object versioning #[doc = include_str!("2602_object_versioning.md")] pub mod rfc_2602_object_versioning {} /// Merge append into write #[doc = include_str!("2758_merge_append_into_write.md")] pub mod rfc_2758_merge_append_into_write {} /// Lister API #[doc = include_str!("2774_lister_api.md")] pub mod rfc_2774_lister_api {} /// List with metakey #[doc = include_str!("2779_list_with_metakey.md")] pub mod rfc_2779_list_with_metakey {} /// Native capability #[doc = include_str!("2852_native_capability.md")] pub mod rfc_2852_native_capability {} /// Remove write copy from #[doc = include_str!("3017_remove_write_copy_from.md")] pub mod rfc_3017_remove_write_copy_from {} /// Config #[doc = include_str!("3197_config.md")] pub mod rfc_3197_config {} /// Align list API #[doc = include_str!("3232_align_list_api.md")] pub mod rfc_3232_align_list_api {} /// List prefix #[doc = include_str!("3243_list_prefix.md")] pub mod rfc_3243_list_prefix {} /// Lazy reader #[doc = include_str!("3356_lazy_reader.md")] pub mod rfc_3356_lazy_reader {} /// List recursive #[doc = include_str!("3526_list_recursive.md")] pub mod rfc_3526_list_recursive {} /// Concurrent stat in list #[doc = include_str!("3574_concurrent_stat_in_list.md")] pub mod rfc_3574_concurrent_stat_in_list {} /// Buffered Reader #[doc = include_str!("3734_buffered_reader.md")] pub mod rfc_3734_buffered_reader {} /// Concurrent Writer #[doc = include_str!("3898_concurrent_writer.md")] pub mod rfc_3898_concurrent_writer {} /// Deleter API #[doc = include_str!("3911_deleter_api.md")] pub mod rfc_3911_deleter_api {} /// Range Based Read API #[doc = include_str!("4382_range_based_read.md")] pub mod rfc_4382_range_based_read {} /// Executor API #[doc = include_str!("4638_executor.md")] pub mod rfc_4638_executor {} /// Remove metakey #[doc = include_str!("5314_remove_metakey.md")] pub mod rfc_5314_remove_metakey {} /// Operator from uri #[doc = include_str!("5444_operator_from_uri.md")] pub mod rfc_5444_operator_from_uri {} /// Context #[doc = include_str!("5479_context.md")] pub mod rfc_5479_context {} /// Conditional Reader #[doc = include_str!("5485_conditional_reader.md")] pub mod rfc_5485_conditional_reader {} /// List With Deleted #[doc = include_str!("5495_list_with_deleted.md")] pub mod rfc_5495_list_with_deleted {} /// Write Returns Metadata #[doc = include_str!("5556_write_returns_metadata.md")] pub mod rfc_5556_write_returns_metadata {} opendal-0.52.0/src/docs/upgrade.md000064400000000000000000001415031046102023000150260ustar 00000000000000# Upgrade to v0.52 ## Public API ### RFC-5556: Write Returns Metadata Since v0.52, all write APIs in OpenDAL have been updated to return `Metadata` instead of `()`. This metadata includes useful information provided by the service, such as `content-length`, `etag`, `version`, and `last-modified`. This feature is not fully ready yet, and many available metadata fields are still not returned. Please visit [Tracking Issues of RFC-5556: Write Returns Metadata](https://github.com/apache/opendal/issues/5557) for progress and contributions. Affected API: - `opendal::Operator::write` - `opendal::Operator::write_with` - `opendal::Operator::writer::close` - `opendal::raw::oio::Write::close` ### Github Actions Cache (ghac) service v2 As [requested](https://github.com/apache/opendal/issues/5620) by GitHub, we have upgraded our GHAC service to ensure compatibility with the latest GitHub Actions cache API. By upgrading to OpenDAL v0.52, your services will continue functioning after the deprecation of the legacy service (2025/03/01). GHES does not yet support GHAC v2, but OpenDAL has handled this properly to prevent any disruptions. ghac service doesn't support `delete` anymore, please use github's API to delete cache instead. This upgrade is mandatory and enabled by default using an environment variable in the GitHub CI environment. No changes are required at the code level. ### Breaking Changes in Dependencies - `OtelTraceLayer` and `OtelMetricsLayer`'s dependence `opentelemetry` bumped to `0.28` - `PrometheusClientLayer`'s dependence `prometheus-client` bumped to `0.23.1` # Upgrade to v0.51 ## Public API ### New VISION: One Layer, All Storage OpenDAL has refined its vision to **One Layer, All Storage**, driven by the following core principles: **Open Community**, **Solid Foundation**, **Fast Access**, **Object Storage First**, and **Extensible Architecture**. Explore the detailed vision at [OpenDAL Vision](https://opendal.apache.org/vision). ### RFC-5313: Remove Metakey OpenDAL v0.51 implements [RFC-5313](https://opendal.apache.org/docs/rust/opendal/docs/rfcs/rfc_5314_remove_metakey/index.html), which removes the concept of metakey. The following structs have been removed: - `Metakey` The following APIs have been removed: - `list_with(path).metakey()` Users no longer need to pass the metakey into the list. Instead, services will make their best effort to return as much metadata as possible. Users can check items like `Capability::list_has_etag` before making a call. ### Remove not used capability: `write_multi_align_size` The capability `write_multi_align_size` is not utilized by any services, and we have no plans to support it in the future; therefore, we have removed it. ### CapabilityCheckLayer and CorrectnessCheckLayer OpenDAL used to perform capability checks for all services, but since v0.51, it only conducts checks that impact data correctness like `write_with_if_not_exists` or `delete_with_version` by default in the `CorrectnessCheckLayer`. If users wish to verify other non-critical capabilities like `write_with_content_type` or `write_with_cache_control`, they should manually enable the `CapabilityCheckLayer`. ### RFC-3911: Deleter API OpenDAL v0.51 implements [RFC-3911](https://opendal.apache.org/docs/rust/opendal/docs/rfcs/rfc_3911_deleter_api/index.html), which adds `Deleter` in OpenDAL to replace `batch` operation. The following new APIs have been added: - [`Operator::delete_iter`] - [`Operator::delete_try_iter`] - [`Operator::delete_stream`] - [`Operator::delete_try_stream`] - [`Operator::deleter`] - [`Deleter::delete`] - [`Deleter::delete_iter`] - [`Deleter::delete_try_iter`] - [`Deleter::delete_stream`] - [`Deleter::delete_try_stream`] - [`Deleter::flush`] - [`Deleter::close`] - [`Deleter::into_sink`] - [`DeleteInput`] - [`IntoDeleteInput`] - [`FuturesDeleteSink`] The following APIs have been deprecated and will be removed in the future releases: - `Operator::remove` (replace with [`Operator::delete_iter`]) - `Operator::remove_via` (replace with [`Operator::delete_stream`]) As a result of this change, the `limit` and `with_limit` APIs on `Operator` have also been deprecated; they are currently no-ops. ## Raw API ### `adapter::kv` now returns `Scanner` instead of `Vec` To support returning key-value entries in a streaming manner instead of loading them all into memory, OpenDAL updated its adapter API to return a `Scanner` instead of a `Vec`. ```diff - async fn scan(&self, path: &str) -> Result> + async fn scan(&self, path: &str) -> Result ``` All services intending to implement `kv::Adapter` should adhere to this API change. ## Align `metadata` API to `info` OpenDAL changes it's old `metadata` API to `info` to align with the new `AccessorInfo` struct. ```diff - fn metadata(&self) -> Arc + fn info(&self) -> Arc ``` ### Remove not used struct: `RangeWriter` The struct `RangeWriter` is not utilized by any services, and we have no plans to support it in the future; therefore, we have removed it. # Upgrade to v0.50 ## Public API ### `services-postgresql`'s connect string now supports only URL format Previously, it supports both URL format and key-value format. After switching the implementation from `tokio-postgres` to `sqlx`, the service now supports only the URL format. ### `list` now returns path itself Previously, `list("a/b")` would not return `a/b` even if it does exist. Since v0.50.0, this behavior has been changed. OpenDAL will now return the path itself if it exists. This change applies to all cases, whether the path is a directory or a file. ### Refactoring of the metrics-related layer In OpenDAL v0.50.0, we did a refactor on all metrics-related layers. They are now sharing the same underlying implementations. `PrometheusLayer`, `PrometheusClientLayer` and `MetricsLayer` are now have similar public APIs and exactly the same metrics value. # Upgrade to v0.49 ## Public API ### `Configurator` now returns associated builder instead `Configurator` used to return `impl Builder`, but now it returns associated builder type directly. This will allow users to use the builder in a more flexible way. ```diff impl Configurator for MemoryConfig { - fn into_builder(self) -> impl Builder { + type Builder = MemoryBuilder; + fn into_builder(self) -> Self::Builder { MemoryBuilder { config: self } } } ``` ### `LoggingLayer` now accepts `LoggingInterceptor` `LoggingLayer` now accepts `LoggingInterceptor` trait instead of configuration. This change will allow users to customize the logging behavior more flexibly. ```diff pub trait LoggingInterceptor: Debug + Clone + Send + Sync + Unpin + 'static { fn log( &self, info: &AccessorInfo, operation: Operation, context: &[(&str, &str)], message: &str, err: Option<&Error>, ); } ``` Users can now implement the log in the way they want. # Upgrade to v0.48 ## Public API ### Typo in `customized_credential_load` Since v0.48, the `customed_credential_load` function has been renamed to `customized_credential_load` to fix the typo of `customized`. ```diff - builder.customed_credential_load(v); + builder.customized_credential_load(v); ``` ### S3 service rename `security_token` to `session_token` [In 2014 Amazon switched](https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/) from `AWS_SECURITY_TOKEN` to `AWS_SESSION_TOKEN`. To be consistent with the naming of AWS STS, we have renamed the `security_token` field to `session_token` in the S3 service. ```diff - builder.security_token(v); + builder.session_token(v); ``` ### Operator `from_iter` and `via_iter` replaces `from_map` and `via_map` Since v0.48, Operator's new APIs `from_iter` and `via_iter` methods have deprecated the `from_map` and `via_map` methods. ```diff - Operator::from_map::(map)?.finish(); + Operator::from_iter::(map)?.finish(); ``` New API `from_iter` and `via_iter` should cover all use cases of `from_map` and `via_map`. ### Service builder now takes ownership Since v0.48, all service builder now takes ownership `self` instead of `&mut self`. This change will allow users to configure the service in a more flexible way. ```diff - let mut builder = S3::default(); - builder.bucket("test"); - builder.root("/path/to/root"); + let builder = S3::default().bucket("test").root("/path/to/root"); let op = Operator::new(builder)?.finish(); ``` ## Raw API ### `oio::Write::write` will write the whole buffer Starting from version 0.48, `oio::Write::write` now writes the entire buffer. This update aligns the API more closely with `oio::Read::read` and simplifies the implementation of concurrent writing. ```diff trait Write { - fn write(&mut self, bs: Buffer) -> impl Future>; + fn write(&mut self, bs: Buffer) -> impl Future>; } ``` `write` will now return `Result<()>` instead of `Result`. The number of bytes written can be obtained from the buffer's length. ### `Access::metadata()` will return `Arc` Starting from version 0.48, `Access::metadata()` will return `Arc` instead of `AccessInfo`. This change is intended to improve performance and reduce memory usage. ```diff trait Access { - fn metadata(&self) -> AccessInfo; + fn metadata(&self) -> Arc; } ``` ### `MinitraceLayer` renamed to `FastraceLayer` The `MinitraceLayer` has been renamed to `FastraceLayer` to respond to the [transition from `minitrace` to `fastrace`](https://github.com/tikv/minitrace-rust/issues/229). ```diff - use opendal::layers::MinitraceLayer; + use opendal::layers::FastraceLayer; ``` ### Use `Configurator` to replace `Builder::from_config` Since v0.48, the `Builder::from_config` and `Builder::from_map` method has been replaced by the `Configurator` trait. The `Configurator` trait provides a more flexible and extensible way to configure OpenDAL. Service implementers should update their code to use the `Configurator` trait instead: ```rust impl Configurator for MemoryConfig { type Builder = MemoryBuilder; fn into_builder(self) -> Self::Builder { MemoryBuilder { config: self } } } impl Builder for MemoryBuilder { const SCHEME: Scheme = Scheme::Memory; type Config = MemoryConfig; fn build(self) -> Result { ... } } ``` # Upgrade to v0.47 ## Public API ### Reader `into_xxx` APIs Since v0.47, `Reader`'s `into_xxx` APIs requires `async` and returns `Result` instead. ```diff - let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048); + let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048).await?; ``` Affected API includes: - `Reader::into_futures_async_read` - `Reader::into_bytes_stream` - `BlockingReader::into_std_read` - `BlockingReader::into_bytes_iterator` ## Raw API ### Bring Streaming Read Back As explained in [core: Bring Streaming Read Back](https://github.com/apache/opendal/issues/4672), we do need read streaming back for better performance and low memory usage. So our `oio::Read` changed back to streaming read instead: ```diff trait Read { - async fn read(&self, offset: u64, size: usize) -> Result; + async fn read(&mut self) -> Result; } ``` All services and layers should be updated to meet this change. # Upgrade to v0.46 ## Public API ### MSRV Changed to 1.75 Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like [`RPITIT`](https://rust-lang.github.io/rfcs/3425-return-position-impl-trait-in-traits.html) and [`AFIT`](https://rust-lang.github.io/rfcs/3185-static-async-fn-in-trait.html). ### Services Feature Flag Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags. Additionally, we have removed all `reqwest`-related feature flags: - Users must now directly use `reqwest`'s feature flags for options like `rustls`, `native-tls`, etc. - The `rustls` feature is no longer enabled by default; it must be activated manually. - OpenDAL no longer offers the `trust-dns` option; users should configure the client builder directly. ### Range Based Read Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO. `opendal::Reader` now have APIs like: ```rust let r = op.reader("test.txt").await?; let buf = r.read(1024..2048).await?; ``` ### Buffer Based IO Since version 0.46, OpenDAL features a native `Buffer` struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO. OpenDAL's `Reader` will return `Buffer` and `Writer` will accept `Buffer` as input. Users who have implemented their own IO traits should update their code to use the new `Buffer` struct. ```rust let r = op.reader("test.txt").await?; // read returns `Buffer` let buf: Buffer = r.read(1024..2048).await?; let w = op.writer("test.txt").await?; // Buffer can be created from continues bytes. w.write("hello, world").await?; // Buffer can also be created from non-continues bytes. w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?; // Make sure file has been written completely. w.close().await?; ``` To enhance usability, we've integrated bridges into `bytes::Buf` and `bytes::BufMut`, allowing users to directly interact with the bytes API. ```rust let r = op.reader("test.txt").await?; let mut bs = vec![]; // read_into accepts bytes::BufMut let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?; let w = op.writer("test.txt").await?; // write_from accepts bytes::Buf w.write_from("hello, world".as_bytes()).await?; // Make sure file has been written completely. w.close().await?; ``` ### Bridge API OpenDAL's `Reader` and `Writer` previously implemented APIs such as `AsyncRead` and `AsyncWrite` directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance. Since v0.46, OpenDAL provides bridge APIs for `Reader` and `Writer` instead. ```rust let r = op.reader("test.txt").await?; // Convert into futures AsyncRead + AsyncSeek. let reader = r.into_futures_async_read(1024..2048); // Convert into futures bytes stream. let stream = r.into_bytes_stream(1024..2048); let w = op.writer("test.txt").await?; // Convert into futures AsyncWrite let writer = w.into_futures_async_write(); // Convert into futures bytes sink; let sink = w.into_bytes_sink(); ``` ## Raw API ### Async in IO trait Since version 0.46, OpenDAL has adopted Rust's native [`async_in_trait`](https://blog.rust-lang.org/2023/12/21/async-fn-rpit-in-traits.html) for our core IO traits, including `oio::Read`, `oio::Write`, and `oio::List`. This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later. Users who have implemented their own IO traits should update their code to use the new async trait syntax. # Upgrade to v0.45 ## Public API ### BlockingLayer is not enabled by default To further enhance the optionality of `tokio`, we have introduced a new feature called `layers-blocking`. The default usage of the blocking layer has been disabled. To utilize the `BlockingLayer`, please enable the `layers-blocking` feature. ### TimeoutLayer deprecated `with_speed` The `with_speed` API has been deprecated. Please use `with_io_timeout` instead. ## Raw API No raw API changes. # Upgrade to v0.44 ## Public API ### Moka Service Configuration - The `thread_pool_enabled` option has been removed. ### List Prefix Supported After [RFC: List Prefix](crate::docs::rfcs::rfc_3243_list_prefix) landed, we have changed the behavior of `list` a path without `/`. OpenDAL used to return `NotADirectory` error, but now we will return the list of entries that start with given prefix instead. # Upgrade to v0.43 ## Public API ### List Recursive After [RFC-3526: List Recursive](crate::docs::rfcs::rfc_3526_list_recursive) landed, we have changed the `list` API to accept `recursive` instead of `delimiter`: Users will need to change the following usage: - `op.list_with(path).delimiter("")` -> `op.list_with(path).recursive(true)` - `op.list_with(path).delimiter("/")` -> `op.list_with(path).recursive(false)` `delimiter` other than `""` and `"/"` is not supported anymore. ### Stat a dir path After [RFC: List Prefix](crate::docs::rfcs::rfc_3243_list_prefix) landed, we have changed the behavior of `stat` a dir path: Here are the behavior list: | Case | Path | Result | |------------------------|-----------------|--------------------------------------------| | stat existing dir | `abc/` | Metadata with dir mode | | stat existing file | `abc/def_file` | Metadata with file mode | | stat dir without `/` | `abc/def_dir` | Error `NotFound` or metadata with dir mode | | stat file with `/` | `abc/def_file/` | Error `NotFound` | | stat not existing path | `xyz` | Error `NotFound` | Services like s3, azblob can handle `stat("abc/")` correctly by check if there are objects with prefix `abc/`. ## Raw API ### Lister Align We changed our internal `lister` implementation to align with the `list` public API for better performance and readability. - trait `Page` => `List` - struct `Pager` => `Lister` - trait `BlockingPage` => `BlockingList` - struct `BlockingPager` => `BlockingLister` Every call to `next` will return an entry instead a page of entries. Also, we changed our async list api into poll based instead of `async_trait`. # Upgrade to v0.42 ## Public API ### MSRV Changed OpenDAL bumps it's MSRV to 1.67.0. ### S3 Service Configuration - The `enable_exact_buf_write` option has been deprecated and is superseded by `BufferedWriter`, introduced in version 0.40. ### Oss Service Configuration - The `write_min_size` option has been deprecated and replaced by `BufferedWriter`, also introduced in version 0.40. - A new setting, `allow_anonymous`, has been added. Since v0.41, OSS will now return an error if credential loading fails. Enabling `allow_anonymous` to fallback to request without credentials. ### Ghac Service Configuration - The `enable_create_simulation` option has been removed. We add this option to allow ghac simulate create empty file, but it could result in unexpected behavior when users create a file with content length `1`. So we remove it. ### Wasabi Service Removed `wasabi` service native support has been removed. Users who want to access wasabi can use our `s3` service instead. # Upgrade to v0.41 There is no public API and raw API changes. # Upgrade to v0.40 ## Public API ### RFC-2578 Merge Append Into Write [RFC-2578](crate::docs::rfcs::rfc_2758_merge_append_into_write) merges `append` into `write` and removes `append` API. - For writing a file at once, please use `op.write()` for convenience. - For appending a file, please use `op.write_with().append(true)` instead of `op.append()`. The same rule applies to `writer()` and `writer_with()`. ### RFC-2774 Lister API [RFC-2774](crate::docs::rfcs::rfc_2774_lister_api) proposes a new `lister` API to replace current `list` and `scan`. And we add a new API `list` to return entries directly. - For listing a directory at once, please use `list()` for convenience. - For listing a directory recursively, please use `list_with().delimiter("")` or `lister_with().delimiter("")` instead of `scan()`. - For listing in streaming, please use `lister()` or `lister_with()` instead. ### RFC-2779 List With Metakey [RFC-2779](crate::docs::rfcs::rfc_2779_list_with_metakey) proposes a new `op.list_with().metakey()` API to allow list with metakey and removes `op.metadata(&entry)` API. Please use `op.list_with().metakey()` instead of `op.metadata(&entry)`, for example: ```rust // Before let entries: Vec = op.list("dir/").await?; for entry in entries { let meta = op.metadata(&entry, Metakey::ContentLength | Metakey::ContentType).await?; println!("{} {}", entry.name(), entry.metadata().content_length()); } // After let entries: Vec = op .list_with("dir/") .metakey(Metakey::ContentLength | Metakey::ContentType).await?; for entry in entries { println!("{} {}", entry.name(), entry.metadata().content_length()); } ``` ### RFC-2852: Native Capability [RFC-2852](crate::docs::rfcs::rfc_2852_native_capability) proposes new `native_capability` and `full_capability` API to allow users to check if the underlying service supports a capability natively. - `native_capability` returns `true` if the capability is supported natively. - `full_capability` returns `true` if the capability is supported, maybe via a layer. Most of time, you can use `full_capability` to replace `capability` call. But to check if the capability is supported natively for better performance design, please use `native_capability` instead. ### Buffered Writer OpenDAL v0.40 added buffered writer support! Users don't need to specify the `content_length()` for writer anymore! ```diff - let mut w = op.writer_with("path/to/file").content_length(1024).await?; + let mut w = op.writer_with("path/to/file").await?; ``` Users can specify the `buffer()` to control the size we call underlying storage: ```rust let mut w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?; ``` If buffer is not specified, we will call underlying storage everytime we call `write`. Otherwise, we will make sure to call underlying storage when buffer is full or `close` is called. ### RangeRead and RangeReader OpenDAL v0.40 removed the origin `range_read` and `range_reader` interfaces, please use `read_with().range()` or `reader_with().range()`. ```diff - op.range_read(path, range_start..range_end).await?; + op.read_with(path).range(range_start..range_end).await?; ``` ```diff - let reader = op.range_reader(path, range_start..range_end).await?; + let reader = op.reader_with(path).range(range_start..range_end).await?; ``` ## Raw API ### RFC-3017 Remove Write Copy From [RFC-3017](crate::docs::rfcs::rfc_3017_remove_write_copy_from) removes `copy_from` API from the `oio::Write` trait. Users who implements services and layers by hand should remove this API. # Upgrade to v0.39 ## Public API ### Service S3 Role Arn Behavior In PR #2687, OpenDAL changed the behavior when `role_arn` has been specified. OpenDAL used to override role_arn simply. But since this version, OpenDAL will make sure to use assume_role with specified `role_arn` and `external_id` (if supplied). ### RetryLayer supports RetryInterceptor In PR #2666, `RetryLayer` supports `RetryInterceptor`. To implement this change, `RetryLayer` changed it's in-memory layout by adding a new generic parameter `I` to `RetryLayer`. Users who stores `RetryLayer` in struct or enum will need to change the type if they don't want to use default behavior. ## Raw API In PR #2698, OpenDAL re-org the internal structure of `opendal::raw::oio` and changed some APIs name. # Upgrade to v0.38 There are no public API changes. ## Raw API OpenDAL add the `Write::sink` API to enable streaming writing. This is a breaking change for users who depend on the raw API. For a quick fix, users who have implemented `opendal::raw::oio::Write` can return an `Unsupported` error for `Write::sink()`. More details could be found at [RFC: Writer `sink` API][crate::docs::rfcs::rfc_2083_writer_sink_api]. # Upgrade to v0.37 In v0.37.0, OpenDAL bump the version of `reqsign` to v0.13.0. There are no public API and raw API changes. # Upgrade to v0.36 ## Public API In v0.36, OpenDAL improving the `xxx_with` API by allow it to be called in chain: After this change, all `xxx_with` alike call will be changed from ```rust let bs = op.read_with( "path/to/file", OpRead::new() .with_range(0..=1024) .with_if_match("") .with_if_none_match("") .with_override_cache_control("") .with_override_content_disposition("") ).await?; ``` to ```rust let bs = op.read_with("path/to/file") .range(0..=1024) .if_match("") .if_none_match("") .override_cache_control("") .override_content_disposition("") .await?; ``` For blocking API calls, we will need a `call()` at the end: ```rust let bs = bop.read_with("path/to/file") .range(0..=1024) .if_match("") .if_none_match("") .override_cache_control("") .override_content_disposition("") .call()?; ``` Along with this change, users don't need to call `OpXxx` anymore so we moved it to `raw` API. More details could be found at [RFC: Chain Based Operator API][crate::docs::rfcs::rfc_2299_chain_based_operator_api]. ## Raw API Migrated `opendal::ops` to `opendal::raw::ops`. # Upgrade to v0.35 ## Public API - OpenDAL removes rarely used `Operator::from_env` and `Operator::from_iter` APIs - Users can use `Operator::via_map` instead. ## Raw API - OpenDAL adds `append` support with could break existing layers. Please make sure `append` requests have been forward correctly. - After the merging of `scan` and `list`, OpenDAL removes the `scan` from raw API. Please use `list_without_delimiter` instead. # Upgrade to v0.34 ## Public API - OpenDAL raises it's MSRV to 1.65 for dependencies changes - `OperatorInfo::can_scan` has been removed, to check if underlying services support scan a dir natively, please use `Capability::list_without_delimiter` instead. ## Raw API ### Merged `scan` into `list` After `Capability` introduced, we have added `delimiter` in `OpList`. Users can specify the delimiter to `""` or `"/"` to control the list behavior. Along with this change, `Operator::scan()` becomes a short alias of `Operator::list_with(OpList::new().with_delimiter(""))`. ### Typed Kv Adapter In v0.34, OpenDAL adds a typed kv adapter for zero-copy read and write. If you are implemented kv adapter for a rust in-memory data struct, please consider migrate. # Upgrade to v0.33 ## Public API OpenDAL 0.33 has redesigned the `Writer` API, replacing all instances of `writer.append()` with `writer.write()`. For more information, please refer to [`Writer`](crate::Writer). ## Raw API In addition to the redesign of the `Writer` API, we have removed `append` from `oio::Write`. Therefore, users who implement services and layers should also remove it. After v0.33 landing, services should handle `OpWrite::content_length` correctly by following these guidelines: - If the writer does not support uploading unsized data, return a response of `NotSupported` if content length is `None`. - Otherwise, continue writing data until either `close` or `abort` has been called. Furthermore, OpenDAL 0.33 introduces a new concept called `Capability` which replaces `AccessorCapability`. Services must adapt to this change. # Upgrade to v0.32 OpenDAL 0.32 doesn't have much breaking changes. We changed `Accessor::create` into `Accessor::create_dir`. Only users who implement `Layer` need to change. # Upgrade to v0.31 In version v0.31 of OpenDAL, we made some internal refactoring to improve its compatibility with the ecosystem. ## MSRV Bump We increased the MSRV to 1.64 from v0.31 onwards. Although it is still possible to build OpenDAL under older rustc versions, we cannot guarantee that any issues related to them will be fixed. ## Accept `std::time::Duration` instead Previously, OpenDAL accepted `time::Duration` as input for `presign_xxx`. However, since v0.31, we have changed this to accept `std::time::Duration` so that users do not need to depend on `time`. Internally, we migrated from `time` to `chrono` for better integration with other parts of the ecosystem. ## `disable_ec2_metadata` for services s3 We have added a new configuration option called `disable_ec2_metadata` for the S3 service in response to a mistake where it was mixed up with another option called `disable_config_load`. Users who want to disable loading credentials from EC2 metadata should set this option instead. ## Services Feature Flag Starting from v0.31, all services in OpenDAL are split into different feature flags. To enable only S3 support, use the following TOML configuration: ```toml opendal = { version = "0.31", default-features = false, features = ["services-s3"] } ``` # Upgrade to v0.30 In version 0.30, we made significant breaking changes by removing objects. Our goal in doing so was to provide our users with APIs that are easier to understand and maintain. More details could be found at [RFC: Remove Object Concept][crate::docs::rfcs::rfc_1477_remove_object_concept]. To upgrade to OpenDAL v0.30, users need to make the following changes: - regex replace `object\((.*)\).reader\(\)` to `reader($1)` - replace the function on your case, it's recommended to do it one by one - rename `ObjectMetakey` => `Metakey` - rename `ObjectMode` => `EntryMode` - replace `ErrorKind::ObjectXxx` to `ErrorKind::Xxx` - rename `AccessorMetadata` => `AccessorInfo` - rename `ObjectMetadata` => `Metadata` - replace `operator.metadata()` => `operator.info()` # Upgrade to v0.29 In v0.29, we introduced [Object Writer][crate::docs::rfcs::rfc_1420_object_writer] to replace existing Multipart related APIs. Users can now append multiparts bytes into object via: ```rust let mut w = o.writer().await?; w.write(bs1).await?; w.write(bs2).await?; w.close() ``` Along with this change, we cleaned up a lot of internal structs and traits. Users who used to depend on `opendal::raw::io::{input,output}` should use `opendal::raw::oio` instead. Also, decompress related feature also removed. Users can use `async-compression` with `ObjectReader` directly. # Upgrade to v0.28 In v0.28, we introduced [Query Based Metadata][crate::docs::rfcs::rfc_1398_query_based_metadata]. Users can query cached metadata with `ObjectMetakey` to make sure that OpenDAL always makes the best decision. ```diff - pub async fn metadata(&self) -> Result; + pub async fn metadata( + &self, + flags: impl Into>, + ) -> Result>; ``` Please visit `Object::metadata()`'s example for more details. # Upgrade to v0.27 In v0.27, we refactored our `list` related logic and added `scan` support. So make `Pager` and `BlockingPager` associated types in `Accessor` too! ```diff pub trait Accessor: Send + Sync + Debug + Unpin + 'static { type Reader: output::Read; type BlockingReader: output::BlockingRead; + type Pager: output::Page; + type BlockingPager: output::BlockingPage; } ``` ## User defined layers Due to this change, all layers implementation should be changed. If there is not changed over pager, they can be changed like the following: ```diff impl LayeredAccessor for MyAccessor { type Inner = A; type Reader = MyReader; type BlockingReader = MyReader; + type Pager = A::Pager; + type BlockingPager = A::BlockingPager; + async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Pager)> { + self.inner.list(path, args).await + } + async fn scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::Pager)> { + self.inner.scan(path, args).await + } + fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingPager)> { + self.inner.blocking_list(path, args) + } + fn blocking_scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::BlockingPager)> { + self.inner.blocking_scan(path, args) + } } ``` ## Usage of ops To reduce the understanding overhead, we move all `OpXxx` into `opendal::ops` now. User may need to change: ```diff - use opendal::OpWrite; + use opendal::ops::OpWrite; ``` ## Usage of RetryLayer `backon` is the implementation detail of our `RetryLayer`, so we hide it from our public API. Users of `RetryLayer` need to change the code like: ```diff - RetryLayer::new(backon::ExponentialBackoff::default()) + RetryLayer::new() ``` # Upgrade to v0.26 In v0.26 we have replaced all internal dynamic dispatch usage with static dispatch. With this change, we can ensure that all operations performed inside OpenDAL are zero cost. Due to this change, we have to refactor the logic of `Operator`'s init logic. In v0.26, we added `opendal::Builder` trait and `opendal::OperatorBuilder`. For the first glance, the only change to existing code will be like: ```diff - let op = Operator::new(builder.build()?); + let op = Operator::new(builder.build()?).finish(); ``` By adding a `finish()` call, we will erase all generic types so that `Operator` can still be easily used everywhere as before. ## Accessor In v0.26, `Accessor` has been changed into trait with associated types. All services need to declare the types returned as `Reader` or `BlockingReader`: ```rust pub trait Accessor: Send + Sync + Debug + Unpin + 'static { type Reader: output::Read; type BlockingReader: output::BlockingRead; } ``` If your service doesn't support `read` or `blocking_read`, we can use `()` to represent a dummy reader: ```rust impl Accessor for MyDummyAccessor { type Reader = (); type BlockingReader = (); } ``` ## Layer As described before, OpenDAL prefer to use static dispatch. Layers are required to implement the new `Layer` and `LayeredAccessor` trait: ```rust pub trait Layer { type LayeredAccessor: Accessor; fn layer(&self, inner: A) -> Self::LayeredAccessor; } #[async_trait] pub trait LayeredAccessor: Send + Sync + Debug + Unpin + 'static { type Inner: Accessor; type Reader: output::Read; type BlockingReader: output::BlockingRead; } ``` `LayeredAccessor` is a wrapper of `Accessor` with the typed `Innder`. All methods that not implemented will be forward to inner instead. ## Builder Since v0.26, we implement `opendal::Builder` for all services, and services' mod will not be exported. ```diff - use opendal::services::s3::Builder; + use opendal::services::S3; ``` ## Conclusion Sorry again for the big changes in this release. It's a big step for OpenDAL to work in more critical systems. # Upgrade to v0.25 In v0.25, we bring the same feature sets from `ObjectReader` to `BlockingObjectReader`. Due to this change, all code that depends on `BlockingBytesReader` should be refactored. - `BlockingBytesReader` => `input::BlockingReader` - `BlockingOutputBytesReader` => `output::BlockingReader` Most changes only happen inside. Users not using `opendal::raw::*` will not be affected. Apart from this change, we refactored s3 credential loading logic. After this change, we can disable the config load instead of the credential methods. - `builder.disable_credential_loader` => `builder.disable_config_load` # Upgrade to v0.24 In v0.24, we made a big refactor on our internal IO-related traits. In this version, we split our IO traits into `input` and `output` versions: Take `Reader` as an example: `input::Reader` is the user input reader, which only requires `futures::AsyncRead + Send`. `output::Reader` is the reader returned by `OpenDAL`, which implements `futures::AsyncRead`, `futures::AsyncSeek`, and `futures::Stream>`. Besides, `output::Reader` also implements `Send + Sync`, which makes it useful for users. Due to this change, all code that depends on `BytesReader` should be refactored. - `BytesReader` => `input::Reader` - `OutputBytesReader` => `output::Reader` Thanks to the change of IO trait split, we make `ObjectReader` implements all needed traits: - `futures::AsyncRead` - `futures::AsyncSeek` - `futures::Stream>` Thus, we removed the `seekable_reader` API. They can be replaced by `range_reader`: - `o.seekable_reader` => `o.range_reader` Most changes only happen inside. Users not using `opendal::raw::*` will not be affected. Sorry for the inconvenience. I think those changes are required and make OpenDAL better! Welcome any comments at [Discussion](https://github.com/apache/opendal/discussions). # Upgrade to v0.21 v0.21 is an internal refactor version of OpenDAL. In this version, we refactored our error handling and our `Accessor` APIs. Thanks to those internal changes, we added an object-level metadata cache, making it nearly zero cost to reuse existing metadata continuously. Let's start with our errors. ## Error Handling As described in [RFC-0977: Refactor Error](https://opendal.apache.org/rfcs/0977-refactor-error.html), we refactor opendal error by a new error called [`opendal::Error`](https://opendal.apache.org/opendal/struct.Error.html). This change will affect all APIs that are used to return `io::Error`. To migrate this, please replace `std::io::Error` with `opendal::Error`: ```diff - use std::io::Result; + use opendal::Result; ``` And the following error kinds should be updated: - `std::io::ErrorKind::NotFound` => `opendal::ErrorKind::ObjectNotFound` - `std::io::ErrorKind::PermissionDenied` => `opendal::ErrorKind::ObjectPermissionDenied` And since v0.21, we will return errors `ObjectIsADirectory` and `ObjectNotADirectory` instead of `anyhow::Error`. ## Accessor API In v0.21, we refactor the whole `Accessor`'s API: ```diff - async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result + async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result ``` Since v0.21, we will return a reply struct for different operations called `RpWrite` instead of an exact type. We can split OpenDAL's public API and raw API with this change. ## ObjectList and Page Since v0.21, `Accessor` will return `Pager` for `List`: ```diff - async fn list(&self, path: &str, args: OpList) -> Result + async fn list(&self, path: &str, args: OpList) -> Result<(RpList, output::Pager)> ``` And `Object` will return an `ObjectLister` which is built upon `Page`: ```rust pub async fn list(&self) -> Result { ... } ``` `ObjectLister` can be used as an object stream as before. It also provides the function `next_page` to get the underlying pages directly: ```rust impl ObjectLister { pub async fn next_page(&mut self) -> Result>>; } ``` ## Code Layout Since v0.21, we have categorized all APIs into `public` and `raw`. Public APIs are exposed under `opendal::Xxx`; they are user-face APIs that are easy to use and understand. Raw APIs are exposed under `opendal::raw::Xxx`; they are implementation details for underlying services and layers. Please replace all usage of `opendal::io_util::*` and `opendal::http_util::*` to `opendal::raw::*` instead. With this change, new users of OpenDAL maybe be it easier to get started. ## Summary Sorry for introducing too much breaking change in a single version. This version can be a solid version for preparing OpenDAL v1.0. # Upgrade to v0.20 v0.20 is a big release that we introduce a lot of performance related changes. To make the best of information from `read` operation, we propose and implemented [RFC-0926: Object Reader](https://opendal.apache.org/rfcs/0926-object-reader.html). By this RFC, we can fetch content length from `ObjectReader` now! ```rust pub struct ObjectReader { inner: BytesReader meta: ObjectMetadata, } impl ObjectReader { pub fn content_length(&self) -> u64 {} pub fn last_modified(&self) -> Option {} pub fn etag(&self) -> Option {} } ``` To make this happen, we changed our `Accessor` API: ```diff - async fn read(&self, path: &str, args: OpRead) -> Result {} + async fn read(&self, path: &str, args: OpRead) -> Result {} ``` All layers should be updated to meet this change. Also, it's required to return `content_length` while building `ObjectReader`. Please make sure the returning `ObjectMetadata` is used correctly. # Upgrade to v0.19 OpenDAL deprecate some features: - `serde`: We will enable it by default. - `layers-retry`: We will enable retry support by default. - `layers-metadata-cache`: We will enable it by default. Deprecated types like `DirEntry` has been removed. # Upgrade to v0.18 OpenDAL v0.18 introduces the following breaking changes: - Deprecated feature flag `services-http` has been removed. - All `DirXxx` items have been renamed to `ObjectXxx` to make them more consistent. - `DirEntry` -> `Entry` - `DirStream` -> `ObjectStream` - `DirStreamer` -> `ObjectStream` - `DirIterate` -> `ObjectIterate` - `DirIterator` -> `ObjectIterator` Besides, we also make a big change to our `Entry` API. Since v0.18, we can fully reuse the metadata that fetched during `list`. Take `entry.content_length()` for example: - If `content_length` is already known, we will return directly. - If not, we will check if the object entry is `complete`: - If `complete`, the entry already fetched all metadata that it could have, return directly. - If not, we will send a `stat` call to get the `metadata` and refresh our cache. This change means: - All API like `content_length` will be changed into async functions. - `metadata` and `blocking_metadata` will not return errors anymore. - To retrieve the latest meta, please use `entry.into_object().metadata()` instead. # Upgrade to v0.17 OpenDAL v0.17 refactor the `Accessor` to make space for future features. We move `path String` out of the `OpXxx` to function args so that we don't need to clone twice. ```diff - async fn read(&self, args: OpRead) -> Result + async fn read(&self, path: &str, args: OpRead) -> Result ``` For more information about this change, please refer to [RFC-0661: Path In Accessor](https://opendal.apache.org/rfcs/0661-path-in-accessor.html). And since OpenDAL v0.17, we will use `rustls` as default tls engine for our underlying http client. Since this release, we will not depend on `openssl` anymore. # Upgrade to v0.16 OpenDAL v0.16 refactor the internal implementation of `http` service. Since v0.16, http service can be used directly without enabling `services-http` feature. Accompany by these changes, http service has the following breaking changes: - `services-http` feature has been deprecated. Enabling `services-http` is a no-op now. - http service is read only services and can't be used to `list` or `write`. OpenDAL introduces a new layer `ImmutableIndexLayer` that can add `list` capability for services: ```rust use opendal::layers::ImmutableIndexLayer; use opendal::Operator; use opendal::Scheme; async fn main() { let mut iil = ImmutableIndexLayer::default(); for i in ["file", "dir/", "dir/file", "dir_without_prefix/file"] { iil.insert(i.to_string()) } let op = Operator::from_env(Scheme::Http)?.layer(iil); } ``` For more information about this change, please refer to [RFC-0627: Split Capabilities](https://opendal.apache.org/rfcs/0627-split-capabilities.html). # Upgrade to v0.14 OpenDAL v0.14 removed all deprecated APIs in previous versions, including: - `Operator::with_backoff` in v0.13 - All services `Builder::finish()` in v0.12 - All services `Backend::build()` in v0.12 Please visit related version's upgrade guide for migration. And in OpenDAL v0.14, we introduce a break change for `write` operations. ```diff pub trait Accessor { - async fn write(&self, args: &OpWrite) -> Result {} + async fn write(&self, args: &OpWrite, r: BytesReader) -> Result {} } ``` The following APIs have affected by this change: - `Object::write` now accept `impl Into>` instead of `AsRef<&[u8]>` - `Object::writer` has been removed. - `Object::write_from` has been added to support write from a reader. - All layers should be refactored to adapt new `Accessor` trait. For more information about this change, please refer to [RFC-0554: Write Refactor](https://opendal.apache.org/rfcs/0554-write-refactor.html). # Upgrade to v0.13 OpenDAL deprecate `Operator::with_backoff` since v0.13. Please use [`RetryLayer`](https://opendal.apache.org/opendal/layers/struct.RetryLayer.html) instead: ```rust use anyhow::Result; use backon::ExponentialBackoff; use opendal::layers::RetryLayer; use opendal::Operator; use opendal::Scheme; let _ = Operator::from_env(Scheme::Fs) .expect("must init") .layer(RetryLayer::new(ExponentialBackoff::default())); ``` # Upgrade to v0.12 OpenDAL introduces breaking changes for services initiation. Since v0.12, `Operator::new` will accept `impl Accessor + 'static` instead of `Arc`: ```rust impl Operator { pub fn new(accessor: impl Accessor + 'static) -> Self { .. } } ``` Every service's `Builder` now have a `build()` API which can be run without async: ```rust let mut builder = fs::Builder::default(); let op: Operator = Operator::new(builder.build()?); ``` Along with these changes, `Operator::from_iter` and `Operator::from_env` now is a blocking API too. For more information about this change, please refer to [RFC-0501: New Builder](https://opendal.apache.org/rfcs/0501-new-builder.html). The following APIs have been deprecated: - All services `Builder::finish()` (replaced by `Builder::build()`) - All services `Backend::build()` (replace by `Builder::default()`) The following APIs have been removed: - public struct `Metadata` (deprecated in v0.8, replaced by `ObjectMetadata`) # Upgrade to v0.8 OpenDAL introduces a breaking change of `list` related operations in v0.8. Since v0.8, `list` will return `DirStreamer` instead: ```rust pub trait Accessor: Send + Sync + Debug { async fn list(&self, args: &OpList) -> Result {} } ``` `DirStreamer` streams `DirEntry` which carries `ObjectMode`, so that we don't need an extra call to get object mode: ```rust impl DirEntry { pub fn mode(&self) -> ObjectMode { self.mode } } ``` And `DirEntry` can be converted into `Object` without overhead: ```rust let o: Object = de.into() ``` Since `v0.8`, `opendal::Metadata` has been deprecated by `opendal::ObjectMetadata`. # Upgrade to v0.7 OpenDAL introduces a breaking change of `decompress_read` related in v0.7. Since v0.7, `decompress_read` and `decompress_reader` will return `Ok(None)` while OpenDAL can't detect the correct compress algorithm. ```rust impl Object { pub async fn decompress_read(&self) -> Result>> {} pub async fn decompress_reader(&self) -> Result> {} } ``` So users should match and check the `None` case: ```rust let bs = o.decompress_read().await?.expect("must have valid compress algorithm"); ``` # Upgrade to v0.4 OpenDAL introduces many breaking changes in v0.4. ## Object::reader() is not `AsyncSeek` anymore Since v0.4, `Object::reader()` will return `impl BytesRead` instead of `Reader` that implements `AsyncRead` and `AsyncSeek`. Users who want `AsyncSeek` please wrapped with `opendal::io_util::seekable_read`: ```rust use opendal::io_util::seekable_read; let o = op.object("test"); let mut r = seekable_read(&o, 10..); r.seek(SeekFrom::Current(10)).await?; let mut bs = vec![0;10]; r.read(&mut bs).await?; ``` ## Use RangeBounds instead Since v0.4, the following APIs will be removed. - `Object::limited_reader(size: u64)` - `Object::offset_reader(offset: u64)` - `Object::range_reader(offset: u64, size: u64)` Instead, OpenDAL is providing a more general `range_reader` powered by `RangeBounds`: ```rust pub async fn range_reader(&self, range: impl RangeBounds) -> Result ``` Users can use their familiar rust range syntax: ```rust let r = o.range_reader(1024..2048).await?; ``` ## Return io::Result instead Since v0.4, all functions in OpenDAL will return `std::io::Result` instead. Please check via `std::io::ErrorKind` directly: ```rust use std::io::ErrorKind; if let Err(e) = op.object("test_file").metadata().await { if e.kind() == ErrorKind::NotFound { println!("object not exist") } } ``` ## Removing Credential Since v0.4, `Credential` has been removed, please use the API provided by `Builder` directly. ```rust builder.access_key_id("access_key_id"); builder.secret_access_key("secret_access_key"); ``` ## Write returns `BytesWriter` instead Since v0.4, `Accessor::write` will return a `BytesWriter` instead accepting a `BoxedAsyncReader`. Along with this change, the old `Writer` has been replaced by a new set of write functions: ```rust pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {} pub async fn writer(&self, size: u64) -> Result {} ``` Users can write into an object more easily: ```rust let _ = op.object("path/to/file").write("Hello, World!").await?; ``` ## `io_util` replaces `readers` Since v0.4, mod `io_util` will replace `readers`. In `io_utils`, OpenDAL provides helpful functions like: - `into_reader`: Convert `BytesStream` into `BytesRead` - `into_sink`: Convert `BytesWrite` into `BytesSink` - `into_stream`: Convert `BytesRead` into `BytesStream` - `into_writer`: Convert `BytesSink` into `BytesWrite` - `observe_read`: Add callback for `BytesReader` - `observe_write`: Add callback for `BytesWrite` ## New type alias For better naming, types that OpenDAL returns have been renamed: - `AsyncRead + Unpin + Send` => `BytesRead` - `BoxedAsyncReader` => `BytesReader` - `AsyncWrite + Unpin + Send` => `BytesWrite` - `BoxedAsyncWriter` => `BytesWriter` - `ObjectStream` => `ObjectStreamer` opendal-0.52.0/src/layers/async_backtrace.rs000064400000000000000000000157671046102023000171220ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; /// Add Efficient, logical 'stack' traces of async functions for the underlying services. /// /// # Async Backtrace /// /// async-backtrace allows developers to get a stack trace of the async functions. /// Read more about [async-backtrace](https://docs.rs/async-backtrace/latest/async_backtrace/) /// /// # Examples /// /// ```no_run /// # use opendal::layers::AsyncBacktraceLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(AsyncBacktraceLayer::default()) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Clone, Default)] pub struct AsyncBacktraceLayer; impl Layer for AsyncBacktraceLayer { type LayeredAccess = AsyncBacktraceAccessor; fn layer(&self, accessor: A) -> Self::LayeredAccess { AsyncBacktraceAccessor { inner: accessor } } } #[derive(Debug, Clone)] pub struct AsyncBacktraceAccessor { inner: A, } impl LayeredAccess for AsyncBacktraceAccessor { type Inner = A; type Reader = AsyncBacktraceWrapper; type BlockingReader = AsyncBacktraceWrapper; type Writer = AsyncBacktraceWrapper; type BlockingWriter = AsyncBacktraceWrapper; type Lister = AsyncBacktraceWrapper; type BlockingLister = AsyncBacktraceWrapper; type Deleter = AsyncBacktraceWrapper; type BlockingDeleter = AsyncBacktraceWrapper; fn inner(&self) -> &Self::Inner { &self.inner } #[async_backtrace::framed] async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner .read(path, args) .await .map(|(rp, r)| (rp, AsyncBacktraceWrapper::new(r))) } #[async_backtrace::framed] async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner .write(path, args) .await .map(|(rp, r)| (rp, AsyncBacktraceWrapper::new(r))) } #[async_backtrace::framed] async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner.copy(from, to, args).await } #[async_backtrace::framed] async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner.rename(from, to, args).await } #[async_backtrace::framed] async fn stat(&self, path: &str, args: OpStat) -> Result { self.inner.stat(path, args).await } #[async_backtrace::framed] async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner .delete() .await .map(|(rp, r)| (rp, AsyncBacktraceWrapper::new(r))) } #[async_backtrace::framed] async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner .list(path, args) .await .map(|(rp, r)| (rp, AsyncBacktraceWrapper::new(r))) } #[async_backtrace::framed] async fn presign(&self, path: &str, args: OpPresign) -> Result { self.inner.presign(path, args).await } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, AsyncBacktraceWrapper::new(r))) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner .blocking_write(path, args) .map(|(rp, r)| (rp, AsyncBacktraceWrapper::new(r))) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner .blocking_list(path, args) .map(|(rp, r)| (rp, AsyncBacktraceWrapper::new(r))) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner .blocking_delete() .map(|(rp, r)| (rp, AsyncBacktraceWrapper::new(r))) } } pub struct AsyncBacktraceWrapper { inner: R, } impl AsyncBacktraceWrapper { fn new(inner: R) -> Self { Self { inner } } } impl oio::Read for AsyncBacktraceWrapper { #[async_backtrace::framed] async fn read(&mut self) -> Result { self.inner.read().await } } impl oio::BlockingRead for AsyncBacktraceWrapper { fn read(&mut self) -> Result { self.inner.read() } } impl oio::Write for AsyncBacktraceWrapper { #[async_backtrace::framed] async fn write(&mut self, bs: Buffer) -> Result<()> { self.inner.write(bs).await } #[async_backtrace::framed] async fn close(&mut self) -> Result { self.inner.close().await } #[async_backtrace::framed] async fn abort(&mut self) -> Result<()> { self.inner.abort().await } } impl oio::BlockingWrite for AsyncBacktraceWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { self.inner.write(bs) } fn close(&mut self) -> Result { self.inner.close() } } impl oio::List for AsyncBacktraceWrapper { #[async_backtrace::framed] async fn next(&mut self) -> Result> { self.inner.next().await } } impl oio::BlockingList for AsyncBacktraceWrapper { fn next(&mut self) -> Result> { self.inner.next() } } impl oio::Delete for AsyncBacktraceWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } #[async_backtrace::framed] async fn flush(&mut self) -> Result { self.inner.flush().await } } impl oio::BlockingDelete for AsyncBacktraceWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } fn flush(&mut self) -> Result { self.inner.flush() } } opendal-0.52.0/src/layers/await_tree.rs000064400000000000000000000177021046102023000161210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use await_tree::InstrumentAwait; use futures::Future; use crate::raw::*; use crate::*; /// Add an Instrument await-tree for actor-based applications to the underlying services. /// /// # AwaitTree /// /// await-tree allows developers to dump this execution tree at runtime, /// with the span of each Future annotated by instrument_await. /// Read more about [await-tree](https://docs.rs/await-tree/latest/await_tree/) /// /// # Examples /// /// ```no_run /// # use opendal::layers::AwaitTreeLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(AwaitTreeLayer::new()) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Clone, Default)] pub struct AwaitTreeLayer {} impl AwaitTreeLayer { /// Create a new `AwaitTreeLayer`. pub fn new() -> Self { Self {} } } impl Layer for AwaitTreeLayer { type LayeredAccess = AwaitTreeAccessor; fn layer(&self, accessor: A) -> Self::LayeredAccess { AwaitTreeAccessor { inner: accessor } } } #[derive(Debug, Clone)] pub struct AwaitTreeAccessor { inner: A, } impl LayeredAccess for AwaitTreeAccessor { type Inner = A; type Reader = AwaitTreeWrapper; type BlockingReader = AwaitTreeWrapper; type Writer = AwaitTreeWrapper; type BlockingWriter = AwaitTreeWrapper; type Lister = AwaitTreeWrapper; type BlockingLister = AwaitTreeWrapper; type Deleter = AwaitTreeWrapper; type BlockingDeleter = AwaitTreeWrapper; fn inner(&self) -> &Self::Inner { &self.inner } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner .read(path, args) .instrument_await(format!("opendal::{}", Operation::Read)) .await .map(|(rp, r)| (rp, AwaitTreeWrapper::new(r))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner .write(path, args) .instrument_await(format!("opendal::{}", Operation::Write)) .await .map(|(rp, r)| (rp, AwaitTreeWrapper::new(r))) } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner() .copy(from, to, args) .instrument_await(format!("opendal::{}", Operation::Copy)) .await } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner() .rename(from, to, args) .instrument_await(format!("opendal::{}", Operation::Rename)) .await } async fn stat(&self, path: &str, args: OpStat) -> Result { self.inner .stat(path, args) .instrument_await(format!("opendal::{}", Operation::Stat)) .await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner .delete() .instrument_await(format!("opendal::{}", Operation::Delete)) .await .map(|(rp, r)| (rp, AwaitTreeWrapper::new(r))) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner .list(path, args) .instrument_await(format!("opendal::{}", Operation::List)) .await .map(|(rp, r)| (rp, AwaitTreeWrapper::new(r))) } async fn presign(&self, path: &str, args: OpPresign) -> Result { self.inner .presign(path, args) .instrument_await(format!("opendal::{}", Operation::Presign)) .await } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, AwaitTreeWrapper::new(r))) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner .blocking_write(path, args) .map(|(rp, r)| (rp, AwaitTreeWrapper::new(r))) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner .blocking_list(path, args) .map(|(rp, r)| (rp, AwaitTreeWrapper::new(r))) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner .blocking_delete() .map(|(rp, r)| (rp, AwaitTreeWrapper::new(r))) } } pub struct AwaitTreeWrapper { inner: R, } impl AwaitTreeWrapper { fn new(inner: R) -> Self { Self { inner } } } impl oio::Read for AwaitTreeWrapper { async fn read(&mut self) -> Result { self.inner .read() .instrument_await(format!("opendal::{}", Operation::ReaderRead)) .await } } impl oio::BlockingRead for AwaitTreeWrapper { fn read(&mut self) -> Result { self.inner.read() } } impl oio::Write for AwaitTreeWrapper { fn write(&mut self, bs: Buffer) -> impl Future> + MaybeSend { self.inner .write(bs) .instrument_await(format!("opendal::{}", Operation::WriterWrite.into_static())) } fn abort(&mut self) -> impl Future> + MaybeSend { self.inner .abort() .instrument_await(format!("opendal::{}", Operation::WriterAbort.into_static())) } fn close(&mut self) -> impl Future> + MaybeSend { self.inner .close() .instrument_await(format!("opendal::{}", Operation::WriterClose.into_static())) } } impl oio::BlockingWrite for AwaitTreeWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { self.inner.write(bs) } fn close(&mut self) -> Result { self.inner.close() } } impl oio::List for AwaitTreeWrapper { async fn next(&mut self) -> Result> { self.inner .next() .instrument_await(format!("opendal::{}", Operation::ListerNext)) .await } } impl oio::BlockingList for AwaitTreeWrapper { fn next(&mut self) -> Result> { self.inner.next() } } impl oio::Delete for AwaitTreeWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } async fn flush(&mut self) -> Result { self.inner .flush() .instrument_await(format!("opendal::{}", Operation::DeleterFlush)) .await } } impl oio::BlockingDelete for AwaitTreeWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } fn flush(&mut self) -> Result { self.inner.flush() } } opendal-0.52.0/src/layers/blocking.rs000064400000000000000000000253011046102023000155570ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use tokio::runtime::Handle; use crate::raw::*; use crate::*; /// Add blocking API support for non-blocking services. /// /// # Notes /// /// - Please only enable this layer when the underlying service does not support blocking. /// /// # Examples /// /// ## In async context /// /// BlockingLayer will use current async context's runtime to handle the async calls. /// /// ```rust,no_run /// # use opendal::layers::BlockingLayer; /// # use opendal::services; /// # use opendal::BlockingOperator; /// # use opendal::Operator; /// # use opendal::Result; /// /// #[tokio::main] /// async fn main() -> Result<()> { /// // Create fs backend builder. /// let mut builder = services::S3::default().bucket("test").region("us-east-1"); /// /// // Build an `BlockingOperator` with blocking layer to start operating the storage. /// let _: BlockingOperator = Operator::new(builder)? /// .layer(BlockingLayer::create()?) /// .finish() /// .blocking(); /// /// Ok(()) /// } /// ``` /// /// ## In async context with blocking functions /// /// If `BlockingLayer` is called in blocking function, please fetch a [`tokio::runtime::EnterGuard`] /// first. You can use [`Handle::try_current`] first to get the handle and then call [`Handle::enter`]. /// This often happens in the case that async function calls blocking function. /// /// ```rust,no_run /// # use opendal::layers::BlockingLayer; /// # use opendal::services; /// # use opendal::BlockingOperator; /// # use opendal::Operator; /// # use opendal::Result; /// /// #[tokio::main] /// async fn main() -> Result<()> { /// let _ = blocking_fn()?; /// Ok(()) /// } /// /// fn blocking_fn() -> Result { /// // Create fs backend builder. /// let mut builder = services::S3::default().bucket("test").region("us-east-1"); /// /// let handle = tokio::runtime::Handle::try_current().unwrap(); /// let _guard = handle.enter(); /// // Build an `BlockingOperator` with blocking layer to start operating the storage. /// let op: BlockingOperator = Operator::new(builder)? /// .layer(BlockingLayer::create()?) /// .finish() /// .blocking(); /// Ok(op) /// } /// ``` /// /// ## In blocking context /// /// In a pure blocking context, we can create a runtime and use it to create the `BlockingLayer`. /// /// > The following code uses a global statically created runtime as an example, please manage the /// > runtime on demand. /// /// ```rust,no_run /// # use once_cell::sync::Lazy; /// # use opendal::layers::BlockingLayer; /// # use opendal::services; /// # use opendal::BlockingOperator; /// # use opendal::Operator; /// # use opendal::Result; /// /// static RUNTIME: Lazy = Lazy::new(|| { /// tokio::runtime::Builder::new_multi_thread() /// .enable_all() /// .build() /// .unwrap() /// }); /// /// fn main() -> Result<()> { /// // Create fs backend builder. /// let mut builder = services::S3::default().bucket("test").region("us-east-1"); /// /// // Fetch the `EnterGuard` from global runtime. /// let _guard = RUNTIME.enter(); /// // Build an `BlockingOperator` with blocking layer to start operating the storage. /// let _: BlockingOperator = Operator::new(builder)? /// .layer(BlockingLayer::create()?) /// .finish() /// .blocking(); /// /// Ok(()) /// } /// ``` #[derive(Debug, Clone)] pub struct BlockingLayer { handle: Handle, } impl BlockingLayer { /// Create a new `BlockingLayer` with the current runtime's handle pub fn create() -> Result { Ok(Self { handle: Handle::try_current() .map_err(|_| Error::new(ErrorKind::Unexpected, "failed to get current handle"))?, }) } } impl Layer for BlockingLayer { type LayeredAccess = BlockingAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { BlockingAccessor { inner, handle: self.handle.clone(), } } } #[derive(Clone, Debug)] pub struct BlockingAccessor { inner: A, handle: Handle, } impl LayeredAccess for BlockingAccessor { type Inner = A; type Reader = A::Reader; type BlockingReader = BlockingWrapper; type Writer = A::Writer; type BlockingWriter = BlockingWrapper; type Lister = A::Lister; type BlockingLister = BlockingWrapper; type Deleter = A::Deleter; type BlockingDeleter = BlockingWrapper; fn inner(&self) -> &Self::Inner { &self.inner } fn info(&self) -> Arc { let mut meta = self.inner.info().as_ref().clone(); meta.full_capability_mut().blocking = true; meta.into() } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.inner.create_dir(path, args).await } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner.read(path, args).await } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner.write(path, args).await } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner.copy(from, to, args).await } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner.rename(from, to, args).await } async fn stat(&self, path: &str, args: OpStat) -> Result { self.inner.stat(path, args).await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner.delete().await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner.list(path, args).await } async fn presign(&self, path: &str, args: OpPresign) -> Result { self.inner.presign(path, args).await } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.handle.block_on(self.inner.create_dir(path, args)) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.handle.block_on(async { let (rp, reader) = self.inner.read(path, args).await?; let blocking_reader = Self::BlockingReader::new(self.handle.clone(), reader); Ok((rp, blocking_reader)) }) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.handle.block_on(async { let (rp, writer) = self.inner.write(path, args).await?; let blocking_writer = Self::BlockingWriter::new(self.handle.clone(), writer); Ok((rp, blocking_writer)) }) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.handle.block_on(self.inner.copy(from, to, args)) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.handle.block_on(self.inner.rename(from, to, args)) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.handle.block_on(self.inner.stat(path, args)) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.handle.block_on(async { let (rp, writer) = self.inner.delete().await?; let blocking_deleter = Self::BlockingDeleter::new(self.handle.clone(), writer); Ok((rp, blocking_deleter)) }) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.handle.block_on(async { let (rp, lister) = self.inner.list(path, args).await?; let blocking_lister = Self::BlockingLister::new(self.handle.clone(), lister); Ok((rp, blocking_lister)) }) } } pub struct BlockingWrapper { handle: Handle, inner: I, } impl BlockingWrapper { fn new(handle: Handle, inner: I) -> Self { Self { handle, inner } } } impl oio::BlockingRead for BlockingWrapper { fn read(&mut self) -> Result { self.handle.block_on(self.inner.read()) } } impl oio::BlockingWrite for BlockingWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { self.handle.block_on(self.inner.write(bs)) } fn close(&mut self) -> Result { self.handle.block_on(self.inner.close()) } } impl oio::BlockingList for BlockingWrapper { fn next(&mut self) -> Result> { self.handle.block_on(self.inner.next()) } } impl oio::BlockingDelete for BlockingWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } fn flush(&mut self) -> Result { self.handle.block_on(self.inner.flush()) } } #[cfg(test)] mod tests { use once_cell::sync::Lazy; use super::*; use crate::types::Result; static RUNTIME: Lazy = Lazy::new(|| { tokio::runtime::Builder::new_multi_thread() .enable_all() .build() .unwrap() }); fn create_blocking_layer() -> Result { let _guard = RUNTIME.enter(); BlockingLayer::create() } #[test] fn test_blocking_layer_in_blocking_context() { // create in a blocking context should fail let layer = BlockingLayer::create(); assert!(layer.is_err()); // create in an async context and drop in a blocking context let layer = create_blocking_layer(); assert!(layer.is_ok()) } #[test] fn test_blocking_layer_in_async_context() { // create and drop in an async context let _guard = RUNTIME.enter(); let layer = BlockingLayer::create(); assert!(layer.is_ok()); } } opendal-0.52.0/src/layers/capability_check.rs000064400000000000000000000230641046102023000172510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::layers::correctness_check::new_unsupported_error; use crate::raw::*; use std::fmt::{Debug, Formatter}; use std::sync::Arc; /// Add an extra capability check layer for every operation /// /// Similar to `CorrectnessChecker`, Before performing any operations, this layer will first verify /// its arguments against the capability of the underlying service. If the arguments is not supported, /// an error will be returned directly. /// /// Notes /// /// There are two main differences between this checker with the `CorrectnessChecker`: /// 1. This checker provides additional checks for capabilities like write_with_content_type and /// list_with_versions, among others. These capabilities do not affect data integrity, even if /// the underlying storage services do not support them. /// /// 2. OpenDAL doesn't apply this checker by default. Users can enable this layer if they want to /// enforce stricter requirements. /// /// # examples /// /// ```no_run /// # use opendal::layers::CapabilityCheckLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// use opendal::layers::CapabilityCheckLayer; /// let _ = Operator::new(services::Memory::default())? /// .layer(CapabilityCheckLayer) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Default)] pub struct CapabilityCheckLayer; impl Layer for CapabilityCheckLayer { type LayeredAccess = CapabilityAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { CapabilityAccessor { info: inner.info(), inner, } } } pub struct CapabilityAccessor { info: Arc, inner: A, } impl Debug for CapabilityAccessor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("CapabilityCheckAccessor") .field("inner", &self.inner) .finish_non_exhaustive() } } impl LayeredAccess for CapabilityAccessor { type Inner = A; type Reader = A::Reader; type Writer = A::Writer; type Lister = A::Lister; type Deleter = A::Deleter; type BlockingReader = A::BlockingReader; type BlockingWriter = A::BlockingWriter; type BlockingLister = A::BlockingLister; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } async fn read(&self, path: &str, args: OpRead) -> crate::Result<(RpRead, Self::Reader)> { self.inner.read(path, args).await } async fn write(&self, path: &str, args: OpWrite) -> crate::Result<(RpWrite, Self::Writer)> { let capability = self.info.full_capability(); if !capability.write_with_content_type && args.content_type().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Write, "content_type", )); } if !capability.write_with_cache_control && args.cache_control().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Write, "cache_control", )); } if !capability.write_with_content_disposition && args.content_disposition().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Write, "content_disposition", )); } self.inner.write(path, args).await } async fn delete(&self) -> crate::Result<(RpDelete, Self::Deleter)> { self.inner.delete().await } async fn list(&self, path: &str, args: OpList) -> crate::Result<(RpList, Self::Lister)> { let capability = self.info.full_capability(); if !capability.list_with_versions && args.versions() { return Err(new_unsupported_error( self.info.as_ref(), Operation::List, "version", )); } self.inner.list(path, args).await } fn blocking_read( &self, path: &str, args: OpRead, ) -> crate::Result<(RpRead, Self::BlockingReader)> { self.inner().blocking_read(path, args) } fn blocking_write( &self, path: &str, args: OpWrite, ) -> crate::Result<(RpWrite, Self::BlockingWriter)> { let capability = self.info.full_capability(); if !capability.write_with_content_type && args.content_type().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::BlockingWrite, "content_type", )); } if !capability.write_with_cache_control && args.cache_control().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::BlockingWrite, "cache_control", )); } if !capability.write_with_content_disposition && args.content_disposition().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::BlockingWrite, "content_disposition", )); } self.inner.blocking_write(path, args) } fn blocking_delete(&self) -> crate::Result<(RpDelete, Self::BlockingDeleter)> { self.inner.blocking_delete() } fn blocking_list( &self, path: &str, args: OpList, ) -> crate::Result<(RpList, Self::BlockingLister)> { let capability = self.info.full_capability(); if !capability.list_with_versions && args.versions() { return Err(new_unsupported_error( self.info.as_ref(), Operation::BlockingList, "version", )); } self.inner.blocking_list(path, args) } } #[cfg(test)] mod tests { use super::*; use crate::{Capability, ErrorKind, Operator}; #[derive(Debug)] struct MockService { capability: Capability, } impl Access for MockService { type Reader = oio::Reader; type Writer = oio::Writer; type Lister = oio::Lister; type Deleter = oio::Deleter; type BlockingReader = oio::BlockingReader; type BlockingWriter = oio::BlockingWriter; type BlockingLister = oio::BlockingLister; type BlockingDeleter = oio::BlockingDeleter; fn info(&self) -> Arc { let mut info = AccessorInfo::default(); info.set_native_capability(self.capability); info.into() } async fn write(&self, _: &str, _: OpWrite) -> crate::Result<(RpWrite, Self::Writer)> { Ok((RpWrite::new(), Box::new(()))) } async fn list(&self, _: &str, _: OpList) -> crate::Result<(RpList, Self::Lister)> { Ok((RpList {}, Box::new(()))) } } fn new_test_operator(capability: Capability) -> Operator { let srv = MockService { capability }; Operator::from_inner(Arc::new(srv)).layer(CapabilityCheckLayer) } #[tokio::test] async fn test_writer_with() { let op = new_test_operator(Capability { write: true, ..Default::default() }); let res = op.writer_with("path").content_type("type").await; assert!(res.is_err()); let res = op.writer_with("path").cache_control("cache").await; assert!(res.is_err()); let res = op .writer_with("path") .content_disposition("disposition") .await; assert!(res.is_err()); let op = new_test_operator(Capability { write: true, write_with_content_type: true, write_with_cache_control: true, write_with_content_disposition: true, ..Default::default() }); let res = op.writer_with("path").content_type("type").await; assert!(res.is_ok()); let res = op.writer_with("path").cache_control("cache").await; assert!(res.is_ok()); let res = op .writer_with("path") .content_disposition("disposition") .await; assert!(res.is_ok()); } #[tokio::test] async fn test_list_with() { let op = new_test_operator(Capability { list: true, ..Default::default() }); let res = op.list_with("path/").versions(true).await; assert!(res.is_err()); assert_eq!(res.unwrap_err().kind(), ErrorKind::Unsupported); let op = new_test_operator(Capability { list: true, list_with_versions: true, ..Default::default() }); let res = op.lister_with("path/").versions(true).await; assert!(res.is_ok()) } } opendal-0.52.0/src/layers/chaos.rs000064400000000000000000000131671046102023000150730ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use std::sync::Mutex; use rand::prelude::*; use rand::rngs::StdRng; use crate::raw::*; use crate::*; /// Inject chaos into underlying services for robustness test. /// /// # Chaos /// /// Chaos tests is a part of stress test. By generating errors at specified /// error ratio, we can reproduce underlying services error more reliable. /// /// Running tests under ChaosLayer will make your application more robust. /// /// For example: If we specify an error rate of 0.5, there is a 50% chance /// of an EOF error for every read operation. /// /// # Note /// /// For now, ChaosLayer only injects read operations. More operations may /// be added in the future. /// /// # Examples /// /// ```no_run /// # use opendal::layers::ChaosLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(ChaosLayer::new(0.1)) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Debug, Clone)] pub struct ChaosLayer { error_ratio: f64, } impl ChaosLayer { /// Create a new chaos layer with specified error ratio. /// /// # Panics /// /// Input error_ratio must in [0.0..=1.0] pub fn new(error_ratio: f64) -> Self { assert!( (0.0..=1.0).contains(&error_ratio), "error_ratio must between 0.0 and 1.0" ); Self { error_ratio } } } impl Layer for ChaosLayer { type LayeredAccess = ChaosAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { ChaosAccessor { inner, rng: StdRng::from_entropy(), error_ratio: self.error_ratio, } } } #[derive(Debug)] pub struct ChaosAccessor { inner: A, rng: StdRng, error_ratio: f64, } impl LayeredAccess for ChaosAccessor { type Inner = A; type Reader = ChaosReader; type BlockingReader = ChaosReader; type Writer = A::Writer; type BlockingWriter = A::BlockingWriter; type Lister = A::Lister; type BlockingLister = A::BlockingLister; type Deleter = A::Deleter; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner .read(path, args) .await .map(|(rp, r)| (rp, ChaosReader::new(r, self.rng.clone(), self.error_ratio))) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, ChaosReader::new(r, self.rng.clone(), self.error_ratio))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner.write(path, args).await } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner.blocking_write(path, args) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner.list(path, args).await } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner.blocking_list(path, args) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner.delete().await } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner.blocking_delete() } } /// ChaosReader will inject error into read operations. pub struct ChaosReader { inner: R, rng: Arc>, error_ratio: f64, } impl ChaosReader { fn new(inner: R, rng: StdRng, error_ratio: f64) -> Self { Self { inner, rng: Arc::new(Mutex::new(rng)), error_ratio, } } /// If I feel lucky, we can return the correct response. Otherwise, /// we need to generate an error. fn i_feel_lucky(&self) -> bool { let point = self.rng.lock().unwrap().gen_range(0..=100); point >= (self.error_ratio * 100.0) as i32 } fn unexpected_eof() -> Error { Error::new(ErrorKind::Unexpected, "I am your chaos!") .with_operation("chaos") .set_temporary() } } impl oio::Read for ChaosReader { async fn read(&mut self) -> Result { if self.i_feel_lucky() { self.inner.read().await } else { Err(Self::unexpected_eof()) } } } impl oio::BlockingRead for ChaosReader { fn read(&mut self) -> Result { if self.i_feel_lucky() { self.inner.read() } else { Err(Self::unexpected_eof()) } } } opendal-0.52.0/src/layers/complete.rs000064400000000000000000000477151046102023000156140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::oio::FlatLister; use crate::raw::oio::PrefixLister; use crate::raw::*; use crate::*; use std::cmp::Ordering; use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; /// Complete underlying services features so that users can use them in /// the same way. /// /// # Notes /// /// CompleteLayer is not a public accessible layer that can be used by /// external users. OpenDAL will make sure every accessor will apply this /// layer once and only once. /// /// # Internal /// /// So far `CompleteLayer` will do the following things: /// /// ## Stat Completion /// /// Not all services support stat dir natively, but we can simulate it via list. /// /// ## Read Completion /// /// OpenDAL requires all reader implements [`oio::Read`] and /// [`oio::BlockingRead`]. However, not all services have the /// capabilities. CompleteLayer will add those capabilities in /// a zero cost way. /// /// Underlying services will return [`AccessorInfo`] to indicate the /// features that returning readers support. /// /// - If both `seekable` and `streamable`, return directly. /// - If not `streamable`, with [`oio::into_read_from_stream`]. /// - If not `seekable`, with [`oio::into_seekable_read_by_range`] /// - If neither not supported, wrap both by_range and into_streamable. /// /// All implementations of Reader should be `zero cost`. In our cases, /// which means others must pay the same cost for the same feature provide /// by us. /// /// For examples, call `read` without `seek` should always act the same as /// calling `read` on plain reader. /// /// ### Read is Seekable /// /// We use [`Capability`] to decide the most suitable implementations. /// /// If [`Capability`] `read_can_seek` is true, we will open it with given args /// directly. Otherwise, we will pick a seekable reader implementation based /// on input range for it. /// /// - `Some(offset), Some(size)` => `RangeReader` /// - `Some(offset), None` and `None, None` => `OffsetReader` /// - `None, Some(size)` => get the total size first to convert as `RangeReader` /// /// No matter which reader we use, we will make sure the `read` operation /// is zero cost. /// /// ### Read is Streamable /// /// We use internal `AccessorHint::ReadStreamable` to decide the most /// suitable implementations. /// /// If [`Capability`] `read_can_next` is true, we will use existing reader /// directly. Otherwise, we will use transform this reader as a stream. /// /// ## List Completion /// /// There are two styles of list, but not all services support both of /// them. CompleteLayer will add those capabilities in a zero cost way. /// /// Underlying services will return [`Capability`] to indicate the /// features that returning listers support. /// /// - If support `list_with_recursive`, return directly. /// - if not, wrap with [`FlatLister`]. /// pub struct CompleteLayer; impl Layer for CompleteLayer { type LayeredAccess = CompleteAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { CompleteAccessor { info: inner.info(), inner: Arc::new(inner), } } } /// Provide complete wrapper for backend. pub struct CompleteAccessor { info: Arc, inner: Arc, } impl Debug for CompleteAccessor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { self.inner.fmt(f) } } impl CompleteAccessor { async fn complete_create_dir(&self, path: &str, args: OpCreateDir) -> Result { let capability = self.info.full_capability(); if capability.create_dir { return self.inner().create_dir(path, args).await; } if capability.write_can_empty && capability.list { let (_, mut w) = self.inner.write(path, OpWrite::default()).await?; oio::Write::close(&mut w).await?; return Ok(RpCreateDir::default()); } self.inner.create_dir(path, args).await } fn complete_blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { let capability = self.info.full_capability(); if capability.create_dir && capability.blocking { return self.inner().blocking_create_dir(path, args); } if capability.write_can_empty && capability.list && capability.blocking { let (_, mut w) = self.inner.blocking_write(path, OpWrite::default())?; oio::BlockingWrite::close(&mut w)?; return Ok(RpCreateDir::default()); } self.inner.blocking_create_dir(path, args) } async fn complete_stat(&self, path: &str, args: OpStat) -> Result { let capability = self.info.full_capability(); if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } // Forward to inner if create_dir is supported. if path.ends_with('/') && capability.create_dir { let meta = self.inner.stat(path, args).await?.into_metadata(); if meta.is_file() { return Err(Error::new( ErrorKind::NotFound, "stat expected a directory, but found a file", )); } return Ok(RpStat::new(meta)); } // Otherwise, we can simulate stat dir via `list`. if path.ends_with('/') && capability.list_with_recursive { let (_, mut l) = self .inner .list(path, OpList::default().with_recursive(true).with_limit(1)) .await?; return if oio::List::next(&mut l).await?.is_some() { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } else { Err(Error::new( ErrorKind::NotFound, "the directory is not found", )) }; } // Forward to underlying storage directly since we don't know how to handle stat dir. self.inner.stat(path, args).await } fn complete_blocking_stat(&self, path: &str, args: OpStat) -> Result { let capability = self.info.full_capability(); if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } // Forward to inner if create dir is supported. if path.ends_with('/') && capability.create_dir { let meta = self.inner.blocking_stat(path, args)?.into_metadata(); if meta.is_file() { return Err(Error::new( ErrorKind::NotFound, "stat expected a directory, but found a file", )); } return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } // Otherwise, we can simulate stat a dir path via `list`. if path.ends_with('/') && capability.list_with_recursive { let (_, mut l) = self .inner .blocking_list(path, OpList::default().with_recursive(true).with_limit(1))?; return if oio::BlockingList::next(&mut l)?.is_some() { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } else { Err(Error::new( ErrorKind::NotFound, "the directory is not found", )) }; } // Forward to underlying storage directly since we don't know how to handle stat dir. self.inner.blocking_stat(path, args) } async fn complete_list( &self, path: &str, args: OpList, ) -> Result<(RpList, CompleteLister)> { let cap = self.info.full_capability(); let recursive = args.recursive(); match (recursive, cap.list_with_recursive) { // - If service can list_with_recursive, we can forward list to it directly. (_, true) => { let (rp, p) = self.inner.list(path, args).await?; Ok((rp, CompleteLister::One(p))) } // If recursive is true but service can't list_with_recursive (true, false) => { // Forward path that ends with / if path.ends_with('/') { let p = FlatLister::new(self.inner.clone(), path); Ok((RpList::default(), CompleteLister::Two(p))) } else { let parent = get_parent(path); let p = FlatLister::new(self.inner.clone(), parent); let p = PrefixLister::new(p, path); Ok((RpList::default(), CompleteLister::Four(p))) } } // If recursive and service doesn't support list_with_recursive, we need to handle // list prefix by ourselves. (false, false) => { // Forward path that ends with / if path.ends_with('/') { let (rp, p) = self.inner.list(path, args).await?; Ok((rp, CompleteLister::One(p))) } else { let parent = get_parent(path); let (rp, p) = self.inner.list(parent, args).await?; let p = PrefixLister::new(p, path); Ok((rp, CompleteLister::Three(p))) } } } } fn complete_blocking_list( &self, path: &str, args: OpList, ) -> Result<(RpList, CompleteLister)> { let cap = self.info.full_capability(); let recursive = args.recursive(); match (recursive, cap.list_with_recursive) { // - If service can list_with_recursive, we can forward list to it directly. (_, true) => { let (rp, p) = self.inner.blocking_list(path, args)?; Ok((rp, CompleteLister::One(p))) } // If recursive is true but service can't list_with_recursive (true, false) => { // Forward path that ends with / if path.ends_with('/') { let p = FlatLister::new(self.inner.clone(), path); Ok((RpList::default(), CompleteLister::Two(p))) } else { let parent = get_parent(path); let p = FlatLister::new(self.inner.clone(), parent); let p = PrefixLister::new(p, path); Ok((RpList::default(), CompleteLister::Four(p))) } } // If recursive and service doesn't support list_with_recursive, we need to handle // list prefix by ourselves. (false, false) => { // Forward path that ends with / if path.ends_with('/') { let (rp, p) = self.inner.blocking_list(path, args)?; Ok((rp, CompleteLister::One(p))) } else { let parent = get_parent(path); let (rp, p) = self.inner.blocking_list(parent, args)?; let p = PrefixLister::new(p, path); Ok((rp, CompleteLister::Three(p))) } } } } } impl LayeredAccess for CompleteAccessor { type Inner = A; type Reader = CompleteReader; type BlockingReader = CompleteReader; type Writer = CompleteWriter; type BlockingWriter = CompleteWriter; type Lister = CompleteLister; type BlockingLister = CompleteLister; type Deleter = A::Deleter; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } // Todo: May move the logic to the implement of Layer::layer of CompleteAccessor fn info(&self) -> Arc { let mut meta = (*self.info).clone(); let cap = meta.full_capability_mut(); if cap.list && cap.write_can_empty { cap.create_dir = true; } // write operations should always return content length cap.write_has_content_length = true; meta.into() } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.complete_create_dir(path, args).await } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let size = args.range().size(); self.inner .read(path, args) .await .map(|(rp, r)| (rp, CompleteReader::new(r, size))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let (rp, w) = self.inner.write(path, args.clone()).await?; let w = CompleteWriter::new(w, args.append()); Ok((rp, w)) } async fn stat(&self, path: &str, args: OpStat) -> Result { self.complete_stat(path, args).await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner().delete().await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.complete_list(path, args).await } async fn presign(&self, path: &str, args: OpPresign) -> Result { self.inner.presign(path, args).await } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.complete_blocking_create_dir(path, args) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let size = args.range().size(); self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, CompleteReader::new(r, size))) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let append = args.append(); self.inner .blocking_write(path, args) .map(|(rp, w)| (rp, CompleteWriter::new(w, append))) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.complete_blocking_stat(path, args) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner().blocking_delete() } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.complete_blocking_list(path, args) } } pub type CompleteLister = FourWays, P>, PrefixLister

, PrefixLister, P>>>; pub struct CompleteReader { inner: R, size: Option, read: u64, } impl CompleteReader { pub fn new(inner: R, size: Option) -> Self { Self { inner, size, read: 0, } } pub fn check(&self) -> Result<()> { let Some(size) = self.size else { return Ok(()); }; match self.read.cmp(&size) { Ordering::Equal => Ok(()), Ordering::Less => Err( Error::new(ErrorKind::Unexpected, "reader got too little data") .with_context("expect", size) .with_context("actual", self.read), ), Ordering::Greater => Err( Error::new(ErrorKind::Unexpected, "reader got too much data") .with_context("expect", size) .with_context("actual", self.read), ), } } } impl oio::Read for CompleteReader { async fn read(&mut self) -> Result { let buf = self.inner.read().await?; if buf.is_empty() { self.check()?; } else { self.read += buf.len() as u64; } Ok(buf) } } impl oio::BlockingRead for CompleteReader { fn read(&mut self) -> Result { let buf = self.inner.read()?; if buf.is_empty() { self.check()?; } else { self.read += buf.len() as u64; } Ok(buf) } } pub struct CompleteWriter { inner: Option, append: bool, size: u64, } impl CompleteWriter { pub fn new(inner: W, append: bool) -> CompleteWriter { CompleteWriter { inner: Some(inner), append, size: 0, } } fn check(&self, content_length: u64) -> Result<()> { if self.append || content_length == 0 { return Ok(()); } match self.size.cmp(&content_length) { Ordering::Equal => Ok(()), Ordering::Less => Err( Error::new(ErrorKind::Unexpected, "writer got too little data") .with_context("expect", content_length) .with_context("actual", self.size), ), Ordering::Greater => Err( Error::new(ErrorKind::Unexpected, "writer got too much data") .with_context("expect", content_length) .with_context("actual", self.size), ), } } } /// Check if the writer has been closed or aborted while debug_assertions /// enabled. This code will never be executed in release mode. #[cfg(debug_assertions)] impl Drop for CompleteWriter { fn drop(&mut self) { if self.inner.is_some() { log::warn!("writer has not been closed or aborted, must be a bug") } } } impl oio::Write for CompleteWriter where W: oio::Write, { async fn write(&mut self, bs: Buffer) -> Result<()> { let w = self.inner.as_mut().ok_or_else(|| { Error::new(ErrorKind::Unexpected, "writer has been closed or aborted") })?; let len = bs.len(); w.write(bs).await?; self.size += len as u64; Ok(()) } async fn close(&mut self) -> Result { let w = self.inner.as_mut().ok_or_else(|| { Error::new(ErrorKind::Unexpected, "writer has been closed or aborted") })?; // we must return `Err` before setting inner to None; otherwise, // we won't be able to retry `close` in `RetryLayer`. let mut ret = w.close().await?; self.check(ret.content_length())?; if ret.content_length() == 0 { ret = ret.with_content_length(self.size); } self.inner = None; Ok(ret) } async fn abort(&mut self) -> Result<()> { let w = self.inner.as_mut().ok_or_else(|| { Error::new(ErrorKind::Unexpected, "writer has been closed or aborted") })?; w.abort().await?; self.inner = None; Ok(()) } } impl oio::BlockingWrite for CompleteWriter where W: oio::BlockingWrite, { fn write(&mut self, bs: Buffer) -> Result<()> { let w = self.inner.as_mut().ok_or_else(|| { Error::new(ErrorKind::Unexpected, "writer has been closed or aborted") })?; let len = bs.len(); w.write(bs)?; self.size += len as u64; Ok(()) } fn close(&mut self) -> Result { let w = self.inner.as_mut().ok_or_else(|| { Error::new(ErrorKind::Unexpected, "writer has been closed or aborted") })?; let mut ret = w.close()?; self.check(ret.content_length())?; if ret.content_length() == 0 { ret = ret.with_content_length(self.size); } self.inner = None; Ok(ret) } } opendal-0.52.0/src/layers/concurrent_limit.rs000064400000000000000000000216121046102023000173500ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::sync::Arc; use tokio::sync::OwnedSemaphorePermit; use tokio::sync::Semaphore; use crate::raw::*; use crate::*; /// Add concurrent request limit. /// /// # Notes /// /// Users can control how many concurrent connections could be established /// between OpenDAL and underlying storage services. /// /// # Examples /// /// ```no_run /// # use opendal::layers::ConcurrentLimitLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(ConcurrentLimitLayer::new(1024)) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Clone)] pub struct ConcurrentLimitLayer { permits: usize, } impl ConcurrentLimitLayer { /// Create a new ConcurrentLimitLayer will specify permits pub fn new(permits: usize) -> Self { Self { permits } } } impl Layer for ConcurrentLimitLayer { type LayeredAccess = ConcurrentLimitAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { ConcurrentLimitAccessor { inner, semaphore: Arc::new(Semaphore::new(self.permits)), } } } #[derive(Debug, Clone)] pub struct ConcurrentLimitAccessor { inner: A, semaphore: Arc, } impl LayeredAccess for ConcurrentLimitAccessor { type Inner = A; type Reader = ConcurrentLimitWrapper; type BlockingReader = ConcurrentLimitWrapper; type Writer = ConcurrentLimitWrapper; type BlockingWriter = ConcurrentLimitWrapper; type Lister = ConcurrentLimitWrapper; type BlockingLister = ConcurrentLimitWrapper; type Deleter = ConcurrentLimitWrapper; type BlockingDeleter = ConcurrentLimitWrapper; fn inner(&self) -> &Self::Inner { &self.inner } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { let _permit = self .semaphore .acquire() .await .expect("semaphore must be valid"); self.inner.create_dir(path, args).await } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let permit = self .semaphore .clone() .acquire_owned() .await .expect("semaphore must be valid"); self.inner .read(path, args) .await .map(|(rp, r)| (rp, ConcurrentLimitWrapper::new(r, permit))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let permit = self .semaphore .clone() .acquire_owned() .await .expect("semaphore must be valid"); self.inner .write(path, args) .await .map(|(rp, w)| (rp, ConcurrentLimitWrapper::new(w, permit))) } async fn stat(&self, path: &str, args: OpStat) -> Result { let _permit = self .semaphore .acquire() .await .expect("semaphore must be valid"); self.inner.stat(path, args).await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { let permit = self .semaphore .clone() .acquire_owned() .await .expect("semaphore must be valid"); self.inner .delete() .await .map(|(rp, w)| (rp, ConcurrentLimitWrapper::new(w, permit))) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let permit = self .semaphore .clone() .acquire_owned() .await .expect("semaphore must be valid"); self.inner .list(path, args) .await .map(|(rp, s)| (rp, ConcurrentLimitWrapper::new(s, permit))) } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { let _permit = self .semaphore .try_acquire() .expect("semaphore must be valid"); self.inner.blocking_create_dir(path, args) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let permit = self .semaphore .clone() .try_acquire_owned() .expect("semaphore must be valid"); self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, ConcurrentLimitWrapper::new(r, permit))) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let permit = self .semaphore .clone() .try_acquire_owned() .expect("semaphore must be valid"); self.inner .blocking_write(path, args) .map(|(rp, w)| (rp, ConcurrentLimitWrapper::new(w, permit))) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { let _permit = self .semaphore .try_acquire() .expect("semaphore must be valid"); self.inner.blocking_stat(path, args) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { let permit = self .semaphore .clone() .try_acquire_owned() .expect("semaphore must be valid"); self.inner .blocking_delete() .map(|(rp, w)| (rp, ConcurrentLimitWrapper::new(w, permit))) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { let permit = self .semaphore .clone() .try_acquire_owned() .expect("semaphore must be valid"); self.inner .blocking_list(path, args) .map(|(rp, it)| (rp, ConcurrentLimitWrapper::new(it, permit))) } } pub struct ConcurrentLimitWrapper { inner: R, // Hold on this permit until this reader has been dropped. _permit: OwnedSemaphorePermit, } impl ConcurrentLimitWrapper { fn new(inner: R, permit: OwnedSemaphorePermit) -> Self { Self { inner, _permit: permit, } } } impl oio::Read for ConcurrentLimitWrapper { async fn read(&mut self) -> Result { self.inner.read().await } } impl oio::BlockingRead for ConcurrentLimitWrapper { fn read(&mut self) -> Result { self.inner.read() } } impl oio::Write for ConcurrentLimitWrapper { async fn write(&mut self, bs: Buffer) -> Result<()> { self.inner.write(bs).await } async fn close(&mut self) -> Result { self.inner.close().await } async fn abort(&mut self) -> Result<()> { self.inner.abort().await } } impl oio::BlockingWrite for ConcurrentLimitWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { self.inner.write(bs) } fn close(&mut self) -> Result { self.inner.close() } } impl oio::List for ConcurrentLimitWrapper { async fn next(&mut self) -> Result> { self.inner.next().await } } impl oio::BlockingList for ConcurrentLimitWrapper { fn next(&mut self) -> Result> { self.inner.next() } } impl oio::Delete for ConcurrentLimitWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } async fn flush(&mut self) -> Result { self.inner.flush().await } } impl oio::BlockingDelete for ConcurrentLimitWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } fn flush(&mut self) -> Result { self.inner.flush() } } opendal-0.52.0/src/layers/correctness_check.rs000064400000000000000000000414671046102023000174710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{Debug, Formatter}; use std::future::Future; use std::sync::Arc; use crate::raw::*; use crate::*; /// Add a correctness capability check layer for every operation /// /// Before performing any operations, we will first verify the operation and its critical arguments /// against the capability of the underlying service. If the operation or arguments is not supported, /// an error will be returned directly. /// /// # Notes /// /// OpenDAL applies this checker to every accessor by default, so users don't need to invoke it manually. /// this checker ensures the operation and its critical arguments, which might affect the correctness of /// the call, are supported by the underlying service. /// /// for example, when calling `write_with_append`, but `append` is not supported by the underlying /// service, an `Unsupported` error is returned. without this check, undesired data may be written. #[derive(Default)] pub struct CorrectnessCheckLayer; impl Layer for CorrectnessCheckLayer { type LayeredAccess = CorrectnessAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { CorrectnessAccessor { info: inner.info(), inner, } } } pub(crate) fn new_unsupported_error(info: &AccessorInfo, op: Operation, args: &str) -> Error { let scheme = info.scheme(); let op = op.into_static(); Error::new( ErrorKind::Unsupported, format!("The service {scheme} does not support the operation {op} with the arguments {args}. Please verify if the relevant flags have been enabled, or submit an issue if you believe this is incorrect."), ) .with_operation(op) } pub struct CorrectnessAccessor { info: Arc, inner: A, } impl Debug for CorrectnessAccessor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("CorrectnessCheckAccessor") .field("inner", &self.inner) .finish_non_exhaustive() } } impl LayeredAccess for CorrectnessAccessor { type Inner = A; type Reader = A::Reader; type Writer = A::Writer; type Lister = A::Lister; type Deleter = CheckWrapper; type BlockingReader = A::BlockingReader; type BlockingWriter = A::BlockingWriter; type BlockingLister = A::BlockingLister; type BlockingDeleter = CheckWrapper; fn inner(&self) -> &Self::Inner { &self.inner } fn info(&self) -> Arc { self.info.clone() } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let capability = self.info.full_capability(); if !capability.read_with_version && args.version().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Read, "version", )); } if !capability.read_with_if_match && args.if_match().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Read, "if_match", )); } if !capability.read_with_if_none_match && args.if_none_match().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Read, "if_none_match", )); } if !capability.read_with_if_modified_since && args.if_modified_since().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Read, "if_modified_since", )); } if !capability.read_with_if_unmodified_since && args.if_unmodified_since().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Read, "if_unmodified_since", )); } self.inner.read(path, args).await } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let capability = self.info.full_capability(); if args.append() && !capability.write_can_append { return Err(new_unsupported_error( &self.info, Operation::Write, "append", )); } if args.if_not_exists() && !capability.write_with_if_not_exists { return Err(new_unsupported_error( &self.info, Operation::Write, "if_not_exists", )); } if let Some(if_none_match) = args.if_none_match() { if !capability.write_with_if_none_match { let mut err = new_unsupported_error(self.info.as_ref(), Operation::Write, "if_none_match"); if if_none_match == "*" && capability.write_with_if_not_exists { err = err.with_context("hint", "use if_not_exists instead"); } return Err(err); } } self.inner.write(path, args).await } async fn stat(&self, path: &str, args: OpStat) -> Result { let capability = self.info.full_capability(); if !capability.stat_with_version && args.version().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Stat, "version", )); } if !capability.stat_with_if_match && args.if_match().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Stat, "if_match", )); } if !capability.stat_with_if_none_match && args.if_none_match().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Stat, "if_none_match", )); } if !capability.stat_with_if_modified_since && args.if_modified_since().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Stat, "if_modified_since", )); } if !capability.stat_with_if_unmodified_since && args.if_unmodified_since().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::Stat, "if_unmodified_since", )); } self.inner.stat(path, args).await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner.delete().await.map(|(rp, deleter)| { let deleter = CheckWrapper::new(deleter, self.info.clone()); (rp, deleter) }) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner.list(path, args).await } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let capability = self.info.full_capability(); if !capability.read_with_version && args.version().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::BlockingRead, "version", )); } self.inner.blocking_read(path, args) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let capability = self.info.full_capability(); if args.append() && !capability.write_can_append { return Err(new_unsupported_error( &self.info, Operation::BlockingWrite, "append", )); } if args.if_not_exists() && !capability.write_with_if_not_exists { return Err(new_unsupported_error( &self.info, Operation::BlockingWrite, "if_not_exists", )); } if args.if_none_match().is_some() && !capability.write_with_if_none_match { return Err(new_unsupported_error( self.info.as_ref(), Operation::BlockingWrite, "if_none_match", )); } self.inner.blocking_write(path, args) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { let capability = self.info.full_capability(); if !capability.stat_with_version && args.version().is_some() { return Err(new_unsupported_error( self.info.as_ref(), Operation::BlockingStat, "version", )); } self.inner.blocking_stat(path, args) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner.blocking_delete().map(|(rp, deleter)| { let deleter = CheckWrapper::new(deleter, self.info.clone()); (rp, deleter) }) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner.blocking_list(path, args) } } pub struct CheckWrapper { info: Arc, inner: T, } impl CheckWrapper { fn new(inner: T, info: Arc) -> Self { Self { inner, info } } fn check_delete(&self, args: &OpDelete) -> Result<()> { if args.version().is_some() && !self.info.full_capability().delete_with_version { return Err(new_unsupported_error( &self.info, Operation::DeleterDelete, "version", )); } Ok(()) } } impl oio::Delete for CheckWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.check_delete(&args)?; self.inner.delete(path, args) } fn flush(&mut self) -> impl Future> + MaybeSend { self.inner.flush() } } impl oio::BlockingDelete for CheckWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.check_delete(&args)?; self.inner.delete(path, args) } fn flush(&mut self) -> Result { self.inner.flush() } } #[cfg(test)] mod tests { use super::*; use crate::raw::oio; use crate::{Capability, EntryMode, Metadata, Operator}; #[derive(Debug)] struct MockService { capability: Capability, } impl Access for MockService { type Reader = oio::Reader; type Writer = oio::Writer; type Lister = oio::Lister; type Deleter = oio::Deleter; type BlockingReader = oio::BlockingReader; type BlockingWriter = oio::BlockingWriter; type BlockingLister = oio::BlockingLister; type BlockingDeleter = oio::BlockingDeleter; fn info(&self) -> Arc { let mut info = AccessorInfo::default(); info.set_native_capability(self.capability); info.into() } async fn stat(&self, _: &str, _: OpStat) -> Result { Ok(RpStat::new(Metadata::new(EntryMode::Unknown))) } async fn read(&self, _: &str, _: OpRead) -> Result<(RpRead, Self::Reader)> { Ok((RpRead::new(), Box::new(bytes::Bytes::new()))) } async fn write(&self, _: &str, _: OpWrite) -> Result<(RpWrite, Self::Writer)> { Ok((RpWrite::new(), Box::new(MockWriter))) } async fn list(&self, _: &str, _: OpList) -> Result<(RpList, Self::Lister)> { Ok((RpList::default(), Box::new(()))) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok((RpDelete::default(), Box::new(MockDeleter))) } } struct MockWriter; impl oio::Write for MockWriter { async fn write(&mut self, _: Buffer) -> Result<()> { Ok(()) } async fn close(&mut self) -> Result { Ok(Metadata::default()) } async fn abort(&mut self) -> Result<()> { Ok(()) } } struct MockDeleter; impl oio::Delete for MockDeleter { fn delete(&mut self, _: &str, _: OpDelete) -> Result<()> { Ok(()) } async fn flush(&mut self) -> Result { Ok(1) } } fn new_test_operator(capability: Capability) -> Operator { let srv = MockService { capability }; Operator::from_inner(Arc::new(srv)).layer(CorrectnessCheckLayer) } #[tokio::test] async fn test_read() { let op = new_test_operator(Capability { read: true, ..Default::default() }); let res = op.read_with("path").version("version").await; assert!(res.is_err()); assert_eq!(res.unwrap_err().kind(), ErrorKind::Unsupported); let op = new_test_operator(Capability { read: true, read_with_version: true, ..Default::default() }); let res = op.read_with("path").version("version").await; assert!(res.is_ok()); } #[tokio::test] async fn test_stat() { let op = new_test_operator(Capability { stat: true, ..Default::default() }); let res = op.stat_with("path").version("version").await; assert!(res.is_err()); assert_eq!(res.unwrap_err().kind(), ErrorKind::Unsupported); let op = new_test_operator(Capability { stat: true, stat_with_version: true, ..Default::default() }); let res = op.stat_with("path").version("version").await; assert!(res.is_ok()); } #[tokio::test] async fn test_write_with() { let op = new_test_operator(Capability { write: true, write_with_if_not_exists: true, ..Default::default() }); let res = op.write_with("path", "".as_bytes()).append(true).await; assert!(res.is_err()); assert_eq!(res.unwrap_err().kind(), ErrorKind::Unsupported); let res = op .write_with("path", "".as_bytes()) .if_none_match("etag") .await; assert!(res.is_err()); assert_eq!( res.unwrap_err().to_string(), "Unsupported (permanent) at write => The service memory does not support the operation write with the arguments if_none_match. Please verify if the relevant flags have been enabled, or submit an issue if you believe this is incorrect." ); // Now try a wildcard if-none-match let res = op .write_with("path", "".as_bytes()) .if_none_match("*") .await; assert!(res.is_err()); assert_eq!( res.unwrap_err().to_string(), "Unsupported (permanent) at write, context: { hint: use if_not_exists instead } => The service memory does not support the operation write with the arguments if_none_match. Please verify if the relevant flags have been enabled, or submit an issue if you believe this is incorrect." ); let res = op .write_with("path", "".as_bytes()) .if_not_exists(true) .await; assert!(res.is_ok()); let op = new_test_operator(Capability { write: true, write_can_append: true, write_with_if_not_exists: true, write_with_if_none_match: true, ..Default::default() }); let res = op.writer_with("path").append(true).await; assert!(res.is_ok()); } #[tokio::test] async fn test_delete() { let op = new_test_operator(Capability { delete: true, ..Default::default() }); let res = op.delete_with("path").version("version").await; assert!(res.is_err()); assert_eq!(res.unwrap_err().kind(), ErrorKind::Unsupported); let op = new_test_operator(Capability { delete: true, delete_with_version: true, ..Default::default() }); let res = op.delete_with("path").version("version").await; assert!(res.is_ok()) } } opendal-0.52.0/src/layers/dtrace.rs000064400000000000000000000346401046102023000152370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ffi::CString; use std::fmt::Debug; use std::fmt::Formatter; use bytes::Buf; use probe::probe_lazy; use crate::raw::Access; use crate::raw::*; use crate::*; /// Support User Statically-Defined Tracing(aka USDT) on Linux /// /// This layer is an experimental feature, it will be enabled by `features = ["layers-dtrace"]` in Cargo.toml. /// /// For now we have following probes: /// /// ### For Accessor /// /// 1. ${operation}_start, arguments: path /// 1. create_dir /// 2. read /// 3. write /// 4. stat /// 5. delete /// 6. list /// 7. presign /// 8. blocking_create_dir /// 9. blocking_read /// 10. blocking_write /// 11. blocking_stat /// 12. blocking_delete /// 13. blocking_list /// 2. ${operation}_end, arguments: path /// 1. create_dir /// 2. read /// 3. write /// 4. stat /// 5. delete /// 6. list /// 7. presign /// 8. blocking_create_dir /// 9. blocking_read /// 10. blocking_write /// 11. blocking_stat /// 12. blocking_delete /// 13. blocking_list /// /// ### For Reader /// /// 1. reader_read_start, arguments: path /// 2. reader_read_ok, arguments: path, length /// 3. reader_read_error, arguments: path /// /// ### For BlockingReader /// /// 1. blocking_reader_read_start, arguments: path /// 2. blocking_reader_read_ok, arguments: path, length /// 3. blocking_reader_read_error, arguments: path /// /// ### For Writer /// /// 1. writer_write_start, arguments: path /// 2. writer_write_ok, arguments: path, length /// 3. writer_write_error, arguments: path /// 4. writer_abort_start, arguments: path /// 5. writer_abort_ok, arguments: path /// 6. writer_abort_error, arguments: path /// 7. writer_close_start, arguments: path /// 8. writer_close_ok, arguments: path /// 9. writer_close_error, arguments: path /// /// ### For BlockingWriter /// /// 1. blocking_writer_write_start, arguments: path /// 2. blocking_writer_write_ok, arguments: path, length /// 3. blocking_writer_write_error, arguments: path /// 4. blocking_writer_close_start, arguments: path /// 5. blocking_writer_close_ok, arguments: path /// 6. blocking_writer_close_error, arguments: path /// /// Example: /// /// ```no_run /// # use opendal::layers::DtraceLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // `Accessor` provides the low level APIs, we will use `Operator` normally. /// let op: Operator = Operator::new(services::Fs::default().root("/tmp"))? /// .layer(DtraceLayer::default()) /// .finish(); /// /// let path = "/tmp/test.txt"; /// for _ in 1..100000 { /// let bs = vec![0; 64 * 1024 * 1024]; /// op.write(path, bs).await?; /// op.read(path).await?; /// } /// Ok(()) /// # } /// ``` /// /// Then you can use `readelf -n target/debug/examples/dtrace` to see the probes: /// /// ```text /// Displaying notes found in: .note.stapsdt /// Owner Data size Description /// stapsdt 0x00000039 NT_STAPSDT (SystemTap probe descriptors) /// Provider: opendal /// Name: create_dir_start /// Location: 0x00000000000f8f05, Base: 0x0000000000000000, Semaphore: 0x00000000003649f8 /// Arguments: -8@%rax /// stapsdt 0x00000037 NT_STAPSDT (SystemTap probe descriptors) /// Provider: opendal /// Name: create_dir_end /// Location: 0x00000000000f9284, Base: 0x0000000000000000, Semaphore: 0x00000000003649fa /// Arguments: -8@%rax /// stapsdt 0x0000003c NT_STAPSDT (SystemTap probe descriptors) /// Provider: opendal /// Name: blocking_list_start /// Location: 0x00000000000f9487, Base: 0x0000000000000000, Semaphore: 0x0000000000364a28 /// Arguments: -8@%rax /// stapsdt 0x0000003a NT_STAPSDT (SystemTap probe descriptors) /// Provider: opendal /// Name: blocking_list_end /// Location: 0x00000000000f9546, Base: 0x0000000000000000, Semaphore: 0x0000000000364a2a /// Arguments: -8@%rax /// stapsdt 0x0000003c NT_STAPSDT (SystemTap probe descriptors) /// ``` #[derive(Default, Debug, Clone)] pub struct DtraceLayer {} impl Layer for DtraceLayer { type LayeredAccess = DTraceAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { DTraceAccessor { inner } } } #[derive(Clone)] pub struct DTraceAccessor { inner: A, } impl Debug for DTraceAccessor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("DTraceAccessor") .field("inner", &self.inner) .finish_non_exhaustive() } } impl LayeredAccess for DTraceAccessor { type Inner = A; type Reader = DtraceLayerWrapper; type BlockingReader = DtraceLayerWrapper; type Writer = DtraceLayerWrapper; type BlockingWriter = DtraceLayerWrapper; type Lister = A::Lister; type BlockingLister = A::BlockingLister; type Deleter = A::Deleter; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, create_dir_start, c_path.as_ptr()); let result = self.inner.create_dir(path, args).await; probe_lazy!(opendal, create_dir_end, c_path.as_ptr()); result } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, read_start, c_path.as_ptr()); let result = self .inner .read(path, args) .await .map(|(rp, r)| (rp, DtraceLayerWrapper::new(r, &path.to_string()))); probe_lazy!(opendal, read_end, c_path.as_ptr()); result } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, write_start, c_path.as_ptr()); let result = self .inner .write(path, args) .await .map(|(rp, r)| (rp, DtraceLayerWrapper::new(r, &path.to_string()))); probe_lazy!(opendal, write_end, c_path.as_ptr()); result } async fn stat(&self, path: &str, args: OpStat) -> Result { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, stat_start, c_path.as_ptr()); let result = self.inner.stat(path, args).await; probe_lazy!(opendal, stat_end, c_path.as_ptr()); result } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner.delete().await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, list_start, c_path.as_ptr()); let result = self.inner.list(path, args).await; probe_lazy!(opendal, list_end, c_path.as_ptr()); result } async fn presign(&self, path: &str, args: OpPresign) -> Result { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, presign_start, c_path.as_ptr()); let result = self.inner.presign(path, args).await; probe_lazy!(opendal, presign_end, c_path.as_ptr()); result } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, blocking_create_dir_start, c_path.as_ptr()); let result = self.inner.blocking_create_dir(path, args); probe_lazy!(opendal, blocking_create_dir_end, c_path.as_ptr()); result } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, blocking_read_start, c_path.as_ptr()); let result = self .inner .blocking_read(path, args) .map(|(rp, r)| (rp, DtraceLayerWrapper::new(r, &path.to_string()))); probe_lazy!(opendal, blocking_read_end, c_path.as_ptr()); result } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, blocking_write_start, c_path.as_ptr()); let result = self .inner .blocking_write(path, args) .map(|(rp, r)| (rp, DtraceLayerWrapper::new(r, &path.to_string()))); probe_lazy!(opendal, blocking_write_end, c_path.as_ptr()); result } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, blocking_stat_start, c_path.as_ptr()); let result = self.inner.blocking_stat(path, args); probe_lazy!(opendal, blocking_stat_end, c_path.as_ptr()); result } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner.blocking_delete() } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { let c_path = CString::new(path).unwrap(); probe_lazy!(opendal, blocking_list_start, c_path.as_ptr()); let result = self.inner.blocking_list(path, args); probe_lazy!(opendal, blocking_list_end, c_path.as_ptr()); result } } pub struct DtraceLayerWrapper { inner: R, path: String, } impl DtraceLayerWrapper { pub fn new(inner: R, path: &String) -> Self { Self { inner, path: path.to_string(), } } } impl oio::Read for DtraceLayerWrapper { async fn read(&mut self) -> Result { let c_path = CString::new(self.path.clone()).unwrap(); probe_lazy!(opendal, reader_read_start, c_path.as_ptr()); match self.inner.read().await { Ok(bs) => { probe_lazy!(opendal, reader_read_ok, c_path.as_ptr(), bs.remaining()); Ok(bs) } Err(e) => { probe_lazy!(opendal, reader_read_error, c_path.as_ptr()); Err(e) } } } } impl oio::BlockingRead for DtraceLayerWrapper { fn read(&mut self) -> Result { let c_path = CString::new(self.path.clone()).unwrap(); probe_lazy!(opendal, blocking_reader_read_start, c_path.as_ptr()); self.inner .read() .map(|bs| { probe_lazy!( opendal, blocking_reader_read_ok, c_path.as_ptr(), bs.remaining() ); bs }) .map_err(|e| { probe_lazy!(opendal, blocking_reader_read_error, c_path.as_ptr()); e }) } } impl oio::Write for DtraceLayerWrapper { async fn write(&mut self, bs: Buffer) -> Result<()> { let c_path = CString::new(self.path.clone()).unwrap(); probe_lazy!(opendal, writer_write_start, c_path.as_ptr()); self.inner .write(bs) .await .map(|_| { probe_lazy!(opendal, writer_write_ok, c_path.as_ptr()); }) .map_err(|err| { probe_lazy!(opendal, writer_write_error, c_path.as_ptr()); err }) } async fn abort(&mut self) -> Result<()> { let c_path = CString::new(self.path.clone()).unwrap(); probe_lazy!(opendal, writer_poll_abort_start, c_path.as_ptr()); self.inner .abort() .await .map(|_| { probe_lazy!(opendal, writer_poll_abort_ok, c_path.as_ptr()); }) .map_err(|err| { probe_lazy!(opendal, writer_poll_abort_error, c_path.as_ptr()); err }) } async fn close(&mut self) -> Result { let c_path = CString::new(self.path.clone()).unwrap(); probe_lazy!(opendal, writer_close_start, c_path.as_ptr()); self.inner .close() .await .map(|meta| { probe_lazy!(opendal, writer_close_ok, c_path.as_ptr()); meta }) .map_err(|err| { probe_lazy!(opendal, writer_close_error, c_path.as_ptr()); err }) } } impl oio::BlockingWrite for DtraceLayerWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { let c_path = CString::new(self.path.clone()).unwrap(); probe_lazy!(opendal, blocking_writer_write_start, c_path.as_ptr()); self.inner .write(bs) .map(|_| { probe_lazy!(opendal, blocking_writer_write_ok, c_path.as_ptr()); }) .map_err(|err| { probe_lazy!(opendal, blocking_writer_write_error, c_path.as_ptr()); err }) } fn close(&mut self) -> Result { let c_path = CString::new(self.path.clone()).unwrap(); probe_lazy!(opendal, blocking_writer_close_start, c_path.as_ptr()); self.inner .close() .map(|meta| { probe_lazy!(opendal, blocking_writer_close_ok, c_path.as_ptr()); meta }) .map_err(|err| { probe_lazy!(opendal, blocking_writer_close_error, c_path.as_ptr()); err }) } } opendal-0.52.0/src/layers/error_context.rs000064400000000000000000000423061046102023000166700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use crate::raw::*; use crate::*; /// ErrorContextLayer will add error context into all layers. /// /// # Notes /// /// This layer will add the following error context into all errors: /// /// - `service`: The [`Scheme`] of underlying service. /// - `operation`: The [`Operation`] of this operation /// - `path`: The path of this operation /// /// Some operations may have additional context: /// /// - `range`: The range the read operation is trying to read. /// - `read`: The already read size in given reader. /// - `size`: The size of the current write operation. /// - `written`: The already written size in given writer. /// - `listed`: The already listed size in given lister. /// - `deleted`: The already deleted size in given deleter. pub struct ErrorContextLayer; impl Layer for ErrorContextLayer { type LayeredAccess = ErrorContextAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { let info = inner.info(); ErrorContextAccessor { info, inner } } } /// Provide error context wrapper for backend. pub struct ErrorContextAccessor { info: Arc, inner: A, } impl Debug for ErrorContextAccessor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { self.inner.fmt(f) } } impl LayeredAccess for ErrorContextAccessor { type Inner = A; type Reader = ErrorContextWrapper; type BlockingReader = ErrorContextWrapper; type Writer = ErrorContextWrapper; type BlockingWriter = ErrorContextWrapper; type Lister = ErrorContextWrapper; type BlockingLister = ErrorContextWrapper; type Deleter = ErrorContextWrapper; type BlockingDeleter = ErrorContextWrapper; fn inner(&self) -> &Self::Inner { &self.inner } fn info(&self) -> Arc { self.info.clone() } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.inner.create_dir(path, args).await.map_err(|err| { err.with_operation(Operation::CreateDir) .with_context("service", self.info.scheme()) .with_context("path", path) }) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let range = args.range(); self.inner .read(path, args) .await .map(|(rp, r)| { ( rp, ErrorContextWrapper::new(self.info.scheme(), path.to_string(), r) .with_range(range), ) }) .map_err(|err| { err.with_operation(Operation::Read) .with_context("service", self.info.scheme()) .with_context("path", path) .with_context("range", range.to_string()) }) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner .write(path, args) .await .map(|(rp, w)| { ( rp, ErrorContextWrapper::new(self.info.scheme(), path.to_string(), w), ) }) .map_err(|err| { err.with_operation(Operation::Write) .with_context("service", self.info.scheme()) .with_context("path", path) }) } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner.copy(from, to, args).await.map_err(|err| { err.with_operation(Operation::Copy) .with_context("service", self.info.scheme()) .with_context("from", from) .with_context("to", to) }) } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner.rename(from, to, args).await.map_err(|err| { err.with_operation(Operation::Rename) .with_context("service", self.info.scheme()) .with_context("from", from) .with_context("to", to) }) } async fn stat(&self, path: &str, args: OpStat) -> Result { self.inner.stat(path, args).await.map_err(|err| { err.with_operation(Operation::Stat) .with_context("service", self.info.scheme()) .with_context("path", path) }) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner .delete() .await .map(|(rp, w)| { ( rp, ErrorContextWrapper::new(self.info.scheme(), "".to_string(), w), ) }) .map_err(|err| { err.with_operation(Operation::Delete) .with_context("service", self.info.scheme()) }) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner .list(path, args) .await .map(|(rp, p)| { ( rp, ErrorContextWrapper::new(self.info.scheme(), path.to_string(), p), ) }) .map_err(|err| { err.with_operation(Operation::List) .with_context("service", self.info.scheme()) .with_context("path", path) }) } async fn presign(&self, path: &str, args: OpPresign) -> Result { self.inner.presign(path, args).await.map_err(|err| { err.with_operation(Operation::Presign) .with_context("service", self.info.scheme()) .with_context("path", path) }) } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.inner.blocking_create_dir(path, args).map_err(|err| { err.with_operation(Operation::BlockingCreateDir) .with_context("service", self.info.scheme()) .with_context("path", path) }) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let range = args.range(); self.inner .blocking_read(path, args) .map(|(rp, os)| { ( rp, ErrorContextWrapper::new(self.info.scheme(), path.to_string(), os) .with_range(range), ) }) .map_err(|err| { err.with_operation(Operation::BlockingRead) .with_context("service", self.info.scheme()) .with_context("path", path) .with_context("range", range.to_string()) }) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner .blocking_write(path, args) .map(|(rp, os)| { ( rp, ErrorContextWrapper::new(self.info.scheme(), path.to_string(), os), ) }) .map_err(|err| { err.with_operation(Operation::BlockingWrite) .with_context("service", self.info.scheme()) .with_context("path", path) }) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner.blocking_copy(from, to, args).map_err(|err| { err.with_operation(Operation::BlockingCopy) .with_context("service", self.info.scheme()) .with_context("from", from) .with_context("to", to) }) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner.blocking_rename(from, to, args).map_err(|err| { err.with_operation(Operation::BlockingRename) .with_context("service", self.info.scheme()) .with_context("from", from) .with_context("to", to) }) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.inner.blocking_stat(path, args).map_err(|err| { err.with_operation(Operation::BlockingStat) .with_context("service", self.info.scheme()) .with_context("path", path) }) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner .blocking_delete() .map(|(rp, w)| { ( rp, ErrorContextWrapper::new(self.info.scheme(), "".to_string(), w), ) }) .map_err(|err| { err.with_operation(Operation::BlockingDelete) .with_context("service", self.info.scheme()) }) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner .blocking_list(path, args) .map(|(rp, os)| { ( rp, ErrorContextWrapper::new(self.info.scheme(), path.to_string(), os), ) }) .map_err(|err| { err.with_operation(Operation::BlockingList) .with_context("service", self.info.scheme()) .with_context("path", path) }) } } pub struct ErrorContextWrapper { scheme: Scheme, path: String, inner: T, range: BytesRange, processed: u64, } impl ErrorContextWrapper { fn new(scheme: Scheme, path: String, inner: T) -> Self { Self { scheme, path, inner, range: BytesRange::default(), processed: 0, } } fn with_range(mut self, range: BytesRange) -> Self { self.range = range; self } } impl oio::Read for ErrorContextWrapper { async fn read(&mut self) -> Result { self.inner .read() .await .map(|bs| { self.processed += bs.len() as u64; bs }) .map_err(|err| { err.with_operation(Operation::ReaderRead) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("range", self.range.to_string()) .with_context("read", self.processed.to_string()) }) } } impl oio::BlockingRead for ErrorContextWrapper { fn read(&mut self) -> Result { self.inner .read() .map(|bs| { self.processed += bs.len() as u64; bs }) .map_err(|err| { err.with_operation(Operation::BlockingReaderRead) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("range", self.range.to_string()) .with_context("read", self.processed.to_string()) }) } } impl oio::Write for ErrorContextWrapper { async fn write(&mut self, bs: Buffer) -> Result<()> { let size = bs.len(); self.inner .write(bs) .await .map(|_| { self.processed += size as u64; }) .map_err(|err| { err.with_operation(Operation::WriterWrite) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("size", size.to_string()) .with_context("written", self.processed.to_string()) }) } async fn close(&mut self) -> Result { self.inner.close().await.map_err(|err| { err.with_operation(Operation::WriterClose) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("written", self.processed.to_string()) }) } async fn abort(&mut self) -> Result<()> { self.inner.abort().await.map_err(|err| { err.with_operation(Operation::WriterAbort) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("processed", self.processed.to_string()) }) } } impl oio::BlockingWrite for ErrorContextWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { let size = bs.len(); self.inner .write(bs) .map(|_| { self.processed += size as u64; }) .map_err(|err| { err.with_operation(Operation::BlockingWriterWrite) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("size", size.to_string()) .with_context("written", self.processed.to_string()) }) } fn close(&mut self) -> Result { self.inner.close().map_err(|err| { err.with_operation(Operation::BlockingWriterClose) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("written", self.processed.to_string()) }) } } impl oio::List for ErrorContextWrapper { async fn next(&mut self) -> Result> { self.inner .next() .await .map(|bs| { self.processed += bs.is_some() as u64; bs }) .map_err(|err| { err.with_operation(Operation::ListerNext) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("listed", self.processed.to_string()) }) } } impl oio::BlockingList for ErrorContextWrapper { fn next(&mut self) -> Result> { self.inner .next() .map(|bs| { self.processed += bs.is_some() as u64; bs }) .map_err(|err| { err.with_operation(Operation::BlockingListerNext) .with_context("service", self.scheme) .with_context("path", &self.path) .with_context("listed", self.processed.to_string()) }) } } impl oio::Delete for ErrorContextWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args).map_err(|err| { err.with_operation(Operation::DeleterDelete) .with_context("service", self.scheme) .with_context("path", path) .with_context("deleted", self.processed.to_string()) }) } async fn flush(&mut self) -> Result { self.inner .flush() .await .map(|n| { self.processed += n as u64; n }) .map_err(|err| { err.with_operation(Operation::DeleterFlush) .with_context("service", self.scheme) .with_context("deleted", self.processed.to_string()) }) } } impl oio::BlockingDelete for ErrorContextWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args).map_err(|err| { err.with_operation(Operation::DeleterDelete) .with_context("service", self.scheme) .with_context("path", path) .with_context("deleted", self.processed.to_string()) }) } fn flush(&mut self) -> Result { self.inner .flush() .map(|n| { self.processed += n as u64; n }) .map_err(|err| { err.with_operation(Operation::DeleterFlush) .with_context("service", self.scheme) .with_context("deleted", self.processed.to_string()) }) } } opendal-0.52.0/src/layers/fastrace.rs000064400000000000000000000302321046102023000155560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::future::Future; use std::sync::Arc; use fastrace::prelude::*; use crate::raw::*; use crate::*; /// Add [fastrace](https://docs.rs/fastrace/) for every operation. /// /// # Examples /// /// ## Basic Setup /// /// ```no_run /// # use opendal::layers::FastraceLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(FastraceLayer) /// .finish(); /// Ok(()) /// # } /// ``` /// /// ## Real usage /// /// ```no_run /// # use anyhow::Result; /// # use fastrace::prelude::*; /// # use opendal::layers::FastraceLayer; /// # use opendal::services; /// # use opendal::Operator; /// /// # fn main() -> Result<()> { /// let reporter = /// fastrace_jaeger::JaegerReporter::new("127.0.0.1:6831".parse()?, "opendal").unwrap(); /// fastrace::set_reporter(reporter, fastrace::collector::Config::default()); /// /// { /// let root = Span::root("op", SpanContext::random()); /// let runtime = tokio::runtime::Runtime::new()?; /// runtime.block_on( /// async { /// let _ = dotenvy::dotenv(); /// let op = Operator::new(services::Memory::default())? /// .layer(FastraceLayer) /// .finish(); /// op.write("test", "0".repeat(16 * 1024 * 1024).into_bytes()) /// .await?; /// op.stat("test").await?; /// op.read("test").await?; /// Ok::<(), opendal::Error>(()) /// } /// .in_span(Span::enter_with_parent("test", &root)), /// )?; /// } /// /// fastrace::flush(); /// /// Ok(()) /// # } /// ``` /// /// # Output /// /// OpenDAL is using [`fastrace`](https://docs.rs/fastrace/latest/fastrace/) for tracing internally. /// /// To enable fastrace output, please init one of the reporter that `fastrace` supports. /// /// For example: /// /// ```no_run /// # use anyhow::Result; /// /// # fn main() -> Result<()> { /// let reporter = /// fastrace_jaeger::JaegerReporter::new("127.0.0.1:6831".parse()?, "opendal").unwrap(); /// fastrace::set_reporter(reporter, fastrace::collector::Config::default()); /// Ok(()) /// # } /// ``` /// /// For real-world usage, please take a look at [`fastrace-datadog`](https://crates.io/crates/fastrace-datadog) or [`fastrace-jaeger`](https://crates.io/crates/fastrace-jaeger) . pub struct FastraceLayer; impl Layer for FastraceLayer { type LayeredAccess = FastraceAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { FastraceAccessor { inner } } } #[derive(Debug)] pub struct FastraceAccessor { inner: A, } impl LayeredAccess for FastraceAccessor { type Inner = A; type Reader = FastraceWrapper; type BlockingReader = FastraceWrapper; type Writer = FastraceWrapper; type BlockingWriter = FastraceWrapper; type Lister = FastraceWrapper; type BlockingLister = FastraceWrapper; type Deleter = FastraceWrapper; type BlockingDeleter = FastraceWrapper; fn inner(&self) -> &Self::Inner { &self.inner } #[trace] fn info(&self) -> Arc { self.inner.info() } #[trace(enter_on_poll = true)] async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.inner.create_dir(path, args).await } #[trace(enter_on_poll = true)] async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner.read(path, args).await.map(|(rp, r)| { ( rp, FastraceWrapper::new( Span::enter_with_local_parent(Operation::Read.into_static()), r, ), ) }) } #[trace(enter_on_poll = true)] async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner.write(path, args).await.map(|(rp, r)| { ( rp, FastraceWrapper::new( Span::enter_with_local_parent(Operation::Write.into_static()), r, ), ) }) } #[trace(enter_on_poll = true)] async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner().copy(from, to, args).await } #[trace(enter_on_poll = true)] async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner().rename(from, to, args).await } #[trace(enter_on_poll = true)] async fn stat(&self, path: &str, args: OpStat) -> Result { self.inner.stat(path, args).await } #[trace(enter_on_poll = true)] async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner.delete().await.map(|(rp, r)| { ( rp, FastraceWrapper::new( Span::enter_with_local_parent(Operation::Delete.into_static()), r, ), ) }) } #[trace(enter_on_poll = true)] async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner.list(path, args).await.map(|(rp, s)| { ( rp, FastraceWrapper::new( Span::enter_with_local_parent(Operation::List.into_static()), s, ), ) }) } #[trace(enter_on_poll = true)] async fn presign(&self, path: &str, args: OpPresign) -> Result { self.inner.presign(path, args).await } #[trace] fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.inner.blocking_create_dir(path, args) } #[trace] fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner.blocking_read(path, args).map(|(rp, r)| { ( rp, FastraceWrapper::new( Span::enter_with_local_parent(Operation::BlockingRead.into_static()), r, ), ) }) } #[trace] fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner.blocking_write(path, args).map(|(rp, r)| { ( rp, FastraceWrapper::new( Span::enter_with_local_parent(Operation::BlockingWrite.into_static()), r, ), ) }) } #[trace] fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner().blocking_copy(from, to, args) } #[trace] fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner().blocking_rename(from, to, args) } #[trace] fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.inner.blocking_stat(path, args) } #[trace] fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner.blocking_delete().map(|(rp, r)| { ( rp, FastraceWrapper::new( Span::enter_with_local_parent(Operation::BlockingDelete.into_static()), r, ), ) }) } #[trace] fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner.blocking_list(path, args).map(|(rp, it)| { ( rp, FastraceWrapper::new( Span::enter_with_local_parent(Operation::BlockingList.into_static()), it, ), ) }) } } pub struct FastraceWrapper { span: Span, inner: R, } impl FastraceWrapper { fn new(span: Span, inner: R) -> Self { Self { span, inner } } } impl oio::Read for FastraceWrapper { #[trace(enter_on_poll = true)] async fn read(&mut self) -> Result { self.inner.read().await } } impl oio::BlockingRead for FastraceWrapper { fn read(&mut self) -> Result { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::BlockingReaderRead.into_static()); self.inner.read() } } impl oio::Write for FastraceWrapper { fn write(&mut self, bs: Buffer) -> impl Future> + MaybeSend { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::WriterWrite.into_static()); self.inner.write(bs) } fn abort(&mut self) -> impl Future> + MaybeSend { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::WriterAbort.into_static()); self.inner.abort() } fn close(&mut self) -> impl Future> + MaybeSend { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::WriterClose.into_static()); self.inner.close() } } impl oio::BlockingWrite for FastraceWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::BlockingWriterWrite.into_static()); self.inner.write(bs) } fn close(&mut self) -> Result { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::BlockingWriterClose.into_static()); self.inner.close() } } impl oio::List for FastraceWrapper { #[trace(enter_on_poll = true)] async fn next(&mut self) -> Result> { self.inner.next().await } } impl oio::BlockingList for FastraceWrapper { fn next(&mut self) -> Result> { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::BlockingListerNext.into_static()); self.inner.next() } } impl oio::Delete for FastraceWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::DeleterDelete.into_static()); self.inner.delete(path, args) } #[trace(enter_on_poll = true)] async fn flush(&mut self) -> Result { self.inner.flush().await } } impl oio::BlockingDelete for FastraceWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::BlockingDeleterDelete.into_static()); self.inner.delete(path, args) } fn flush(&mut self) -> Result { let _g = self.span.set_local_parent(); let _span = LocalSpan::enter_with_local_parent(Operation::BlockingDeleterFlush.into_static()); self.inner.flush() } } opendal-0.52.0/src/layers/immutable_index.rs000064400000000000000000000323351046102023000171420ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashSet; use std::fmt::Debug; use std::sync::Arc; use std::vec::IntoIter; use crate::raw::*; use crate::*; /// Add an immutable in-memory index for underlying storage services. /// /// Especially useful for services without list capability like HTTP. /// /// # Examples /// /// ```rust, no_run /// # use std::collections::HashMap; /// /// # use opendal::layers::ImmutableIndexLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # fn main() -> Result<()> { /// let mut iil = ImmutableIndexLayer::default(); /// /// for i in ["file", "dir/", "dir/file", "dir_without_prefix/file"] { /// iil.insert(i.to_string()) /// } /// /// let op = Operator::from_iter::(HashMap::<_, _>::default())? /// .layer(iil) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Default, Debug, Clone)] pub struct ImmutableIndexLayer { vec: Vec, } impl ImmutableIndexLayer { /// Insert a key into index. pub fn insert(&mut self, key: String) { self.vec.push(key); } /// Insert keys from iter. pub fn extend_iter(&mut self, iter: I) where I: IntoIterator, { self.vec.extend(iter); } } impl Layer for ImmutableIndexLayer { type LayeredAccess = ImmutableIndexAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { ImmutableIndexAccessor { vec: self.vec.clone(), inner, } } } #[derive(Debug, Clone)] pub struct ImmutableIndexAccessor { inner: A, vec: Vec, } impl ImmutableIndexAccessor { fn children_flat(&self, path: &str) -> Vec { self.vec .iter() .filter(|v| v.starts_with(path) && v.as_str() != path) .cloned() .collect() } fn children_hierarchy(&self, path: &str) -> Vec { let mut res = HashSet::new(); for i in self.vec.iter() { // `/xyz` should not belong to `/abc` if !i.starts_with(path) { continue; } // remove `/abc` if self if i == path { continue; } match i[path.len()..].find('/') { // File `/abc/def.csv` must belong to `/abc` None => { res.insert(i.to_string()); } Some(idx) => { // The index of first `/` after `/abc`. let dir_idx = idx + 1 + path.len(); if dir_idx == i.len() { // Dir `/abc/def/` belongs to `/abc/` res.insert(i.to_string()); } else { // File/Dir `/abc/def/xyz` doesn't belong to `/abc`. // But we need to list `/abc/def` out so that we can walk down. res.insert(i[..dir_idx].to_string()); } } } } res.into_iter().collect() } } impl LayeredAccess for ImmutableIndexAccessor { type Inner = A; type Reader = A::Reader; type Writer = A::Writer; type Lister = ImmutableDir; type Deleter = A::Deleter; type BlockingReader = A::BlockingReader; type BlockingWriter = A::BlockingWriter; type BlockingLister = ImmutableDir; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } /// Add list capabilities for underlying storage services. fn info(&self) -> Arc { let mut meta = (*self.inner.info()).clone(); let cap = meta.full_capability_mut(); cap.list = true; cap.list_with_recursive = true; meta.into() } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner.read(path, args).await } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner.write(path, args).await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let mut path = path; if path == "/" { path = "" } let idx = if args.recursive() { self.children_flat(path) } else { self.children_hierarchy(path) }; Ok((RpList::default(), ImmutableDir::new(idx))) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner.delete().await } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner.blocking_read(path, args) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner.blocking_write(path, args) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { let mut path = path; if path == "/" { path = "" } let idx = if args.recursive() { self.children_flat(path) } else { self.children_hierarchy(path) }; Ok((RpList::default(), ImmutableDir::new(idx))) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner.blocking_delete() } } pub struct ImmutableDir { idx: IntoIter, } impl ImmutableDir { fn new(idx: Vec) -> Self { Self { idx: idx.into_iter(), } } fn inner_next(&mut self) -> Option { self.idx.next().map(|v| { let mode = if v.ends_with('/') { EntryMode::DIR } else { EntryMode::FILE }; let meta = Metadata::new(mode); oio::Entry::with(v, meta) }) } } impl oio::List for ImmutableDir { async fn next(&mut self) -> Result> { Ok(self.inner_next()) } } impl oio::BlockingList for ImmutableDir { fn next(&mut self) -> Result> { Ok(self.inner_next()) } } #[cfg(test)] #[cfg(feature = "services-http")] mod tests { use std::collections::HashMap; use std::collections::HashSet; use anyhow::Result; use futures::TryStreamExt; use log::debug; use super::*; use crate::layers::LoggingLayer; use crate::services::HttpConfig; use crate::EntryMode; use crate::Operator; #[tokio::test] async fn test_list() -> Result<()> { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let mut iil = ImmutableIndexLayer::default(); for i in ["file", "dir/", "dir/file", "dir_without_prefix/file"] { iil.insert(i.to_string()) } let op = HttpConfig::from_iter({ let mut map = HashMap::new(); map.insert("endpoint".to_string(), "https://xuanwo.io".to_string()); map }) .and_then(Operator::from_config)? .layer(LoggingLayer::default()) .layer(iil) .finish(); let mut map = HashMap::new(); let mut set = HashSet::new(); let mut ds = op.lister("").await?; while let Some(entry) = ds.try_next().await? { debug!("got entry: {}", entry.path()); assert!( set.insert(entry.path().to_string()), "duplicated value: {}", entry.path() ); map.insert(entry.path().to_string(), entry.metadata().mode()); } assert_eq!(map["file"], EntryMode::FILE); assert_eq!(map["dir/"], EntryMode::DIR); assert_eq!(map["dir_without_prefix/"], EntryMode::DIR); Ok(()) } #[tokio::test] async fn test_scan() -> Result<()> { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let mut iil = ImmutableIndexLayer::default(); for i in ["file", "dir/", "dir/file", "dir_without_prefix/file"] { iil.insert(i.to_string()) } let op = HttpConfig::from_iter({ let mut map = HashMap::new(); map.insert("endpoint".to_string(), "https://xuanwo.io".to_string()); map }) .and_then(Operator::from_config)? .layer(LoggingLayer::default()) .layer(iil) .finish(); let mut ds = op.lister_with("/").recursive(true).await?; let mut set = HashSet::new(); let mut map = HashMap::new(); while let Some(entry) = ds.try_next().await? { debug!("got entry: {}", entry.path()); assert!( set.insert(entry.path().to_string()), "duplicated value: {}", entry.path() ); map.insert(entry.path().to_string(), entry.metadata().mode()); } debug!("current files: {:?}", map); assert_eq!(map["file"], EntryMode::FILE); assert_eq!(map["dir/"], EntryMode::DIR); assert_eq!(map["dir_without_prefix/file"], EntryMode::FILE); Ok(()) } #[tokio::test] async fn test_list_dir() -> Result<()> { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let mut iil = ImmutableIndexLayer::default(); for i in [ "dataset/stateful/ontime_2007_200.csv", "dataset/stateful/ontime_2008_200.csv", "dataset/stateful/ontime_2009_200.csv", ] { iil.insert(i.to_string()) } let op = HttpConfig::from_iter({ let mut map = HashMap::new(); map.insert("endpoint".to_string(), "https://xuanwo.io".to_string()); map }) .and_then(Operator::from_config)? .layer(LoggingLayer::default()) .layer(iil) .finish(); // List / let mut map = HashMap::new(); let mut set = HashSet::new(); let mut ds = op.lister("/").await?; while let Some(entry) = ds.try_next().await? { assert!( set.insert(entry.path().to_string()), "duplicated value: {}", entry.path() ); map.insert(entry.path().to_string(), entry.metadata().mode()); } assert_eq!(map.len(), 1); assert_eq!(map["dataset/"], EntryMode::DIR); // List dataset/stateful/ let mut map = HashMap::new(); let mut set = HashSet::new(); let mut ds = op.lister("dataset/stateful/").await?; while let Some(entry) = ds.try_next().await? { assert!( set.insert(entry.path().to_string()), "duplicated value: {}", entry.path() ); map.insert(entry.path().to_string(), entry.metadata().mode()); } assert_eq!(map["dataset/stateful/ontime_2007_200.csv"], EntryMode::FILE); assert_eq!(map["dataset/stateful/ontime_2008_200.csv"], EntryMode::FILE); assert_eq!(map["dataset/stateful/ontime_2009_200.csv"], EntryMode::FILE); Ok(()) } #[tokio::test] async fn test_walk_top_down_dir() -> Result<()> { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let mut iil = ImmutableIndexLayer::default(); for i in [ "dataset/stateful/ontime_2007_200.csv", "dataset/stateful/ontime_2008_200.csv", "dataset/stateful/ontime_2009_200.csv", ] { iil.insert(i.to_string()) } let op = HttpConfig::from_iter({ let mut map = HashMap::new(); map.insert("endpoint".to_string(), "https://xuanwo.io".to_string()); map }) .and_then(Operator::from_config)? .layer(LoggingLayer::default()) .layer(iil) .finish(); let mut ds = op.lister_with("/").recursive(true).await?; let mut map = HashMap::new(); let mut set = HashSet::new(); while let Some(entry) = ds.try_next().await? { assert!( set.insert(entry.path().to_string()), "duplicated value: {}", entry.path() ); map.insert(entry.path().to_string(), entry.metadata().mode()); } debug!("current files: {:?}", map); assert_eq!(map["dataset/stateful/ontime_2007_200.csv"], EntryMode::FILE); assert_eq!(map["dataset/stateful/ontime_2008_200.csv"], EntryMode::FILE); assert_eq!(map["dataset/stateful/ontime_2009_200.csv"], EntryMode::FILE); Ok(()) } } opendal-0.52.0/src/layers/logging.rs000064400000000000000000001245401046102023000154220ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Display; use std::sync::Arc; use log::log; use log::Level; use crate::raw::*; use crate::*; /// Add [log](https://docs.rs/log/) for every operation. /// /// # Logging /// /// - OpenDAL will log in structural way. /// - Every operation will start with a `started` log entry. /// - Every operation will finish with the following status: /// - `succeeded`: the operation is successful, but might have more to take. /// - `finished`: the whole operation is finished. /// - `failed`: the operation returns an unexpected error. /// - The default log level while expected error happened is `Warn`. /// - The default log level while unexpected failure happened is `Error`. /// /// # Examples /// /// ```no_run /// # use opendal::layers::LoggingLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(LoggingLayer::default()) /// .finish(); /// Ok(()) /// # } /// ``` /// /// # Output /// /// OpenDAL is using [`log`](https://docs.rs/log/latest/log/) for logging internally. /// /// To enable logging output, please set `RUST_LOG`: /// /// ```shell /// RUST_LOG=debug ./app /// ``` /// /// To config logging output, please refer to [Configure Logging](https://rust-lang-nursery.github.io/rust-cookbook/development_tools/debugging/config_log.html): /// /// ```shell /// RUST_LOG="info,opendal::services=debug" ./app /// ``` /// /// # Logging Interceptor /// /// You can implement your own logging interceptor to customize the logging behavior. /// /// ```no_run /// # use opendal::layers::LoggingInterceptor; /// # use opendal::layers::LoggingLayer; /// # use opendal::raw; /// # use opendal::services; /// # use opendal::Error; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// #[derive(Debug, Clone)] /// struct MyLoggingInterceptor; /// /// impl LoggingInterceptor for MyLoggingInterceptor { /// fn log( /// &self, /// info: &raw::AccessorInfo, /// operation: raw::Operation, /// context: &[(&str, &str)], /// message: &str, /// err: Option<&Error>, /// ) { /// // log something /// } /// } /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(LoggingLayer::new(MyLoggingInterceptor)) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Debug)] pub struct LoggingLayer { logger: I, } impl Default for LoggingLayer { fn default() -> Self { Self { logger: DefaultLoggingInterceptor, } } } impl LoggingLayer { /// Create the layer with specific logging interceptor. pub fn new(logger: I) -> LoggingLayer { LoggingLayer { logger } } } impl Layer for LoggingLayer { type LayeredAccess = LoggingAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { let info = inner.info(); LoggingAccessor { inner, info, logger: self.logger.clone(), } } } /// LoggingInterceptor is used to intercept the log. pub trait LoggingInterceptor: Debug + Clone + Send + Sync + Unpin + 'static { /// Everytime there is a log, this function will be called. /// /// # Inputs /// /// - info: The service's access info. /// - operation: The operation to log. /// - context: Additional context of the log like path, etc. /// - message: The log message. /// - err: The error to log. /// /// # Note /// /// Users should avoid calling resource-intensive operations such as I/O or network /// functions here, especially anything that takes longer than 10ms. Otherwise, Opendal /// could perform unexpectedly slow. fn log( &self, info: &AccessorInfo, operation: Operation, context: &[(&str, &str)], message: &str, err: Option<&Error>, ); } /// The DefaultLoggingInterceptor will log the message by the standard logging macro. #[derive(Debug, Copy, Clone, Default)] pub struct DefaultLoggingInterceptor; impl LoggingInterceptor for DefaultLoggingInterceptor { #[inline] fn log( &self, info: &AccessorInfo, operation: Operation, context: &[(&str, &str)], message: &str, err: Option<&Error>, ) { if let Some(err) = err { // Print error if it's unexpected, otherwise in warn. let lvl = if err.kind() == ErrorKind::Unexpected { Level::Error } else { Level::Warn }; log!( target: LOGGING_TARGET, lvl, "service={} name={}{}: {operation} {message} {}", info.scheme(), info.name(), LoggingContext(context), // Print error message with debug output while unexpected happened. // // It's super sad that we can't bind `format_args!()` here. // See: https://github.com/rust-lang/rust/issues/92698 if err.kind() != ErrorKind::Unexpected { format!("{err}") } else { format!("{err:?}") } ); } // Print debug message if operation is oneshot, otherwise in trace. let lvl = if operation.is_oneshot() { Level::Debug } else { Level::Trace }; log!( target: LOGGING_TARGET, lvl, "service={} name={}{}: {operation} {message}", info.scheme(), info.name(), LoggingContext(context), ); } } struct LoggingContext<'a>(&'a [(&'a str, &'a str)]); impl Display for LoggingContext<'_> { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { for (k, v) in self.0.iter() { write!(f, " {}={}", k, v)?; } Ok(()) } } #[derive(Clone, Debug)] pub struct LoggingAccessor { inner: A, info: Arc, logger: I, } static LOGGING_TARGET: &str = "opendal::services"; impl LayeredAccess for LoggingAccessor { type Inner = A; type Reader = LoggingReader; type BlockingReader = LoggingReader; type Writer = LoggingWriter; type BlockingWriter = LoggingWriter; type Lister = LoggingLister; type BlockingLister = LoggingLister; type Deleter = LoggingDeleter; type BlockingDeleter = LoggingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } fn info(&self) -> Arc { self.logger .log(&self.info, Operation::Info, &[], "started", None); let info = self.info.clone(); self.logger .log(&self.info, Operation::Info, &[], "finished", None); info } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.logger.log( &self.info, Operation::CreateDir, &[("path", path)], "started", None, ); self.inner .create_dir(path, args) .await .map(|v| { self.logger.log( &self.info, Operation::CreateDir, &[("path", path)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::CreateDir, &[("path", path)], "failed", Some(&err), ); err }) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.logger.log( &self.info, Operation::Read, &[("path", path)], "started", None, ); self.inner .read(path, args) .await .map(|(rp, r)| { self.logger.log( &self.info, Operation::Read, &[("path", path)], "created reader", None, ); ( rp, LoggingReader::new(self.info.clone(), self.logger.clone(), path, r), ) }) .map_err(|err| { self.logger.log( &self.info, Operation::Read, &[("path", path)], "failed", Some(&err), ); err }) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.logger.log( &self.info, Operation::Write, &[("path", path)], "started", None, ); self.inner .write(path, args) .await .map(|(rp, w)| { self.logger.log( &self.info, Operation::Write, &[("path", path)], "created writer", None, ); let w = LoggingWriter::new(self.info.clone(), self.logger.clone(), path, w); (rp, w) }) .map_err(|err| { self.logger.log( &self.info, Operation::Write, &[("path", path)], "failed", Some(&err), ); err }) } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.logger.log( &self.info, Operation::Copy, &[("from", from), ("to", to)], "started", None, ); self.inner .copy(from, to, args) .await .map(|v| { self.logger.log( &self.info, Operation::Copy, &[("from", from), ("to", to)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::Copy, &[("from", from), ("to", to)], "failed", Some(&err), ); err }) } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.logger.log( &self.info, Operation::Rename, &[("from", from), ("to", to)], "started", None, ); self.inner .rename(from, to, args) .await .map(|v| { self.logger.log( &self.info, Operation::Rename, &[("from", from), ("to", to)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::Rename, &[("from", from), ("to", to)], "failed", Some(&err), ); err }) } async fn stat(&self, path: &str, args: OpStat) -> Result { self.logger.log( &self.info, Operation::Stat, &[("path", path)], "started", None, ); self.inner .stat(path, args) .await .map(|v| { self.logger.log( &self.info, Operation::Stat, &[("path", path)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::Stat, &[("path", path)], "failed", Some(&err), ); err }) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.logger .log(&self.info, Operation::Delete, &[], "started", None); self.inner .delete() .await .map(|(rp, d)| { self.logger .log(&self.info, Operation::Delete, &[], "finished", None); let d = LoggingDeleter::new(self.info.clone(), self.logger.clone(), d); (rp, d) }) .map_err(|err| { self.logger .log(&self.info, Operation::Delete, &[], "failed", Some(&err)); err }) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.logger.log( &self.info, Operation::List, &[("path", path)], "started", None, ); self.inner .list(path, args) .await .map(|(rp, v)| { self.logger.log( &self.info, Operation::List, &[("path", path)], "created lister", None, ); let streamer = LoggingLister::new(self.info.clone(), self.logger.clone(), path, v); (rp, streamer) }) .map_err(|err| { self.logger.log( &self.info, Operation::List, &[("path", path)], "failed", Some(&err), ); err }) } async fn presign(&self, path: &str, args: OpPresign) -> Result { self.logger.log( &self.info, Operation::Presign, &[("path", path)], "started", None, ); self.inner .presign(path, args) .await .map(|v| { self.logger.log( &self.info, Operation::Presign, &[("path", path)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::Presign, &[("path", path)], "failed", Some(&err), ); err }) } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.logger.log( &self.info, Operation::BlockingCreateDir, &[("path", path)], "started", None, ); self.inner .blocking_create_dir(path, args) .map(|v| { self.logger.log( &self.info, Operation::BlockingCreateDir, &[("path", path)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::BlockingCreateDir, &[("path", path)], "failed", Some(&err), ); err }) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.logger.log( &self.info, Operation::BlockingRead, &[("path", path)], "started", None, ); self.inner .blocking_read(path, args.clone()) .map(|(rp, r)| { self.logger.log( &self.info, Operation::BlockingRead, &[("path", path)], "created reader", None, ); let r = LoggingReader::new(self.info.clone(), self.logger.clone(), path, r); (rp, r) }) .map_err(|err| { self.logger.log( &self.info, Operation::BlockingRead, &[("path", path)], "failed", Some(&err), ); err }) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.logger.log( &self.info, Operation::BlockingWrite, &[("path", path)], "started", None, ); self.inner .blocking_write(path, args) .map(|(rp, w)| { self.logger.log( &self.info, Operation::BlockingWrite, &[("path", path)], "created writer", None, ); let w = LoggingWriter::new(self.info.clone(), self.logger.clone(), path, w); (rp, w) }) .map_err(|err| { self.logger.log( &self.info, Operation::BlockingWrite, &[("path", path)], "failed", Some(&err), ); err }) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.logger.log( &self.info, Operation::BlockingCopy, &[("from", from), ("to", to)], "started", None, ); self.inner .blocking_copy(from, to, args) .map(|v| { self.logger.log( &self.info, Operation::BlockingCopy, &[("from", from), ("to", to)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::BlockingCopy, &[("from", from), ("to", to)], "", Some(&err), ); err }) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.logger.log( &self.info, Operation::BlockingRename, &[("from", from), ("to", to)], "started", None, ); self.inner .blocking_rename(from, to, args) .map(|v| { self.logger.log( &self.info, Operation::BlockingRename, &[("from", from), ("to", to)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::BlockingRename, &[("from", from), ("to", to)], "failed", Some(&err), ); err }) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.logger.log( &self.info, Operation::BlockingStat, &[("path", path)], "started", None, ); self.inner .blocking_stat(path, args) .map(|v| { self.logger.log( &self.info, Operation::BlockingStat, &[("path", path)], "finished", None, ); v }) .map_err(|err| { self.logger.log( &self.info, Operation::BlockingStat, &[("path", path)], "failed", Some(&err), ); err }) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.logger .log(&self.info, Operation::BlockingDelete, &[], "started", None); self.inner .blocking_delete() .map(|(rp, d)| { self.logger .log(&self.info, Operation::BlockingDelete, &[], "finished", None); let d = LoggingDeleter::new(self.info.clone(), self.logger.clone(), d); (rp, d) }) .map_err(|err| { self.logger.log( &self.info, Operation::BlockingDelete, &[], "failed", Some(&err), ); err }) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.logger.log( &self.info, Operation::BlockingList, &[("path", path)], "started", None, ); self.inner .blocking_list(path, args) .map(|(rp, v)| { self.logger.log( &self.info, Operation::BlockingList, &[("path", path)], "created lister", None, ); let li = LoggingLister::new(self.info.clone(), self.logger.clone(), path, v); (rp, li) }) .map_err(|err| { self.logger.log( &self.info, Operation::BlockingList, &[("path", path)], "", Some(&err), ); err }) } } /// `LoggingReader` is a wrapper of `BytesReader`, with logging functionality. pub struct LoggingReader { info: Arc, logger: I, path: String, read: u64, inner: R, } impl LoggingReader { fn new(info: Arc, logger: I, path: &str, reader: R) -> Self { Self { info, logger, path: path.to_string(), read: 0, inner: reader, } } } impl oio::Read for LoggingReader { async fn read(&mut self) -> Result { self.logger.log( &self.info, Operation::ReaderRead, &[("path", &self.path), ("read", &self.read.to_string())], "started", None, ); match self.inner.read().await { Ok(bs) => { self.read += bs.len() as u64; self.logger.log( &self.info, Operation::ReaderRead, &[ ("path", &self.path), ("read", &self.read.to_string()), ("size", &bs.len().to_string()), ], if bs.is_empty() { "finished" } else { "succeeded" }, None, ); Ok(bs) } Err(err) => { self.logger.log( &self.info, Operation::ReaderRead, &[("path", &self.path), ("read", &self.read.to_string())], "failed", Some(&err), ); Err(err) } } } } impl oio::BlockingRead for LoggingReader { fn read(&mut self) -> Result { self.logger.log( &self.info, Operation::BlockingReaderRead, &[("path", &self.path), ("read", &self.read.to_string())], "started", None, ); match self.inner.read() { Ok(bs) => { self.read += bs.len() as u64; self.logger.log( &self.info, Operation::BlockingReaderRead, &[ ("path", &self.path), ("read", &self.read.to_string()), ("size", &bs.len().to_string()), ], if bs.is_empty() { "finished" } else { "succeeded" }, None, ); Ok(bs) } Err(err) => { self.logger.log( &self.info, Operation::BlockingReaderRead, &[("path", &self.path), ("read", &self.read.to_string())], "failed", Some(&err), ); Err(err) } } } } pub struct LoggingWriter { info: Arc, logger: I, path: String, written: u64, inner: W, } impl LoggingWriter { fn new(info: Arc, logger: I, path: &str, writer: W) -> Self { Self { info, logger, path: path.to_string(), written: 0, inner: writer, } } } impl oio::Write for LoggingWriter { async fn write(&mut self, bs: Buffer) -> Result<()> { let size = bs.len(); self.logger.log( &self.info, Operation::WriterWrite, &[ ("path", &self.path), ("written", &self.written.to_string()), ("size", &size.to_string()), ], "started", None, ); match self.inner.write(bs).await { Ok(_) => { self.written += size as u64; self.logger.log( &self.info, Operation::WriterWrite, &[ ("path", &self.path), ("written", &self.written.to_string()), ("size", &size.to_string()), ], "succeeded", None, ); Ok(()) } Err(err) => { self.logger.log( &self.info, Operation::WriterWrite, &[ ("path", &self.path), ("written", &self.written.to_string()), ("size", &size.to_string()), ], "failed", Some(&err), ); Err(err) } } } async fn abort(&mut self) -> Result<()> { self.logger.log( &self.info, Operation::WriterAbort, &[("path", &self.path), ("written", &self.written.to_string())], "started", None, ); match self.inner.abort().await { Ok(_) => { self.logger.log( &self.info, Operation::WriterAbort, &[("path", &self.path), ("written", &self.written.to_string())], "succeeded", None, ); Ok(()) } Err(err) => { self.logger.log( &self.info, Operation::WriterAbort, &[("path", &self.path), ("written", &self.written.to_string())], "failed", Some(&err), ); Err(err) } } } async fn close(&mut self) -> Result { self.logger.log( &self.info, Operation::WriterClose, &[("path", &self.path), ("written", &self.written.to_string())], "started", None, ); match self.inner.close().await { Ok(meta) => { self.logger.log( &self.info, Operation::WriterClose, &[("path", &self.path), ("written", &self.written.to_string())], "succeeded", None, ); Ok(meta) } Err(err) => { self.logger.log( &self.info, Operation::WriterClose, &[("path", &self.path), ("written", &self.written.to_string())], "failed", Some(&err), ); Err(err) } } } } impl oio::BlockingWrite for LoggingWriter { fn write(&mut self, bs: Buffer) -> Result<()> { let size = bs.len(); self.logger.log( &self.info, Operation::BlockingWriterWrite, &[ ("path", &self.path), ("written", &self.written.to_string()), ("size", &size.to_string()), ], "started", None, ); match self.inner.write(bs) { Ok(_) => { self.logger.log( &self.info, Operation::BlockingWriterWrite, &[ ("path", &self.path), ("written", &self.written.to_string()), ("size", &size.to_string()), ], "succeeded", None, ); Ok(()) } Err(err) => { self.logger.log( &self.info, Operation::BlockingWriterWrite, &[ ("path", &self.path), ("written", &self.written.to_string()), ("size", &size.to_string()), ], "failed", Some(&err), ); Err(err) } } } fn close(&mut self) -> Result { self.logger.log( &self.info, Operation::BlockingWriterClose, &[("path", &self.path), ("written", &self.written.to_string())], "started", None, ); match self.inner.close() { Ok(meta) => { self.logger.log( &self.info, Operation::BlockingWriterWrite, &[("path", &self.path), ("written", &self.written.to_string())], "succeeded", None, ); Ok(meta) } Err(err) => { self.logger.log( &self.info, Operation::BlockingWriterClose, &[("path", &self.path), ("written", &self.written.to_string())], "failed", Some(&err), ); Err(err) } } } } pub struct LoggingLister { info: Arc, logger: I, path: String, listed: usize, inner: P, } impl LoggingLister { fn new(info: Arc, logger: I, path: &str, inner: P) -> Self { Self { info, logger, path: path.to_string(), listed: 0, inner, } } } impl oio::List for LoggingLister { async fn next(&mut self) -> Result> { self.logger.log( &self.info, Operation::ListerNext, &[("path", &self.path), ("listed", &self.listed.to_string())], "started", None, ); let res = self.inner.next().await; match &res { Ok(Some(de)) => { self.listed += 1; self.logger.log( &self.info, Operation::ListerNext, &[ ("path", &self.path), ("listed", &self.listed.to_string()), ("entry", de.path()), ], "succeeded", None, ); } Ok(None) => { self.logger.log( &self.info, Operation::ListerNext, &[("path", &self.path), ("listed", &self.listed.to_string())], "finished", None, ); } Err(err) => { self.logger.log( &self.info, Operation::ListerNext, &[("path", &self.path), ("listed", &self.listed.to_string())], "failed", Some(err), ); } }; res } } impl oio::BlockingList for LoggingLister { fn next(&mut self) -> Result> { self.logger.log( &self.info, Operation::BlockingListerNext, &[("path", &self.path), ("listed", &self.listed.to_string())], "started", None, ); let res = self.inner.next(); match &res { Ok(Some(de)) => { self.listed += 1; self.logger.log( &self.info, Operation::BlockingListerNext, &[ ("path", &self.path), ("listed", &self.listed.to_string()), ("entry", de.path()), ], "succeeded", None, ); } Ok(None) => { self.logger.log( &self.info, Operation::BlockingListerNext, &[("path", &self.path), ("listed", &self.listed.to_string())], "finished", None, ); } Err(err) => { self.logger.log( &self.info, Operation::BlockingListerNext, &[("path", &self.path), ("listed", &self.listed.to_string())], "failed", Some(err), ); } }; res } } pub struct LoggingDeleter { info: Arc, logger: I, queued: usize, deleted: usize, inner: D, } impl LoggingDeleter { fn new(info: Arc, logger: I, inner: D) -> Self { Self { info, logger, queued: 0, deleted: 0, inner, } } } impl oio::Delete for LoggingDeleter { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { let version = args .version() .map(|v| v.to_string()) .unwrap_or_else(|| "".to_string()); self.logger.log( &self.info, Operation::DeleterDelete, &[("path", path), ("version", &version)], "started", None, ); let res = self.inner.delete(path, args); match &res { Ok(_) => { self.queued += 1; self.logger.log( &self.info, Operation::DeleterDelete, &[ ("path", path), ("version", &version), ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "succeeded", None, ); } Err(err) => { self.logger.log( &self.info, Operation::DeleterDelete, &[ ("path", path), ("version", &version), ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "failed", Some(err), ); } }; res } async fn flush(&mut self) -> Result { self.logger.log( &self.info, Operation::DeleterFlush, &[ ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "started", None, ); let res = self.inner.flush().await; match &res { Ok(flushed) => { self.queued -= flushed; self.deleted += flushed; self.logger.log( &self.info, Operation::DeleterFlush, &[ ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "succeeded", None, ); } Err(err) => { self.logger.log( &self.info, Operation::DeleterFlush, &[ ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "failed", Some(err), ); } }; res } } impl oio::BlockingDelete for LoggingDeleter { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { let version = args .version() .map(|v| v.to_string()) .unwrap_or_else(|| "".to_string()); self.logger.log( &self.info, Operation::BlockingDeleterDelete, &[("path", path), ("version", &version)], "started", None, ); let res = self.inner.delete(path, args); match &res { Ok(_) => { self.queued += 1; self.logger.log( &self.info, Operation::BlockingDeleterDelete, &[ ("path", path), ("version", &version), ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "succeeded", None, ); } Err(err) => { self.logger.log( &self.info, Operation::BlockingDeleterDelete, &[ ("path", path), ("version", &version), ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "failed", Some(err), ); } }; res } fn flush(&mut self) -> Result { self.logger.log( &self.info, Operation::BlockingDeleterFlush, &[ ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "started", None, ); let res = self.inner.flush(); match &res { Ok(flushed) => { self.queued -= flushed; self.deleted += flushed; self.logger.log( &self.info, Operation::BlockingDeleterFlush, &[ ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "succeeded", None, ); } Err(err) => { self.logger.log( &self.info, Operation::BlockingDeleterFlush, &[ ("queued", &self.queued.to_string()), ("deleted", &self.deleted.to_string()), ], "failed", Some(err), ); } }; res } } opendal-0.52.0/src/layers/metrics.rs000064400000000000000000000142561046102023000154440ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use std::time::Duration; use metrics::counter; use metrics::histogram; use metrics::Label; use crate::layers::observe; use crate::raw::*; use crate::*; /// Add [metrics](https://docs.rs/metrics/) for every operation. /// /// # Metrics /// /// We provide several metrics, please see the documentation of [`observe`] module. /// /// # Notes /// /// Please make sure the exporter has been pulled in regular time. /// Otherwise, the histogram data collected by `requests_duration_seconds` /// could result in OOM. /// /// # Examples /// /// ```no_run /// # use opendal::layers::MetricsLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(MetricsLayer::default()) /// .finish(); /// Ok(()) /// # } /// ``` /// /// # Output /// /// OpenDAL is using [`metrics`](https://docs.rs/metrics/latest/metrics/) for metrics internally. /// /// To enable metrics output, please enable one of the exporters that `metrics` supports. /// /// Take [`metrics_exporter_prometheus`](https://docs.rs/metrics-exporter-prometheus/latest/metrics_exporter_prometheus/) as an example: /// /// ```ignore /// let builder = PrometheusBuilder::new(); /// builder.install().expect("failed to install recorder/exporter"); /// let handle = builder.install_recorder().expect("failed to install recorder"); /// let (recorder, exporter) = builder.build().expect("failed to build recorder/exporter"); /// let recorder = builder.build_recorder().expect("failed to build recorder"); /// ``` #[derive(Clone, Debug, Default)] pub struct MetricsLayer { path_label_level: usize, } impl MetricsLayer { /// Set the level of path label. /// /// - level = 0: we will ignore the path label. /// - level > 0: the path label will be the path split by "/" and get the last n level, /// if n=1 and input path is "abc/def/ghi", and then we will get "abc/" as the path label. pub fn path_label(mut self, level: usize) -> Self { self.path_label_level = level; self } } impl Layer for MetricsLayer { type LayeredAccess = observe::MetricsAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { let interceptor = MetricsInterceptor { path_label_level: self.path_label_level, }; observe::MetricsLayer::new(interceptor).layer(inner) } } #[derive(Clone, Debug)] pub struct MetricsInterceptor { path_label_level: usize, } impl observe::MetricsIntercept for MetricsInterceptor { fn observe_operation_duration_seconds( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, duration: Duration, ) { let labels = OperationLabels { scheme, namespace, root, path, operation: op, error: None, } .into_labels(self.path_label_level); histogram!(observe::METRIC_OPERATION_DURATION_SECONDS.name(), labels).record(duration) } fn observe_operation_bytes( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, bytes: usize, ) { let labels = OperationLabels { scheme, namespace, root, path, operation: op, error: None, } .into_labels(self.path_label_level); histogram!(observe::METRIC_OPERATION_BYTES.name(), labels).record(bytes as f64) } fn observe_operation_errors_total( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, error: ErrorKind, ) { let labels = OperationLabels { scheme, namespace, root, path, operation: op, error: Some(error), } .into_labels(self.path_label_level); counter!(observe::METRIC_OPERATION_ERRORS_TOTAL.name(), labels).increment(1) } } struct OperationLabels<'a> { scheme: Scheme, namespace: Arc, root: Arc, path: &'a str, operation: Operation, error: Option, } impl OperationLabels<'_> { /// labels: /// /// 1. `["scheme", "namespace", "root", "operation"]` /// 2. `["scheme", "namespace", "root", "operation", "path"]` /// 3. `["scheme", "namespace", "root", "operation", "error"]` /// 4. `["scheme", "namespace", "root", "operation", "path", "error"]` fn into_labels(self, path_label_level: usize) -> Vec for MimeGuessLayer { type LayeredAccess = MimeGuessAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { MimeGuessAccessor(inner) } } #[derive(Clone, Debug)] pub struct MimeGuessAccessor(A); fn mime_from_path(path: &str) -> Option<&str> { mime_guess::from_path(path).first_raw() } fn opwrite_with_mime(path: &str, op: OpWrite) -> OpWrite { if op.content_type().is_some() { return op; } if let Some(mime) = mime_from_path(path) { return op.with_content_type(mime); } op } fn rpstat_with_mime(path: &str, rp: RpStat) -> RpStat { rp.map_metadata(|metadata| { if metadata.content_type().is_some() { return metadata; } if let Some(mime) = mime_from_path(path) { return metadata.with_content_type(mime.into()); } metadata }) } impl LayeredAccess for MimeGuessAccessor { type Inner = A; type Reader = A::Reader; type Writer = A::Writer; type Lister = A::Lister; type Deleter = A::Deleter; type BlockingReader = A::BlockingReader; type BlockingWriter = A::BlockingWriter; type BlockingLister = A::BlockingLister; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.0 } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner().read(path, args).await } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner() .write(path, opwrite_with_mime(path, args)) .await } async fn stat(&self, path: &str, args: OpStat) -> Result { self.inner() .stat(path, args) .await .map(|rp| rpstat_with_mime(path, rp)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner().delete().await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner().list(path, args).await } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner().blocking_read(path, args) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner() .blocking_write(path, opwrite_with_mime(path, args)) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.inner() .blocking_stat(path, args) .map(|rp| rpstat_with_mime(path, rp)) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner().blocking_delete() } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner().blocking_list(path, args) } } #[cfg(test)] mod tests { use super::*; use crate::services::Memory; use crate::Metadata; use crate::Operator; use futures::TryStreamExt; const DATA: &str = "test"; const CUSTOM: &str = "text/custom"; const HTML: &str = "text/html"; #[tokio::test] async fn test_async() { let op = Operator::new(Memory::default()) .unwrap() .layer(MimeGuessLayer::default()) .finish(); op.write("test0.html", DATA).await.unwrap(); assert_eq!( op.stat("test0.html").await.unwrap().content_type(), Some(HTML) ); op.write("test1.asdfghjkl", DATA).await.unwrap(); assert_eq!( op.stat("test1.asdfghjkl").await.unwrap().content_type(), None ); op.write_with("test2.html", DATA) .content_type(CUSTOM) .await .unwrap(); assert_eq!( op.stat("test2.html").await.unwrap().content_type(), Some(CUSTOM) ); let entries: Vec = op .lister_with("") .await .unwrap() .and_then(|entry| { let op = op.clone(); async move { op.stat(entry.path()).await } }) .try_collect() .await .unwrap(); assert_eq!(entries[0].content_type(), Some(HTML)); assert_eq!(entries[1].content_type(), None); assert_eq!(entries[2].content_type(), Some(CUSTOM)); } #[test] fn test_blocking() { let op = Operator::new(Memory::default()) .unwrap() .layer(MimeGuessLayer::default()) .finish() .blocking(); op.write("test0.html", DATA).unwrap(); assert_eq!(op.stat("test0.html").unwrap().content_type(), Some(HTML)); op.write("test1.asdfghjkl", DATA).unwrap(); assert_eq!(op.stat("test1.asdfghjkl").unwrap().content_type(), None); op.write_with("test2.html", DATA) .content_type(CUSTOM) .call() .unwrap(); assert_eq!(op.stat("test2.html").unwrap().content_type(), Some(CUSTOM)); let entries: Vec = op .lister_with("") .call() .unwrap() .map(|entry| { let op = op.clone(); op.stat(entry.unwrap().path()).unwrap() }) .collect(); assert_eq!(entries[0].content_type(), Some(HTML)); assert_eq!(entries[1].content_type(), None); assert_eq!(entries[2].content_type(), Some(CUSTOM)); } } opendal-0.52.0/src/layers/mod.rs000064400000000000000000000071101046102023000145440ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! `Layer` is the mechanism to intercept operations. mod type_eraser; pub(crate) use type_eraser::TypeEraseLayer; mod error_context; pub(crate) use error_context::ErrorContextLayer; mod complete; pub(crate) use complete::CompleteLayer; mod concurrent_limit; pub use concurrent_limit::ConcurrentLimitLayer; mod immutable_index; pub use immutable_index::ImmutableIndexLayer; mod logging; pub use logging::LoggingInterceptor; pub use logging::LoggingLayer; mod timeout; pub use timeout::TimeoutLayer; #[cfg(feature = "layers-blocking")] mod blocking; #[cfg(feature = "layers-blocking")] pub use blocking::BlockingLayer; #[cfg(feature = "layers-chaos")] mod chaos; #[cfg(feature = "layers-chaos")] pub use chaos::ChaosLayer; #[cfg(feature = "layers-metrics")] mod metrics; #[cfg(feature = "layers-metrics")] pub use self::metrics::MetricsLayer; #[cfg(feature = "layers-mime-guess")] mod mime_guess; #[cfg(feature = "layers-mime-guess")] pub use self::mime_guess::MimeGuessLayer; #[cfg(feature = "layers-prometheus")] mod prometheus; #[cfg(feature = "layers-prometheus")] pub use self::prometheus::PrometheusLayer; #[cfg(feature = "layers-prometheus")] pub use self::prometheus::PrometheusLayerBuilder; #[cfg(feature = "layers-prometheus-client")] mod prometheus_client; #[cfg(feature = "layers-prometheus-client")] pub use self::prometheus_client::PrometheusClientLayer; #[cfg(feature = "layers-prometheus-client")] pub use self::prometheus_client::PrometheusClientLayerBuilder; mod retry; pub use self::retry::RetryInterceptor; pub use self::retry::RetryLayer; #[cfg(feature = "layers-tracing")] mod tracing; #[cfg(feature = "layers-tracing")] pub use self::tracing::TracingLayer; #[cfg(feature = "layers-fastrace")] mod fastrace; #[cfg(feature = "layers-fastrace")] pub use self::fastrace::FastraceLayer; #[cfg(feature = "layers-otel-metrics")] mod otelmetrics; #[cfg(feature = "layers-otel-metrics")] pub use self::otelmetrics::OtelMetricsLayer; #[cfg(feature = "layers-otel-trace")] mod oteltrace; #[cfg(feature = "layers-otel-trace")] pub use self::oteltrace::OtelTraceLayer; #[cfg(feature = "layers-throttle")] mod throttle; #[cfg(feature = "layers-throttle")] pub use self::throttle::ThrottleLayer; #[cfg(feature = "layers-await-tree")] mod await_tree; #[cfg(feature = "layers-await-tree")] pub use self::await_tree::AwaitTreeLayer; #[cfg(feature = "layers-async-backtrace")] mod async_backtrace; #[cfg(feature = "layers-async-backtrace")] pub use self::async_backtrace::AsyncBacktraceLayer; #[cfg(all(target_os = "linux", feature = "layers-dtrace"))] mod dtrace; #[cfg(all(target_os = "linux", feature = "layers-dtrace"))] pub use self::dtrace::DtraceLayer; pub mod observe; mod correctness_check; pub(crate) use correctness_check::CorrectnessCheckLayer; mod capability_check; pub use capability_check::CapabilityCheckLayer; opendal-0.52.0/src/layers/observe/metrics.rs000064400000000000000000001116201046102023000171020ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::Write; use std::sync::Arc; use std::time::Duration; use std::time::Instant; use crate::raw::*; use crate::*; /// The metric metadata which contains the metric name and help. pub struct MetricMetadata { name: &'static str, help: &'static str, } impl MetricMetadata { /// Returns the metric name. /// /// We default to using the metric name with the prefix `opendal_`. pub fn name(&self) -> String { self.name_with_prefix("opendal_".to_string()) } /// Returns the metric name with a given prefix. pub fn name_with_prefix(&self, mut prefix: String) -> String { // This operation must succeed. If an error does occur, let's just ignore it. let _ = prefix.write_str(self.name); prefix } /// Returns the metric help. pub fn help(&self) -> &'static str { self.help } } /// The metric metadata for the operation duration in seconds. pub static METRIC_OPERATION_DURATION_SECONDS: MetricMetadata = MetricMetadata { name: "operation_duration_seconds", help: "Histogram of time spent during opendal operations", }; /// The metric metadata for the operation bytes. pub static METRIC_OPERATION_BYTES: MetricMetadata = MetricMetadata { name: "operation_bytes", help: "Histogram of the bytes transferred during opendal operations", }; /// The metric metadata for the operation errors total. pub static METRIC_OPERATION_ERRORS_TOTAL: MetricMetadata = MetricMetadata { name: "operation_errors_total", help: "Error counter during opendal operations", }; /// The metric label for the scheme like s3, fs, cos. pub static LABEL_SCHEME: &str = "scheme"; /// The metric label for the namespace like bucket name in s3. pub static LABEL_NAMESPACE: &str = "namespace"; /// The metric label for the root path. pub static LABEL_ROOT: &str = "root"; /// The metric label for the path used by request. pub static LABEL_PATH: &str = "path"; /// The metric label for the operation like read, write, list. pub static LABEL_OPERATION: &str = "operation"; /// The metric label for the error kind. pub static LABEL_ERROR: &str = "error"; /// The interceptor for metrics. /// /// All metrics related libs should implement this trait to observe opendal's internal operations. pub trait MetricsIntercept: Debug + Clone + Send + Sync + Unpin + 'static { /// Observe the operation duration in seconds. fn observe_operation_duration_seconds( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, duration: Duration, ); /// Observe the operation bytes happened in IO like read and write. fn observe_operation_bytes( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, bytes: usize, ); /// Observe the operation errors total. fn observe_operation_errors_total( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, error: ErrorKind, ); } /// The metrics layer for opendal. #[derive(Clone, Debug)] pub struct MetricsLayer { interceptor: I, } impl MetricsLayer { /// Create a new metrics layer. pub fn new(interceptor: I) -> Self { Self { interceptor } } } impl Layer for MetricsLayer { type LayeredAccess = MetricsAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { let meta = inner.info(); let scheme = meta.scheme(); let name = meta.name().to_string(); let root = meta.root().to_string(); MetricsAccessor { inner: Arc::new(inner), interceptor: self.interceptor.clone(), scheme, namespace: Arc::new(name), root: Arc::new(root), } } } /// The metrics accessor for opendal. #[derive(Clone)] pub struct MetricsAccessor { inner: Arc, interceptor: I, scheme: Scheme, namespace: Arc, root: Arc, } impl Debug for MetricsAccessor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("MetricsAccessor") .field("inner", &self.inner) .finish_non_exhaustive() } } impl LayeredAccess for MetricsAccessor { type Inner = A; type Reader = MetricsWrapper; type BlockingReader = MetricsWrapper; type Writer = MetricsWrapper; type BlockingWriter = MetricsWrapper; type Lister = MetricsWrapper; type BlockingLister = MetricsWrapper; type Deleter = MetricsWrapper; type BlockingDeleter = MetricsWrapper; fn inner(&self) -> &Self::Inner { &self.inner } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { let op = Operation::CreateDir; let start = Instant::now(); self.inner() .create_dir(path, args) .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(move |err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err }) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let op = Operation::Read; let start = Instant::now(); let (rp, reader) = self .inner .read(path, args) .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err })?; Ok(( rp, MetricsWrapper::new( reader, self.interceptor.clone(), self.scheme, self.namespace.clone(), self.root.clone(), path.to_string(), ), )) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let op = Operation::Write; let start = Instant::now(); let (rp, writer) = self .inner .write(path, args) .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err })?; Ok(( rp, MetricsWrapper::new( writer, self.interceptor.clone(), self.scheme, self.namespace.clone(), self.root.clone(), path.to_string(), ), )) } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { let op = Operation::Copy; let start = Instant::now(); self.inner() .copy(from, to, args) .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), from, op, start.elapsed(), ); v }) .map_err(move |err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), from, op, err.kind(), ); err }) } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { let op = Operation::Rename; let start = Instant::now(); self.inner() .rename(from, to, args) .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), from, op, start.elapsed(), ); v }) .map_err(move |err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), from, op, err.kind(), ); err }) } async fn stat(&self, path: &str, args: OpStat) -> Result { let op = Operation::Stat; let start = Instant::now(); self.inner() .stat(path, args) .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(move |err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err }) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { let op = Operation::Delete; let start = Instant::now(); let (rp, writer) = self .inner .delete() .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), "", op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), "", op, err.kind(), ); err })?; Ok(( rp, MetricsWrapper::new( writer, self.interceptor.clone(), self.scheme, self.namespace.clone(), self.root.clone(), "".to_string(), ), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let op = Operation::List; let start = Instant::now(); let (rp, lister) = self .inner .list(path, args) .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err })?; Ok(( rp, MetricsWrapper::new( lister, self.interceptor.clone(), self.scheme, self.namespace.clone(), self.root.clone(), path.to_string(), ), )) } async fn presign(&self, path: &str, args: OpPresign) -> Result { let op = Operation::Presign; let start = Instant::now(); self.inner() .presign(path, args) .await .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(move |err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err }) } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { let op = Operation::BlockingCreateDir; let start = Instant::now(); self.inner() .blocking_create_dir(path, args) .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(move |err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err }) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let op = Operation::BlockingRead; let start = Instant::now(); let (rp, reader) = self .inner .blocking_read(path, args) .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err })?; Ok(( rp, MetricsWrapper::new( reader, self.interceptor.clone(), self.scheme, self.namespace.clone(), self.root.clone(), path.to_string(), ), )) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let op = Operation::BlockingWrite; let start = Instant::now(); let (rp, writer) = self .inner .blocking_write(path, args) .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err })?; Ok(( rp, MetricsWrapper::new( writer, self.interceptor.clone(), self.scheme, self.namespace.clone(), self.root.clone(), path.to_string(), ), )) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { let op = Operation::BlockingCopy; let start = Instant::now(); self.inner() .blocking_copy(from, to, args) .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), from, op, start.elapsed(), ); v }) .map_err(move |err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), from, op, err.kind(), ); err }) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { let op = Operation::BlockingRename; let start = Instant::now(); self.inner() .blocking_rename(from, to, args) .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), from, op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), from, op, err.kind(), ); err }) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { let op = Operation::BlockingStat; let start = Instant::now(); self.inner() .blocking_stat(path, args) .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(move |err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err }) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { let op = Operation::BlockingDelete; let start = Instant::now(); let (rp, writer) = self .inner .blocking_delete() .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), "", op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), "", op, err.kind(), ); err })?; Ok(( rp, MetricsWrapper::new( writer, self.interceptor.clone(), self.scheme, self.namespace.clone(), self.root.clone(), "".to_string(), ), )) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { let op = Operation::BlockingList; let start = Instant::now(); let (rp, lister) = self .inner .blocking_list(path, args) .map(|v| { self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), path, op, start.elapsed(), ); v }) .map_err(|err| { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), path, op, err.kind(), ); err })?; Ok(( rp, MetricsWrapper::new( lister, self.interceptor.clone(), self.scheme, self.namespace.clone(), self.root.clone(), path.to_string(), ), )) } } pub struct MetricsWrapper { inner: R, interceptor: I, scheme: Scheme, namespace: Arc, root: Arc, path: String, } impl MetricsWrapper { fn new( inner: R, interceptor: I, scheme: Scheme, namespace: Arc, root: Arc, path: String, ) -> Self { Self { inner, interceptor, scheme, namespace, root, path, } } } impl oio::Read for MetricsWrapper { async fn read(&mut self) -> Result { let op = Operation::ReaderRead; let start = Instant::now(); let res = match self.inner.read().await { Ok(bs) => { self.interceptor.observe_operation_bytes( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, bs.len(), ); Ok(bs) } Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } } impl oio::BlockingRead for MetricsWrapper { fn read(&mut self) -> Result { let op = Operation::BlockingReaderRead; let start = Instant::now(); let res = match self.inner.read() { Ok(bs) => { self.interceptor.observe_operation_bytes( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, bs.len(), ); Ok(bs) } Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } } impl oio::Write for MetricsWrapper { async fn write(&mut self, bs: Buffer) -> Result<()> { let op = Operation::WriterWrite; let start = Instant::now(); let size = bs.len(); let res = match self.inner.write(bs).await { Ok(()) => { self.interceptor.observe_operation_bytes( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, size, ); Ok(()) } Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } async fn close(&mut self) -> Result { let op = Operation::WriterClose; let start = Instant::now(); let res = match self.inner.close().await { Ok(meta) => Ok(meta), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } async fn abort(&mut self) -> Result<()> { let op = Operation::WriterAbort; let start = Instant::now(); let res = match self.inner.abort().await { Ok(()) => Ok(()), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } } impl oio::BlockingWrite for MetricsWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { let op = Operation::BlockingWriterWrite; let start = Instant::now(); let size = bs.len(); let res = match self.inner.write(bs) { Ok(()) => { self.interceptor.observe_operation_bytes( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, size, ); Ok(()) } Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } fn close(&mut self) -> Result { let op = Operation::BlockingWriterClose; let start = Instant::now(); let res = match self.inner.close() { Ok(meta) => Ok(meta), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } } impl oio::List for MetricsWrapper { async fn next(&mut self) -> Result> { let op = Operation::ListerNext; let start = Instant::now(); let res = match self.inner.next().await { Ok(entry) => Ok(entry), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } } impl oio::BlockingList for MetricsWrapper { fn next(&mut self) -> Result> { let op = Operation::BlockingListerNext; let start = Instant::now(); let res = match self.inner.next() { Ok(entry) => Ok(entry), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } } impl oio::Delete for MetricsWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { let op = Operation::DeleterDelete; let start = Instant::now(); let res = match self.inner.delete(path, args) { Ok(entry) => Ok(entry), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } async fn flush(&mut self) -> Result { let op = Operation::DeleterFlush; let start = Instant::now(); let res = match self.inner.flush().await { Ok(entry) => Ok(entry), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } } impl oio::BlockingDelete for MetricsWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { let op = Operation::BlockingDeleterDelete; let start = Instant::now(); let res = match self.inner.delete(path, args) { Ok(entry) => Ok(entry), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } fn flush(&mut self) -> Result { let op = Operation::BlockingDeleterFlush; let start = Instant::now(); let res = match self.inner.flush() { Ok(entry) => Ok(entry), Err(err) => { self.interceptor.observe_operation_errors_total( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, err.kind(), ); Err(err) } }; self.interceptor.observe_operation_duration_seconds( self.scheme, self.namespace.clone(), self.root.clone(), &self.path, op, start.elapsed(), ); res } } opendal-0.52.0/src/layers/observe/mod.rs000064400000000000000000000071001046102023000162100ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! OpenDAL Observability Layer //! //! This module offers essential components to facilitate the implementation of observability in OpenDAL. //! //! # Prometheus Metrics //! //! These metrics are essential for understanding the behavior and performance of our applications. //! //! | Metric Name | Type | Description | Labels | //! |------------------------------|-----------|--------------------------------------------------------------|-------------------------------------------------| //! | operation_duration_seconds | Histogram | Histogram of time spent during opendal operations | scheme, namespace, root, operation, path | //! | operation_bytes. | Histogram | Histogram of the bytes transferred during opendal operations | scheme, operation, root, operation, path | //! | operation_errors_total | Counter | Error counter during opendal operations | scheme, operation, root, operation, path, error | mod metrics; pub use metrics::MetricMetadata; pub use metrics::MetricsAccessor; pub use metrics::MetricsIntercept; pub use metrics::MetricsLayer; pub use metrics::LABEL_ERROR; pub use metrics::LABEL_NAMESPACE; pub use metrics::LABEL_OPERATION; pub use metrics::LABEL_PATH; pub use metrics::LABEL_ROOT; pub use metrics::LABEL_SCHEME; pub use metrics::METRIC_OPERATION_BYTES; pub use metrics::METRIC_OPERATION_DURATION_SECONDS; pub use metrics::METRIC_OPERATION_ERRORS_TOTAL; /// Return the path label value according to the given `path` and `level`. /// /// - level = 0: return `None`, which means we ignore the path label. /// - level > 0: the path label will be the path split by "/" and get the last n level, /// if n=1 and input path is "abc/def/ghi", and then we'll use "abc/" as the path label. pub fn path_label_value(path: &str, level: usize) -> Option<&str> { if level > 0 { if path.is_empty() { return Some(""); } let label_value = path .char_indices() .filter(|&(_, c)| c == '/') .nth(level - 1) .map_or(path, |(i, _)| &path[..i]); Some(label_value) } else { None } } #[cfg(test)] mod tests { use super::*; #[test] fn test_path_label_value() { let path = "abc/def/ghi"; assert_eq!(path_label_value(path, 0), None); assert_eq!(path_label_value(path, 1), Some("abc")); assert_eq!(path_label_value(path, 2), Some("abc/def")); assert_eq!(path_label_value(path, 3), Some("abc/def/ghi")); assert_eq!(path_label_value(path, usize::MAX), Some("abc/def/ghi")); assert_eq!(path_label_value("", 0), None); assert_eq!(path_label_value("", 1), Some("")); } } opendal-0.52.0/src/layers/otelmetrics.rs000064400000000000000000000245041046102023000163250ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use std::time::Duration; use opentelemetry::metrics::Counter; use opentelemetry::metrics::Histogram; use opentelemetry::metrics::Meter; use opentelemetry::KeyValue; use crate::layers::observe; use crate::raw::*; use crate::*; /// Add [opentelemetry::metrics](https://docs.rs/opentelemetry/latest/opentelemetry/metrics/index.html) for every operation. /// /// # Examples /// /// ```no_run /// # use opendal::layers::OtelMetricsLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # fn main() -> Result<()> { /// let meter = opentelemetry::global::meter("opendal"); /// let _ = Operator::new(services::Memory::default())? /// .layer(OtelMetricsLayer::builder().register(&meter)) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Clone, Debug)] pub struct OtelMetricsLayer { interceptor: OtelMetricsInterceptor, } impl OtelMetricsLayer { /// Create a [`OtelMetricsLayerBuilder`] to set the configuration of metrics. /// /// # Default Configuration /// /// - `path_label`: `0` /// /// # Examples /// /// ```no_run /// # use opendal::layers::OtelMetricsLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// let meter = opentelemetry::global::meter("opendal"); /// let op = Operator::new(services::Memory::default())? /// .layer(OtelMetricsLayer::builder().path_label(1).register(&meter)) /// .finish(); /// /// Ok(()) /// # } /// ``` pub fn builder() -> OtelMetricsLayerBuilder { OtelMetricsLayerBuilder::new() } } /// [`OtelMetricsLayerBuilder`] is a config builder to build a [`OtelMetricsLayer`]. pub struct OtelMetricsLayerBuilder { operation_duration_seconds_boundaries: Vec, operation_bytes_boundaries: Vec, path_label_level: usize, } impl OtelMetricsLayerBuilder { fn new() -> Self { Self { operation_duration_seconds_boundaries: exponential_boundary(0.01, 2.0, 16), operation_bytes_boundaries: exponential_boundary(1.0, 2.0, 16), path_label_level: 0, } } /// Set the level of path label. /// /// - level = 0: we will ignore the path label. /// - level > 0: the path label will be the path split by "/" and get the last n level, /// if n=1 and input path is "abc/def/ghi", and then we will get "abc/" as the path label. /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use opendal::layers::OtelMetricsLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// let meter = opentelemetry::global::meter("opendal"); /// let op = Operator::new(services::Memory::default())? /// .layer(OtelMetricsLayer::builder().path_label(1).register(&meter)) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn path_label(mut self, level: usize) -> Self { self.path_label_level = level; self } /// Set boundaries for `operation_duration_seconds` histogram. /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use opendal::layers::OtelMetricsLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// let meter = opentelemetry::global::meter("opendal"); /// let op = Operator::new(services::Memory::default())? /// .layer( /// OtelMetricsLayer::builder() /// .operation_duration_seconds_boundaries(vec![0.01, 0.02, 0.05, 0.1, 0.2, 0.5]) /// .register(&meter) /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn operation_duration_seconds_boundaries(mut self, boundaries: Vec) -> Self { if !boundaries.is_empty() { self.operation_duration_seconds_boundaries = boundaries; } self } /// Set boundaries for `operation_bytes` histogram. /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use opendal::layers::OtelMetricsLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// let meter = opentelemetry::global::meter("opendal"); /// let op = Operator::new(services::Memory::default())? /// .layer( /// OtelMetricsLayer::builder() /// .operation_bytes_boundaries(vec![1.0, 2.0, 5.0, 10.0, 20.0, 50.0]) /// .register(&meter) /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn operation_bytes_boundaries(mut self, boundaries: Vec) -> Self { if !boundaries.is_empty() { self.operation_bytes_boundaries = boundaries; } self } /// Register the metrics and return a [`OtelMetricsLayer`]. /// /// # Examples /// /// ```no_run /// # use opendal::layers::OtelMetricsLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// let meter = opentelemetry::global::meter("opendal"); /// let op = Operator::new(services::Memory::default())? /// .layer(OtelMetricsLayer::builder().register(&meter)) /// .finish(); /// /// Ok(()) /// # } /// ``` pub fn register(self, meter: &Meter) -> OtelMetricsLayer { let duration_seconds = meter .f64_histogram("opendal.operation.duration") .with_description("Duration of operations") .with_unit("second") .with_boundaries(self.operation_duration_seconds_boundaries) .build(); let bytes = meter .u64_histogram("opendal.operation.size") .with_description("Size of operations") .with_unit("byte") .with_boundaries(self.operation_bytes_boundaries) .build(); let errors = meter .u64_counter("opendal.operation.errors") .with_description("Number of operation errors") .build(); OtelMetricsLayer { interceptor: OtelMetricsInterceptor { duration_seconds, bytes, errors, path_label_level: self.path_label_level, }, } } } impl Layer for OtelMetricsLayer { type LayeredAccess = observe::MetricsAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { observe::MetricsLayer::new(self.interceptor.clone()).layer(inner) } } #[derive(Clone, Debug)] pub struct OtelMetricsInterceptor { duration_seconds: Histogram, bytes: Histogram, errors: Counter, path_label_level: usize, } impl observe::MetricsIntercept for OtelMetricsInterceptor { fn observe_operation_duration_seconds( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, duration: Duration, ) { let attributes = self.create_attributes(scheme, namespace, root, path, op, None); self.duration_seconds .record(duration.as_secs_f64(), &attributes); } fn observe_operation_bytes( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, bytes: usize, ) { let attributes = self.create_attributes(scheme, namespace, root, path, op, None); self.bytes.record(bytes as u64, &attributes); } fn observe_operation_errors_total( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, error: ErrorKind, ) { let attributes = self.create_attributes(scheme, namespace, root, path, op, Some(error)); self.errors.add(1, &attributes); } } impl OtelMetricsInterceptor { fn create_attributes( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, operation: Operation, error: Option, ) -> Vec { let mut attributes = Vec::with_capacity(6); attributes.extend([ KeyValue::new(observe::LABEL_SCHEME, scheme.into_static()), KeyValue::new(observe::LABEL_NAMESPACE, (*namespace).clone()), KeyValue::new(observe::LABEL_ROOT, (*root).clone()), KeyValue::new(observe::LABEL_OPERATION, operation.into_static()), ]); if let Some(path) = observe::path_label_value(path, self.path_label_level) { attributes.push(KeyValue::new(observe::LABEL_PATH, path.to_owned())); } if let Some(error) = error { attributes.push(KeyValue::new(observe::LABEL_ERROR, error.into_static())); } attributes } } fn exponential_boundary(start: f64, factor: f64, count: usize) -> Vec { let mut boundaries = Vec::with_capacity(count); let mut current = start; for _ in 0..count { boundaries.push(current); current *= factor; } boundaries } opendal-0.52.0/src/layers/oteltrace.rs000064400000000000000000000262551046102023000157620ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use std::sync::Arc; use opentelemetry::global; use opentelemetry::global::BoxedSpan; use opentelemetry::trace::FutureExt as TraceFutureExt; use opentelemetry::trace::Span; use opentelemetry::trace::TraceContextExt; use opentelemetry::trace::Tracer; use opentelemetry::Context as TraceContext; use opentelemetry::KeyValue; use crate::raw::*; use crate::*; /// Add [opentelemetry::trace](https://docs.rs/opentelemetry/latest/opentelemetry/trace/index.html) for every operation. /// /// Examples /// /// ## Basic Setup /// /// ```no_run /// # use opendal::layers::OtelTraceLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(OtelTraceLayer) /// .finish(); /// Ok(()) /// # } /// ``` pub struct OtelTraceLayer; impl Layer for OtelTraceLayer { type LayeredAccess = OtelTraceAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { OtelTraceAccessor { inner } } } #[derive(Debug)] pub struct OtelTraceAccessor { inner: A, } impl LayeredAccess for OtelTraceAccessor { type Inner = A; type Reader = OtelTraceWrapper; type BlockingReader = OtelTraceWrapper; type Writer = OtelTraceWrapper; type BlockingWriter = OtelTraceWrapper; type Lister = OtelTraceWrapper; type BlockingLister = OtelTraceWrapper; type Deleter = A::Deleter; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } fn info(&self) -> Arc { let tracer = global::tracer("opendal"); tracer.in_span("info", |_cx| self.inner.info()) } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { let tracer = global::tracer("opendal"); let mut span = tracer.start("create"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); let cx = TraceContext::current_with_span(span); self.inner.create_dir(path, args).with_context(cx).await } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let tracer = global::tracer("opendal"); let mut span = tracer.start("read"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner .read(path, args) .await .map(|(rp, r)| (rp, OtelTraceWrapper::new(span, r))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let tracer = global::tracer("opendal"); let mut span = tracer.start("write"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner .write(path, args) .await .map(|(rp, r)| (rp, OtelTraceWrapper::new(span, r))) } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { let tracer = global::tracer("opendal"); let mut span = tracer.start("copy"); span.set_attribute(KeyValue::new("from", from.to_string())); span.set_attribute(KeyValue::new("to", to.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); let cx = TraceContext::current_with_span(span); self.inner().copy(from, to, args).with_context(cx).await } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { let tracer = global::tracer("opendal"); let mut span = tracer.start("rename"); span.set_attribute(KeyValue::new("from", from.to_string())); span.set_attribute(KeyValue::new("to", to.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); let cx = TraceContext::current_with_span(span); self.inner().rename(from, to, args).with_context(cx).await } async fn stat(&self, path: &str, args: OpStat) -> Result { let tracer = global::tracer("opendal"); let mut span = tracer.start("stat"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); let cx = TraceContext::current_with_span(span); self.inner().stat(path, args).with_context(cx).await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner().delete().await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let tracer = global::tracer("opendal"); let mut span = tracer.start("list"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner .list(path, args) .await .map(|(rp, s)| (rp, OtelTraceWrapper::new(span, s))) } async fn presign(&self, path: &str, args: OpPresign) -> Result { let tracer = global::tracer("opendal"); let mut span = tracer.start("presign"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); let cx = TraceContext::current_with_span(span); self.inner().presign(path, args).with_context(cx).await } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { let tracer = global::tracer("opendal"); tracer.in_span("blocking_create_dir", |cx| { let span = cx.span(); // let mut span = cx.(); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner().blocking_create_dir(path, args) }) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let tracer = global::tracer("opendal"); let mut span = tracer.start("blocking_read"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, OtelTraceWrapper::new(span, r))) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let tracer = global::tracer("opendal"); let mut span = tracer.start("blocking_write"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner .blocking_write(path, args) .map(|(rp, r)| (rp, OtelTraceWrapper::new(span, r))) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { let tracer = global::tracer("opendal"); tracer.in_span("blocking_copy", |cx| { let span = cx.span(); span.set_attribute(KeyValue::new("from", from.to_string())); span.set_attribute(KeyValue::new("to", to.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner().blocking_copy(from, to, args) }) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { let tracer = global::tracer("opendal"); tracer.in_span("blocking_rename", |cx| { let span = cx.span(); span.set_attribute(KeyValue::new("from", from.to_string())); span.set_attribute(KeyValue::new("to", to.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner().blocking_rename(from, to, args) }) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { let tracer = global::tracer("opendal"); tracer.in_span("blocking_stat", |cx| { let span = cx.span(); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner().blocking_stat(path, args) }) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner().blocking_delete() } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { let tracer = global::tracer("opendal"); let mut span = tracer.start("blocking_list"); span.set_attribute(KeyValue::new("path", path.to_string())); span.set_attribute(KeyValue::new("args", format!("{:?}", args))); self.inner .blocking_list(path, args) .map(|(rp, it)| (rp, OtelTraceWrapper::new(span, it))) } } pub struct OtelTraceWrapper { _span: BoxedSpan, inner: R, } impl OtelTraceWrapper { fn new(_span: BoxedSpan, inner: R) -> Self { Self { _span, inner } } } impl oio::Read for OtelTraceWrapper { async fn read(&mut self) -> Result { self.inner.read().await } } impl oio::BlockingRead for OtelTraceWrapper { fn read(&mut self) -> Result { self.inner.read() } } impl oio::Write for OtelTraceWrapper { fn write(&mut self, bs: Buffer) -> impl Future> + MaybeSend { self.inner.write(bs) } fn abort(&mut self) -> impl Future> + MaybeSend { self.inner.abort() } fn close(&mut self) -> impl Future> + MaybeSend { self.inner.close() } } impl oio::BlockingWrite for OtelTraceWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { self.inner.write(bs) } fn close(&mut self) -> Result { self.inner.close() } } impl oio::List for OtelTraceWrapper { async fn next(&mut self) -> Result> { self.inner.next().await } } impl oio::BlockingList for OtelTraceWrapper { fn next(&mut self) -> Result> { self.inner.next() } } opendal-0.52.0/src/layers/prometheus.rs000064400000000000000000000403741046102023000161710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use std::time::Duration; use prometheus::core::AtomicU64; use prometheus::core::GenericCounterVec; use prometheus::exponential_buckets; use prometheus::histogram_opts; use prometheus::HistogramVec; use prometheus::Opts; use prometheus::Registry; use crate::layers::observe; use crate::raw::Access; use crate::raw::*; use crate::*; /// Add [prometheus](https://docs.rs/prometheus) for every operation. /// /// # Prometheus Metrics /// /// We provide several metrics, please see the documentation of [`observe`] module. /// For a more detailed explanation of these metrics and how they are used, please refer to the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/). /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use log::info; /// # use opendal::layers::PrometheusLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use prometheus::Encoder; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// let registry = prometheus::default_registry(); /// /// let op = Operator::new(services::Memory::default())? /// .layer( /// PrometheusLayer::builder() /// .register(registry) /// .expect("register metrics successfully"), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// // Write data into object test. /// op.write("test", "Hello, World!").await?; /// // Read data from object. /// let bs = op.read("test").await?; /// info!("content: {}", String::from_utf8_lossy(&bs.to_bytes())); /// /// // Get object metadata. /// let meta = op.stat("test").await?; /// info!("meta: {:?}", meta); /// /// // Export prometheus metrics. /// let mut buffer = Vec::::new(); /// let encoder = prometheus::TextEncoder::new(); /// encoder.encode(&prometheus::gather(), &mut buffer).unwrap(); /// println!("## Prometheus Metrics"); /// println!("{}", String::from_utf8(buffer.clone()).unwrap()); /// /// Ok(()) /// # } /// ``` #[derive(Clone, Debug)] pub struct PrometheusLayer { interceptor: PrometheusInterceptor, } impl PrometheusLayer { /// Create a [`PrometheusLayerBuilder`] to set the configuration of metrics. /// /// # Default Configuration /// /// - `operation_duration_seconds_buckets`: `exponential_buckets(0.01, 2.0, 16)` /// - `operation_bytes_buckets`: `exponential_buckets(1.0, 2.0, 16)` /// - `path_label`: `0` /// /// # Example /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let registry = prometheus::default_registry(); /// /// let duration_seconds_buckets = prometheus::exponential_buckets(0.01, 2.0, 16).unwrap(); /// let bytes_buckets = prometheus::exponential_buckets(1.0, 2.0, 16).unwrap(); /// let op = Operator::new(builder)? /// .layer( /// PrometheusLayer::builder() /// .operation_duration_seconds_buckets(duration_seconds_buckets) /// .operation_bytes_buckets(bytes_buckets) /// .path_label(0) /// .register(registry) /// .expect("register metrics successfully"), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn builder() -> PrometheusLayerBuilder { let operation_duration_seconds_buckets = exponential_buckets(0.01, 2.0, 16).unwrap(); let operation_bytes_buckets = exponential_buckets(1.0, 2.0, 16).unwrap(); let path_label_level = 0; PrometheusLayerBuilder::new( operation_duration_seconds_buckets, operation_bytes_buckets, path_label_level, ) } } impl Layer for PrometheusLayer { type LayeredAccess = observe::MetricsAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { observe::MetricsLayer::new(self.interceptor.clone()).layer(inner) } } /// [`PrometheusLayerBuilder`] is a config builder to build a [`PrometheusLayer`]. pub struct PrometheusLayerBuilder { operation_duration_seconds_buckets: Vec, operation_bytes_buckets: Vec, path_label_level: usize, } impl PrometheusLayerBuilder { fn new( operation_duration_seconds_buckets: Vec, operation_bytes_buckets: Vec, path_label_level: usize, ) -> Self { Self { operation_duration_seconds_buckets, operation_bytes_buckets, path_label_level, } } /// Set buckets for `operation_duration_seconds` histogram. /// /// # Example /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let registry = prometheus::default_registry(); /// /// let buckets = prometheus::exponential_buckets(0.01, 2.0, 16).unwrap(); /// let op = Operator::new(builder)? /// .layer( /// PrometheusLayer::builder() /// .operation_duration_seconds_buckets(buckets) /// .register(registry) /// .expect("register metrics successfully"), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn operation_duration_seconds_buckets(mut self, buckets: Vec) -> Self { if !buckets.is_empty() { self.operation_duration_seconds_buckets = buckets; } self } /// Set buckets for `operation_bytes` histogram. /// /// # Example /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let registry = prometheus::default_registry(); /// /// let buckets = prometheus::exponential_buckets(1.0, 2.0, 16).unwrap(); /// let op = Operator::new(builder)? /// .layer( /// PrometheusLayer::builder() /// .operation_bytes_buckets(buckets) /// .register(registry) /// .expect("register metrics successfully"), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn operation_bytes_buckets(mut self, buckets: Vec) -> Self { if !buckets.is_empty() { self.operation_bytes_buckets = buckets; } self } /// Set the level of path label. /// /// - level = 0: we will ignore the path label. /// - level > 0: the path label will be the path split by "/" and get the last n level, /// if n=1 and input path is "abc/def/ghi", and then we will get "abc/" as the path label. /// /// # Example /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let registry = prometheus::default_registry(); /// /// let op = Operator::new(builder)? /// .layer( /// PrometheusLayer::builder() /// .path_label(1) /// .register(registry) /// .expect("register metrics successfully"), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn path_label(mut self, level: usize) -> Self { self.path_label_level = level; self } /// Register the metrics into the given registry and return a [`PrometheusLayer`]. /// /// # Example /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let registry = prometheus::default_registry(); /// /// let op = Operator::new(builder)? /// .layer( /// PrometheusLayer::builder() /// .register(registry) /// .expect("register metrics successfully"), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn register(self, registry: &Registry) -> Result { let labels = OperationLabels::names(false, self.path_label_level); let operation_duration_seconds = HistogramVec::new( histogram_opts!( observe::METRIC_OPERATION_DURATION_SECONDS.name(), observe::METRIC_OPERATION_DURATION_SECONDS.help(), self.operation_duration_seconds_buckets ), &labels, ) .map_err(parse_prometheus_error)?; let operation_bytes = HistogramVec::new( histogram_opts!( observe::METRIC_OPERATION_BYTES.name(), observe::METRIC_OPERATION_BYTES.help(), self.operation_bytes_buckets ), &labels, ) .map_err(parse_prometheus_error)?; let labels = OperationLabels::names(true, self.path_label_level); let operation_errors_total = GenericCounterVec::new( Opts::new( observe::METRIC_OPERATION_ERRORS_TOTAL.name(), observe::METRIC_OPERATION_ERRORS_TOTAL.help(), ), &labels, ) .map_err(parse_prometheus_error)?; registry .register(Box::new(operation_duration_seconds.clone())) .map_err(parse_prometheus_error)?; registry .register(Box::new(operation_bytes.clone())) .map_err(parse_prometheus_error)?; registry .register(Box::new(operation_errors_total.clone())) .map_err(parse_prometheus_error)?; Ok(PrometheusLayer { interceptor: PrometheusInterceptor { operation_duration_seconds, operation_bytes, operation_errors_total, path_label_level: self.path_label_level, }, }) } /// Register the metrics into the default registry and return a [`PrometheusLayer`]. /// /// # Example /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// /// let op = Operator::new(builder)? /// .layer( /// PrometheusLayer::builder() /// .register_default() /// .expect("register metrics successfully"), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn register_default(self) -> Result { let registry = prometheus::default_registry(); self.register(registry) } } /// Convert the [`prometheus::Error`] to [`Error`]. fn parse_prometheus_error(err: prometheus::Error) -> Error { Error::new(ErrorKind::Unexpected, err.to_string()).set_source(err) } #[derive(Clone, Debug)] pub struct PrometheusInterceptor { operation_duration_seconds: HistogramVec, operation_bytes: HistogramVec, operation_errors_total: GenericCounterVec, path_label_level: usize, } impl observe::MetricsIntercept for PrometheusInterceptor { fn observe_operation_duration_seconds( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, duration: Duration, ) { let labels = OperationLabels { scheme, namespace: &namespace, root: &root, operation: op, error: None, path, } .into_values(self.path_label_level); self.operation_duration_seconds .with_label_values(&labels) .observe(duration.as_secs_f64()) } fn observe_operation_bytes( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, bytes: usize, ) { let labels = OperationLabels { scheme, namespace: &namespace, root: &root, operation: op, error: None, path, } .into_values(self.path_label_level); self.operation_bytes .with_label_values(&labels) .observe(bytes as f64); } fn observe_operation_errors_total( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, error: ErrorKind, ) { let labels = OperationLabels { scheme, namespace: &namespace, root: &root, operation: op, error: Some(error), path, } .into_values(self.path_label_level); self.operation_errors_total.with_label_values(&labels).inc(); } } struct OperationLabels<'a> { scheme: Scheme, namespace: &'a str, root: &'a str, operation: Operation, path: &'a str, error: Option, } impl<'a> OperationLabels<'a> { fn names(error: bool, path_label_level: usize) -> Vec<&'a str> { let mut names = Vec::with_capacity(6); names.extend([ observe::LABEL_SCHEME, observe::LABEL_NAMESPACE, observe::LABEL_ROOT, observe::LABEL_OPERATION, ]); if path_label_level > 0 { names.push(observe::LABEL_PATH); } if error { names.push(observe::LABEL_ERROR); } names } /// labels: /// /// 1. `["scheme", "namespace", "root", "operation"]` /// 2. `["scheme", "namespace", "root", "operation", "path"]` /// 3. `["scheme", "namespace", "root", "operation", "error"]` /// 4. `["scheme", "namespace", "root", "operation", "path", "error"]` fn into_values(self, path_label_level: usize) -> Vec<&'a str> { let mut labels = Vec::with_capacity(6); labels.extend([ self.scheme.into_static(), self.namespace, self.root, self.operation.into_static(), ]); if let Some(path) = observe::path_label_value(self.path, path_label_level) { labels.push(path); } if let Some(error) = self.error { labels.push(error.into_static()); } labels } } opendal-0.52.0/src/layers/prometheus_client.rs000064400000000000000000000355151046102023000175300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt; use std::sync::Arc; use std::time::Duration; use prometheus_client::encoding::EncodeLabel; use prometheus_client::encoding::EncodeLabelSet; use prometheus_client::encoding::LabelSetEncoder; use prometheus_client::metrics::counter::Counter; use prometheus_client::metrics::family::Family; use prometheus_client::metrics::family::MetricConstructor; use prometheus_client::metrics::histogram::exponential_buckets; use prometheus_client::metrics::histogram::Histogram; use prometheus_client::registry::Registry; use crate::layers::observe; use crate::raw::*; use crate::*; /// Add [prometheus-client](https://docs.rs/prometheus-client) for every operation. /// /// # Prometheus Metrics /// /// We provide several metrics, please see the documentation of [`observe`] module. /// For a more detailed explanation of these metrics and how they are used, please refer to the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/). /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use log::info; /// # use opendal::layers::PrometheusClientLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// let mut registry = prometheus_client::registry::Registry::default(); /// /// let op = Operator::new(services::Memory::default())? /// .layer(PrometheusClientLayer::builder().register(&mut registry)) /// .finish(); /// debug!("operator: {op:?}"); /// /// // Write data into object test. /// op.write("test", "Hello, World!").await?; /// // Read data from object. /// let bs = op.read("test").await?; /// info!("content: {}", String::from_utf8_lossy(&bs.to_bytes())); /// /// // Get object metadata. /// let meta = op.stat("test").await?; /// info!("meta: {:?}", meta); /// /// // Export prometheus metrics. /// let mut buf = String::new(); /// prometheus_client::encoding::text::encode(&mut buf, ®istry).unwrap(); /// println!("## Prometheus Metrics"); /// println!("{}", buf); /// /// Ok(()) /// # } /// ``` #[derive(Clone, Debug)] pub struct PrometheusClientLayer { interceptor: PrometheusClientInterceptor, } impl PrometheusClientLayer { /// Create a [`PrometheusClientLayerBuilder`] to set the configuration of metrics. /// /// # Default Configuration /// /// - `operation_duration_seconds_buckets`: `exponential_buckets(0.01, 2.0, 16)` /// - `operation_bytes_buckets`: `exponential_buckets(1.0, 2.0, 16)` /// - `path_label`: `0` /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusClientLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let mut registry = prometheus_client::registry::Registry::default(); /// /// let duration_seconds_buckets = /// prometheus_client::metrics::histogram::exponential_buckets(0.01, 2.0, 16).collect(); /// let bytes_buckets = /// prometheus_client::metrics::histogram::exponential_buckets(1.0, 2.0, 16).collect(); /// let op = Operator::new(builder)? /// .layer( /// PrometheusClientLayer::builder() /// .operation_duration_seconds_buckets(duration_seconds_buckets) /// .operation_bytes_buckets(bytes_buckets) /// .path_label(0) /// .register(&mut registry), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn builder() -> PrometheusClientLayerBuilder { let operation_duration_seconds_buckets = exponential_buckets(0.01, 2.0, 16).collect(); let operation_bytes_buckets = exponential_buckets(1.0, 2.0, 16).collect(); PrometheusClientLayerBuilder::new( operation_duration_seconds_buckets, operation_bytes_buckets, 0, ) } } impl Layer for PrometheusClientLayer { type LayeredAccess = observe::MetricsAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { observe::MetricsLayer::new(self.interceptor.clone()).layer(inner) } } /// [`PrometheusClientLayerBuilder`] is a config builder to build a [`PrometheusClientLayer`]. pub struct PrometheusClientLayerBuilder { operation_duration_seconds_buckets: Vec, operation_bytes_buckets: Vec, path_label_level: usize, } impl PrometheusClientLayerBuilder { fn new( operation_duration_seconds_buckets: Vec, operation_bytes_buckets: Vec, path_label_level: usize, ) -> Self { Self { operation_duration_seconds_buckets, operation_bytes_buckets, path_label_level, } } /// Set buckets for `operation_duration_seconds` histogram. /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusClientLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let mut registry = prometheus_client::registry::Registry::default(); /// /// let buckets = /// prometheus_client::metrics::histogram::exponential_buckets(0.01, 2.0, 16).collect(); /// let op = Operator::new(builder)? /// .layer( /// PrometheusClientLayer::builder() /// .operation_duration_seconds_buckets(buckets) /// .register(&mut registry), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn operation_duration_seconds_buckets(mut self, buckets: Vec) -> Self { if !buckets.is_empty() { self.operation_duration_seconds_buckets = buckets; } self } /// Set buckets for `operation_bytes` histogram. /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusClientLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let mut registry = prometheus_client::registry::Registry::default(); /// /// let buckets = /// prometheus_client::metrics::histogram::exponential_buckets(1.0, 2.0, 16).collect(); /// let op = Operator::new(builder)? /// .layer( /// PrometheusClientLayer::builder() /// .operation_bytes_buckets(buckets) /// .register(&mut registry), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn operation_bytes_buckets(mut self, buckets: Vec) -> Self { if !buckets.is_empty() { self.operation_bytes_buckets = buckets; } self } /// Set the level of path label. /// /// - level = 0: we will ignore the path label. /// - level > 0: the path label will be the path split by "/" and get the last n level, /// if n=1 and input path is "abc/def/ghi", and then we will get "abc/" as the path label. /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusClientLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let mut registry = prometheus_client::registry::Registry::default(); /// /// let op = Operator::new(builder)? /// .layer( /// PrometheusClientLayer::builder() /// .path_label(1) /// .register(&mut registry), /// ) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn path_label(mut self, level: usize) -> Self { self.path_label_level = level; self } /// Register the metrics into the registry and return a [`PrometheusClientLayer`]. /// /// # Examples /// /// ```no_run /// # use log::debug; /// # use opendal::layers::PrometheusClientLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # #[tokio::main] /// # async fn main() -> Result<()> { /// // Pick a builder and configure it. /// let builder = services::Memory::default(); /// let mut registry = prometheus_client::registry::Registry::default(); /// /// let op = Operator::new(builder)? /// .layer(PrometheusClientLayer::builder().register(&mut registry)) /// .finish(); /// debug!("operator: {op:?}"); /// /// Ok(()) /// # } /// ``` pub fn register(self, registry: &mut Registry) -> PrometheusClientLayer { let operation_duration_seconds = Family::::new_with_constructor(HistogramConstructor { buckets: self.operation_duration_seconds_buckets, }); let operation_bytes = Family::::new_with_constructor(HistogramConstructor { buckets: self.operation_bytes_buckets, }); let operation_errors_total = Family::::default(); registry.register( observe::METRIC_OPERATION_DURATION_SECONDS.name(), observe::METRIC_OPERATION_DURATION_SECONDS.help(), operation_duration_seconds.clone(), ); registry.register( observe::METRIC_OPERATION_BYTES.name(), observe::METRIC_OPERATION_BYTES.help(), operation_bytes.clone(), ); // `prometheus-client` will automatically add `_total` suffix into the name of counter // metrics, so we can't use `METRIC_OPERATION_ERRORS_TOTAL.name()` here. registry.register( "opendal_operation_errors", observe::METRIC_OPERATION_ERRORS_TOTAL.help(), operation_errors_total.clone(), ); PrometheusClientLayer { interceptor: PrometheusClientInterceptor { operation_duration_seconds, operation_bytes, operation_errors_total, path_label_level: self.path_label_level, }, } } } #[derive(Clone)] struct HistogramConstructor { buckets: Vec, } impl MetricConstructor for HistogramConstructor { fn new_metric(&self) -> Histogram { Histogram::new(self.buckets.iter().cloned()) } } #[derive(Clone, Debug)] pub struct PrometheusClientInterceptor { operation_duration_seconds: Family, operation_bytes: Family, operation_errors_total: Family, path_label_level: usize, } impl observe::MetricsIntercept for PrometheusClientInterceptor { fn observe_operation_duration_seconds( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, duration: Duration, ) { self.operation_duration_seconds .get_or_create(&OperationLabels { scheme, namespace, root, operation: op, path: observe::path_label_value(path, self.path_label_level).map(Into::into), error: None, }) .observe(duration.as_secs_f64()) } fn observe_operation_bytes( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, bytes: usize, ) { self.operation_bytes .get_or_create(&OperationLabels { scheme, namespace, root, operation: op, path: observe::path_label_value(path, self.path_label_level).map(Into::into), error: None, }) .observe(bytes as f64) } fn observe_operation_errors_total( &self, scheme: Scheme, namespace: Arc, root: Arc, path: &str, op: Operation, error: ErrorKind, ) { self.operation_errors_total .get_or_create(&OperationLabels { scheme, namespace, root, operation: op, path: observe::path_label_value(path, self.path_label_level).map(Into::into), error: Some(error.into_static()), }) .inc(); } } #[derive(Clone, Debug, Eq, PartialEq, Hash)] struct OperationLabels { scheme: Scheme, namespace: Arc, root: Arc, operation: Operation, path: Option, error: Option<&'static str>, } impl EncodeLabelSet for OperationLabels { fn encode(&self, mut encoder: LabelSetEncoder) -> Result<(), fmt::Error> { (observe::LABEL_SCHEME, self.scheme.into_static()).encode(encoder.encode_label())?; (observe::LABEL_NAMESPACE, self.namespace.as_str()).encode(encoder.encode_label())?; (observe::LABEL_ROOT, self.root.as_str()).encode(encoder.encode_label())?; (observe::LABEL_OPERATION, self.operation.into_static()).encode(encoder.encode_label())?; if let Some(path) = &self.path { (observe::LABEL_PATH, path.as_str()).encode(encoder.encode_label())?; } if let Some(error) = self.error { (observe::LABEL_ERROR, error).encode(encoder.encode_label())?; } Ok(()) } } opendal-0.52.0/src/layers/retry.rs000064400000000000000000001050401046102023000151330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use std::time::Duration; use backon::BlockingRetryable; use backon::ExponentialBuilder; use backon::Retryable; use log::warn; use crate::raw::*; use crate::*; /// Add retry for temporary failed operations. /// /// # Notes /// /// This layer will retry failed operations when [`Error::is_temporary`] /// returns true. If operation still failed, this layer will set error to /// `Persistent` which means error has been retried. /// /// # Panics /// /// While retrying `Reader` or `Writer` operations, please make sure either: /// /// - All futures generated by `Reader::read` or `Writer::close` are resolved to `Ready`. /// - Or, won't call any `Reader` or `Writer` methods after retry returns final error. /// /// Otherwise, `RetryLayer` could panic while hitting in bad states. /// /// For example, while composing `RetryLayer` with `TimeoutLayer`. The order of layer is sensitive. /// /// ```no_run /// # use std::time::Duration; /// /// # use opendal::layers::RetryLayer; /// # use opendal::layers::TimeoutLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # fn main() -> Result<()> { /// let op = Operator::new(services::Memory::default())? /// // This is fine, since timeout happen during retry. /// .layer(TimeoutLayer::new().with_io_timeout(Duration::from_nanos(1))) /// .layer(RetryLayer::new()) /// // This is wrong. Since timeout layer will drop future, leaving retry layer in a bad state. /// .layer(TimeoutLayer::new().with_io_timeout(Duration::from_nanos(1))) /// .finish(); /// Ok(()) /// # } /// ``` /// /// # Examples /// /// ```no_run /// # use opendal::layers::RetryLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(RetryLayer::new()) /// .finish(); /// Ok(()) /// # } /// ``` /// /// ## Customize retry interceptor /// /// RetryLayer accepts [`RetryInterceptor`] to allow users to customize /// their own retry interceptor logic. /// /// ```no_run /// # use std::time::Duration; /// /// # use opendal::layers::RetryInterceptor; /// # use opendal::layers::RetryLayer; /// # use opendal::services; /// # use opendal::Error; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// struct MyRetryInterceptor; /// /// impl RetryInterceptor for MyRetryInterceptor { /// fn intercept(&self, err: &Error, dur: Duration) { /// // do something /// } /// } /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(RetryLayer::new().with_notify(MyRetryInterceptor)) /// .finish(); /// Ok(()) /// # } /// ``` pub struct RetryLayer { builder: ExponentialBuilder, notify: Arc, } impl Clone for RetryLayer { fn clone(&self) -> Self { Self { builder: self.builder, notify: self.notify.clone(), } } } impl Default for RetryLayer { fn default() -> Self { Self { builder: ExponentialBuilder::default(), notify: Arc::new(DefaultRetryInterceptor), } } } impl RetryLayer { /// Create a new retry layer. /// # Examples /// /// ```no_run /// use anyhow::Result; /// use opendal::layers::RetryLayer; /// use opendal::services; /// use opendal::Operator; /// use opendal::Scheme; /// /// let _ = Operator::new(services::Memory::default()) /// .expect("must init") /// .layer(RetryLayer::new()); /// ``` pub fn new() -> Self { Self::default() } /// Set the retry interceptor as new notify. /// /// ```no_run /// use std::time::Duration; /// /// use anyhow::Result; /// use opendal::layers::RetryInterceptor; /// use opendal::layers::RetryLayer; /// use opendal::services; /// use opendal::Error; /// use opendal::Operator; /// use opendal::Scheme; /// /// struct MyRetryInterceptor; /// /// impl RetryInterceptor for MyRetryInterceptor { /// fn intercept(&self, err: &Error, dur: Duration) { /// // do something /// } /// } /// /// let _ = Operator::new(services::Memory::default()) /// .expect("must init") /// .layer(RetryLayer::new().with_notify(MyRetryInterceptor)) /// .finish(); /// ``` pub fn with_notify(self, notify: I) -> RetryLayer { RetryLayer { builder: self.builder, notify: Arc::new(notify), } } /// Set jitter of current backoff. /// /// If jitter is enabled, ExponentialBackoff will add a random jitter in `[0, min_delay) /// to current delay. pub fn with_jitter(mut self) -> Self { self.builder = self.builder.with_jitter(); self } /// Set factor of current backoff. /// /// # Panics /// /// This function will panic if input factor smaller than `1.0`. pub fn with_factor(mut self, factor: f32) -> Self { self.builder = self.builder.with_factor(factor); self } /// Set min_delay of current backoff. pub fn with_min_delay(mut self, min_delay: Duration) -> Self { self.builder = self.builder.with_min_delay(min_delay); self } /// Set max_delay of current backoff. /// /// Delay will not increasing if current delay is larger than max_delay. pub fn with_max_delay(mut self, max_delay: Duration) -> Self { self.builder = self.builder.with_max_delay(max_delay); self } /// Set max_times of current backoff. /// /// Backoff will return `None` if max times is reaching. pub fn with_max_times(mut self, max_times: usize) -> Self { self.builder = self.builder.with_max_times(max_times); self } } impl Layer for RetryLayer { type LayeredAccess = RetryAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { RetryAccessor { inner: Arc::new(inner), builder: self.builder, notify: self.notify.clone(), } } } /// RetryInterceptor is used to intercept while retry happened. pub trait RetryInterceptor: Send + Sync + 'static { /// Everytime RetryLayer is retrying, this function will be called. /// /// # Timing /// /// just before the retry sleep. /// /// # Inputs /// /// - err: The error that caused the current retry. /// - dur: The duration that will sleep before next retry. /// /// # Notes /// /// The intercept must be quick and non-blocking. No heavy IO is /// allowed. Otherwise the retry will be blocked. fn intercept(&self, err: &Error, dur: Duration); } /// The DefaultRetryInterceptor will log the retry error in warning level. pub struct DefaultRetryInterceptor; impl RetryInterceptor for DefaultRetryInterceptor { fn intercept(&self, err: &Error, dur: Duration) { warn!( target: "opendal::layers::retry", "will retry after {}s because: {}", dur.as_secs_f64(), err) } } pub struct RetryAccessor { inner: Arc, builder: ExponentialBuilder, notify: Arc, } impl Debug for RetryAccessor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("RetryAccessor") .field("inner", &self.inner) .finish_non_exhaustive() } } impl LayeredAccess for RetryAccessor { type Inner = A; type Reader = RetryWrapper, I>; type BlockingReader = RetryWrapper, I>; type Writer = RetryWrapper; type BlockingWriter = RetryWrapper; type Lister = RetryWrapper; type BlockingLister = RetryWrapper; type Deleter = RetryWrapper; type BlockingDeleter = RetryWrapper; fn inner(&self) -> &Self::Inner { &self.inner } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { { || self.inner.create_dir(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur: Duration| self.notify.intercept(err, dur)) .await .map_err(|e| e.set_persistent()) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let (rp, reader) = { || self.inner.read(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .await .map_err(|e| e.set_persistent())?; let retry_reader = RetryReader::new(self.inner.clone(), path.to_string(), args, reader); let retry_wrapper = RetryWrapper::new(retry_reader, self.notify.clone(), self.builder); Ok((rp, retry_wrapper)) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { { || self.inner.write(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .await .map(|(rp, r)| (rp, RetryWrapper::new(r, self.notify.clone(), self.builder))) .map_err(|e| e.set_persistent()) } async fn stat(&self, path: &str, args: OpStat) -> Result { { || self.inner.stat(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .await .map_err(|e| e.set_persistent()) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { { || self.inner.delete() } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .await .map(|(rp, r)| (rp, RetryWrapper::new(r, self.notify.clone(), self.builder))) .map_err(|e| e.set_persistent()) } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { { || self.inner.copy(from, to, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .await .map_err(|e| e.set_persistent()) } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { { || self.inner.rename(from, to, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .await .map_err(|e| e.set_persistent()) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { { || self.inner.list(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .await .map(|(rp, r)| (rp, RetryWrapper::new(r, self.notify.clone(), self.builder))) .map_err(|e| e.set_persistent()) } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { { || self.inner.blocking_create_dir(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .call() .map_err(|e| e.set_persistent()) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let (rp, reader) = { || self.inner.blocking_read(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .call() .map_err(|e| e.set_persistent())?; let retry_reader = RetryReader::new(self.inner.clone(), path.to_string(), args, reader); let retry_wrapper = RetryWrapper::new(retry_reader, self.notify.clone(), self.builder); Ok((rp, retry_wrapper)) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { { || self.inner.blocking_write(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .call() .map(|(rp, r)| (rp, RetryWrapper::new(r, self.notify.clone(), self.builder))) .map_err(|e| e.set_persistent()) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { { || self.inner.blocking_stat(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .call() .map_err(|e| e.set_persistent()) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { { || self.inner.blocking_delete() } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .call() .map(|(rp, r)| (rp, RetryWrapper::new(r, self.notify.clone(), self.builder))) .map_err(|e| e.set_persistent()) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { { || self.inner.blocking_copy(from, to, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .call() .map_err(|e| e.set_persistent()) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { { || self.inner.blocking_rename(from, to, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .call() .map_err(|e| e.set_persistent()) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { { || self.inner.blocking_list(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| self.notify.intercept(err, dur)) .call() .map(|(rp, p)| { let p = RetryWrapper::new(p, self.notify.clone(), self.builder); (rp, p) }) .map_err(|e| e.set_persistent()) } } pub struct RetryReader { inner: Arc, reader: Option, path: String, args: OpRead, } impl RetryReader { fn new(inner: Arc, path: String, args: OpRead, r: R) -> Self { Self { inner, reader: Some(r), path, args, } } } impl oio::Read for RetryReader { async fn read(&mut self) -> Result { loop { match self.reader.take() { None => { let (_, r) = self.inner.read(&self.path, self.args.clone()).await?; self.reader = Some(r); continue; } Some(mut reader) => { let buf = reader.read().await?; self.reader = Some(reader); self.args.range_mut().advance(buf.len() as u64); return Ok(buf); } } } } } impl oio::BlockingRead for RetryReader { fn read(&mut self) -> Result { loop { match self.reader.take() { None => { let (_, r) = self.inner.blocking_read(&self.path, self.args.clone())?; self.reader = Some(r); continue; } Some(mut reader) => { let buf = reader.read()?; self.reader = Some(reader); self.args.range_mut().advance(buf.len() as u64); return Ok(buf); } } } } } pub struct RetryWrapper { inner: Option, notify: Arc, builder: ExponentialBuilder, } impl RetryWrapper { fn new(inner: R, notify: Arc, backoff: ExponentialBuilder) -> Self { Self { inner: Some(inner), notify, builder: backoff, } } fn take_inner(&mut self) -> Result { self.inner.take().ok_or_else(|| { Error::new( ErrorKind::Unexpected, "retry layer is in bad state, please make sure future not dropped before ready", ) }) } } impl oio::Read for RetryWrapper { async fn read(&mut self) -> Result { use backon::RetryableWithContext; let inner = self.take_inner()?; let (inner, res) = { |mut r: R| async move { let res = r.read().await; (r, res) } } .retry(self.builder) .when(|e| e.is_temporary()) .context(inner) .notify(|err, dur| self.notify.intercept(err, dur)) .await; self.inner = Some(inner); res.map_err(|err| err.set_persistent()) } } impl oio::BlockingRead for RetryWrapper { fn read(&mut self) -> Result { use backon::BlockingRetryableWithContext; let inner = self.take_inner()?; let (inner, res) = { |mut r: R| { let res = r.read(); (r, res) } } .retry(self.builder) .when(|e| e.is_temporary()) .context(inner) .notify(|err, dur| self.notify.intercept(err, dur)) .call(); self.inner = Some(inner); res.map_err(|err| err.set_persistent()) } } impl oio::Write for RetryWrapper { async fn write(&mut self, bs: Buffer) -> Result<()> { use backon::RetryableWithContext; let inner = self.take_inner()?; let ((inner, _), res) = { |(mut r, bs): (R, Buffer)| async move { let res = r.write(bs.clone()).await; ((r, bs), res) } } .retry(self.builder) .when(|e| e.is_temporary()) .context((inner, bs)) .notify(|err, dur| self.notify.intercept(err, dur)) .await; self.inner = Some(inner); res.map_err(|err| err.set_persistent()) } async fn abort(&mut self) -> Result<()> { use backon::RetryableWithContext; let inner = self.take_inner()?; let (inner, res) = { |mut r: R| async move { let res = r.abort().await; (r, res) } } .retry(self.builder) .when(|e| e.is_temporary()) .context(inner) .notify(|err, dur| self.notify.intercept(err, dur)) .await; self.inner = Some(inner); res.map_err(|err| err.set_persistent()) } async fn close(&mut self) -> Result { use backon::RetryableWithContext; let inner = self.take_inner()?; let (inner, res) = { |mut r: R| async move { let res = r.close().await; (r, res) } } .retry(self.builder) .when(|e| e.is_temporary()) .context(inner) .notify(|err, dur| self.notify.intercept(err, dur)) .await; self.inner = Some(inner); res.map_err(|err| err.set_persistent()) } } impl oio::BlockingWrite for RetryWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { { || self.inner.as_mut().unwrap().write(bs.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| { self.notify.intercept(err, dur); }) .call() .map_err(|e| e.set_persistent()) } fn close(&mut self) -> Result { { || self.inner.as_mut().unwrap().close() } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| { self.notify.intercept(err, dur); }) .call() .map_err(|e| e.set_persistent()) } } impl oio::List for RetryWrapper { async fn next(&mut self) -> Result> { use backon::RetryableWithContext; let inner = self.take_inner()?; let (inner, res) = { |mut p: P| async move { let res = p.next().await; (p, res) } } .retry(self.builder) .when(|e| e.is_temporary()) .context(inner) .notify(|err, dur| self.notify.intercept(err, dur)) .await; self.inner = Some(inner); res.map_err(|err| err.set_persistent()) } } impl oio::BlockingList for RetryWrapper { fn next(&mut self) -> Result> { { || self.inner.as_mut().unwrap().next() } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| { self.notify.intercept(err, dur); }) .call() .map_err(|e| e.set_persistent()) } } impl oio::Delete for RetryWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { { || self.inner.as_mut().unwrap().delete(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| { self.notify.intercept(err, dur); }) .call() .map_err(|e| e.set_persistent()) } async fn flush(&mut self) -> Result { use backon::RetryableWithContext; let inner = self.take_inner()?; let (inner, res) = { |mut p: P| async move { let res = p.flush().await; (p, res) } } .retry(self.builder) .when(|e| e.is_temporary()) .context(inner) .notify(|err, dur| self.notify.intercept(err, dur)) .await; self.inner = Some(inner); res.map_err(|err| err.set_persistent()) } } impl oio::BlockingDelete for RetryWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { { || self.inner.as_mut().unwrap().delete(path, args.clone()) } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| { self.notify.intercept(err, dur); }) .call() .map_err(|e| e.set_persistent()) } fn flush(&mut self) -> Result { { || self.inner.as_mut().unwrap().flush() } .retry(self.builder) .when(|e| e.is_temporary()) .notify(|err, dur| { self.notify.intercept(err, dur); }) .call() .map_err(|e| e.set_persistent()) } } #[cfg(test)] mod tests { use std::mem; use std::sync::Arc; use std::sync::Mutex; use bytes::Bytes; use futures::{stream, TryStreamExt}; use tracing_subscriber::filter::LevelFilter; use super::*; use crate::layers::LoggingLayer; #[derive(Default, Clone)] struct MockBuilder { attempt: Arc>, } impl Builder for MockBuilder { const SCHEME: Scheme = Scheme::Custom("mock"); type Config = (); fn build(self) -> Result { Ok(MockService { attempt: self.attempt.clone(), }) } } #[derive(Debug, Clone, Default)] struct MockService { attempt: Arc>, } impl Access for MockService { type Reader = MockReader; type Writer = MockWriter; type Lister = MockLister; type Deleter = MockDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Custom("mock")) .set_native_capability(Capability { read: true, write: true, write_can_multi: true, delete: true, delete_max_size: Some(10), stat: true, list: true, list_with_recursive: true, ..Default::default() }); am.into() } async fn stat(&self, _: &str, _: OpStat) -> Result { Ok(RpStat::new( Metadata::new(EntryMode::FILE).with_content_length(13), )) } async fn read(&self, _: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { Ok(( RpRead::new(), MockReader { buf: Bytes::from("Hello, World!").into(), range: args.range(), attempt: self.attempt.clone(), }, )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), MockDeleter { size: 0, attempt: self.attempt.clone(), }, )) } async fn write(&self, _: &str, _: OpWrite) -> Result<(RpWrite, Self::Writer)> { Ok((RpWrite::new(), MockWriter {})) } async fn list(&self, _: &str, _: OpList) -> Result<(RpList, Self::Lister)> { let lister = MockLister::default(); Ok((RpList::default(), lister)) } } #[derive(Debug, Clone, Default)] struct MockReader { buf: Buffer, range: BytesRange, attempt: Arc>, } impl oio::Read for MockReader { async fn read(&mut self) -> Result { let mut attempt = self.attempt.lock().unwrap(); *attempt += 1; match *attempt { 1 => Err( Error::new(ErrorKind::Unexpected, "retryable_error from reader") .set_temporary(), ), 2 => Err( Error::new(ErrorKind::Unexpected, "retryable_error from reader") .set_temporary(), ), // Should read out all data. 3 => Ok(self.buf.slice(self.range.to_range_as_usize())), 4 => Err( Error::new(ErrorKind::Unexpected, "retryable_error from reader") .set_temporary(), ), // Should be empty. 5 => Ok(self.buf.slice(self.range.to_range_as_usize())), _ => unreachable!(), } } } #[derive(Debug, Clone, Default)] struct MockWriter {} impl oio::Write for MockWriter { async fn write(&mut self, _: Buffer) -> Result<()> { Ok(()) } async fn close(&mut self) -> Result { Err(Error::new(ErrorKind::Unexpected, "always close failed").set_temporary()) } async fn abort(&mut self) -> Result<()> { Ok(()) } } #[derive(Debug, Clone, Default)] struct MockLister { attempt: usize, } impl oio::List for MockLister { async fn next(&mut self) -> Result> { self.attempt += 1; match self.attempt { 1 => Err(Error::new( ErrorKind::RateLimited, "retryable rate limited error from lister", ) .set_temporary()), 2 => Ok(Some(oio::Entry::new( "hello", Metadata::new(EntryMode::FILE), ))), 3 => Ok(Some(oio::Entry::new( "world", Metadata::new(EntryMode::FILE), ))), 4 => Err( Error::new(ErrorKind::Unexpected, "retryable internal server error") .set_temporary(), ), 5 => Ok(Some(oio::Entry::new( "2023/", Metadata::new(EntryMode::DIR), ))), 6 => Ok(Some(oio::Entry::new( "0208/", Metadata::new(EntryMode::DIR), ))), 7 => Ok(None), _ => { unreachable!() } } } } #[derive(Debug, Clone, Default)] struct MockDeleter { size: usize, attempt: Arc>, } impl oio::Delete for MockDeleter { fn delete(&mut self, _: &str, _: OpDelete) -> Result<()> { self.size += 1; Ok(()) } async fn flush(&mut self) -> Result { let mut attempt = self.attempt.lock().unwrap(); *attempt += 1; match *attempt { 1 => Err( Error::new(ErrorKind::Unexpected, "retryable_error from deleter") .set_temporary(), ), 2 => { self.size -= 1; Ok(1) } 3 => Err( Error::new(ErrorKind::Unexpected, "retryable_error from deleter") .set_temporary(), ), 4 => Err( Error::new(ErrorKind::Unexpected, "retryable_error from deleter") .set_temporary(), ), 5 => { let s = mem::take(&mut self.size); Ok(s) } _ => unreachable!(), } } } #[tokio::test] async fn test_retry_read() { let _ = tracing_subscriber::fmt() .with_max_level(LevelFilter::TRACE) .with_test_writer() .try_init(); let builder = MockBuilder::default(); let op = Operator::new(builder.clone()) .unwrap() .layer(LoggingLayer::default()) .layer(RetryLayer::new()) .finish(); let r = op.reader("retryable_error").await.unwrap(); let mut content = Vec::new(); let size = r .read_into(&mut content, ..) .await .expect("read must succeed"); assert_eq!(size, 13); assert_eq!(content, "Hello, World!".as_bytes()); // The error is retryable, we should request it 3 times. assert_eq!(*builder.attempt.lock().unwrap(), 5); } /// This test is used to reproduce the panic issue while composing retry layer with timeout layer. #[tokio::test] async fn test_retry_write_fail_on_close() { let _ = tracing_subscriber::fmt() .with_max_level(LevelFilter::TRACE) .with_test_writer() .try_init(); let builder = MockBuilder::default(); let op = Operator::new(builder.clone()) .unwrap() .layer( RetryLayer::new() .with_min_delay(Duration::from_millis(1)) .with_max_delay(Duration::from_millis(1)) .with_jitter(), ) // Uncomment this to reproduce timeout layer panic. // .layer(TimeoutLayer::new().with_io_timeout(Duration::from_nanos(1))) .layer(LoggingLayer::default()) .finish(); let mut w = op.writer("test_write").await.unwrap(); w.write("aaa").await.unwrap(); w.write("bbb").await.unwrap(); match w.close().await { Ok(_) => (), Err(_) => { w.abort().await.unwrap(); } }; } #[tokio::test] async fn test_retry_list() { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let builder = MockBuilder::default(); let op = Operator::new(builder.clone()) .unwrap() .layer(RetryLayer::new()) .finish(); let expected = vec!["hello", "world", "2023/", "0208/"]; let mut lister = op .lister("retryable_error/") .await .expect("service must support list"); let mut actual = Vec::new(); while let Some(obj) = lister.try_next().await.expect("must success") { actual.push(obj.name().to_owned()); } assert_eq!(actual, expected); } #[tokio::test] async fn test_retry_batch() { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let builder = MockBuilder::default(); // set to a lower delay to make it run faster let op = Operator::new(builder.clone()) .unwrap() .layer( RetryLayer::new() .with_min_delay(Duration::from_secs_f32(0.1)) .with_max_times(5), ) .finish(); let paths = vec!["hello", "world", "test", "batch"]; op.delete_stream(stream::iter(paths)).await.unwrap(); assert_eq!(*builder.attempt.lock().unwrap(), 5); } } opendal-0.52.0/src/layers/throttle.rs000064400000000000000000000216201046102023000156340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::num::NonZeroU32; use std::sync::Arc; use std::thread; use governor::clock::Clock; use governor::clock::DefaultClock; use governor::middleware::NoOpMiddleware; use governor::state::InMemoryState; use governor::state::NotKeyed; use governor::Quota; use governor::RateLimiter; use crate::raw::*; use crate::*; /// Add a bandwidth rate limiter to the underlying services. /// /// # Throttle /// /// There are several algorithms when it come to rate limiting techniques. /// This throttle layer uses Generic Cell Rate Algorithm (GCRA) provided by /// [Governor](https://docs.rs/governor/latest/governor/index.html). /// By setting the `bandwidth` and `burst`, we can control the byte flow rate of underlying services. /// /// # Note /// /// When setting the ThrottleLayer, always consider the largest possible operation size as the burst size, /// as **the burst size should be larger than any possible byte length to allow it to pass through**. /// /// Read more about [Quota](https://docs.rs/governor/latest/governor/struct.Quota.html#examples) /// /// # Examples /// /// This example limits bandwidth to 10 KiB/s and burst size to 10 MiB. /// /// ```no_run /// # use opendal::layers::ThrottleLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default()) /// .expect("must init") /// .layer(ThrottleLayer::new(10 * 1024, 10000 * 1024)) /// .finish(); /// Ok(()) /// # } /// ``` #[derive(Clone)] pub struct ThrottleLayer { bandwidth: NonZeroU32, burst: NonZeroU32, } impl ThrottleLayer { /// Create a new `ThrottleLayer` with given bandwidth and burst. /// /// - bandwidth: the maximum number of bytes allowed to pass through per second. /// - burst: the maximum number of bytes allowed to pass through at once. pub fn new(bandwidth: u32, burst: u32) -> Self { assert!(bandwidth > 0); assert!(burst > 0); Self { bandwidth: NonZeroU32::new(bandwidth).unwrap(), burst: NonZeroU32::new(burst).unwrap(), } } } impl Layer for ThrottleLayer { type LayeredAccess = ThrottleAccessor; fn layer(&self, accessor: A) -> Self::LayeredAccess { let rate_limiter = Arc::new(RateLimiter::direct( Quota::per_second(self.bandwidth).allow_burst(self.burst), )); ThrottleAccessor { inner: accessor, rate_limiter, } } } /// Share an atomic RateLimiter instance across all threads in one operator. /// If want to add more observability in the future, replace the default NoOpMiddleware with other middleware types. /// Read more about [Middleware](https://docs.rs/governor/latest/governor/middleware/index.html) type SharedRateLimiter = Arc>; #[derive(Debug, Clone)] pub struct ThrottleAccessor { inner: A, rate_limiter: SharedRateLimiter, } impl LayeredAccess for ThrottleAccessor { type Inner = A; type Reader = ThrottleWrapper; type Writer = ThrottleWrapper; type Lister = A::Lister; type Deleter = A::Deleter; type BlockingReader = ThrottleWrapper; type BlockingWriter = ThrottleWrapper; type BlockingLister = A::BlockingLister; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let limiter = self.rate_limiter.clone(); self.inner .read(path, args) .await .map(|(rp, r)| (rp, ThrottleWrapper::new(r, limiter))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let limiter = self.rate_limiter.clone(); self.inner .write(path, args) .await .map(|(rp, w)| (rp, ThrottleWrapper::new(w, limiter))) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner.delete().await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner.list(path, args).await } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let limiter = self.rate_limiter.clone(); self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, ThrottleWrapper::new(r, limiter))) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let limiter = self.rate_limiter.clone(); self.inner .blocking_write(path, args) .map(|(rp, w)| (rp, ThrottleWrapper::new(w, limiter))) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner.blocking_delete() } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner.blocking_list(path, args) } } pub struct ThrottleWrapper { inner: R, limiter: SharedRateLimiter, } impl ThrottleWrapper { pub fn new(inner: R, rate_limiter: SharedRateLimiter) -> Self { Self { inner, limiter: rate_limiter, } } } impl oio::Read for ThrottleWrapper { async fn read(&mut self) -> Result { self.inner.read().await } } impl oio::BlockingRead for ThrottleWrapper { fn read(&mut self) -> Result { self.inner.read() } } impl oio::Write for ThrottleWrapper { async fn write(&mut self, bs: Buffer) -> Result<()> { let buf_length = NonZeroU32::new(bs.len() as u32).unwrap(); loop { match self.limiter.check_n(buf_length) { Ok(res) => match res { Ok(_) => return self.inner.write(bs).await, // the query is valid but the Decider can not accommodate them. Err(not_until) => { let _ = not_until.wait_time_from(DefaultClock::default().now()); // TODO: Should lock the limiter and wait for the wait_time, or should let other small requests go first? // FIXME: we should sleep here. // tokio::time::sleep(wait_time).await; } }, // the query was invalid as the rate limit parameters can "never" accommodate the number of cells queried for. Err(_) => return Err(Error::new( ErrorKind::RateLimited, "InsufficientCapacity due to burst size being smaller than the request size", )), } } } async fn abort(&mut self) -> Result<()> { self.inner.abort().await } async fn close(&mut self) -> Result { self.inner.close().await } } impl oio::BlockingWrite for ThrottleWrapper { fn write(&mut self, bs: Buffer) -> Result<()> { let buf_length = NonZeroU32::new(bs.len() as u32).unwrap(); loop { match self.limiter.check_n(buf_length) { Ok(res) => match res { Ok(_) => return self.inner.write(bs), // the query is valid but the Decider can not accommodate them. Err(not_until) => { let wait_time = not_until.wait_time_from(DefaultClock::default().now()); thread::sleep(wait_time); } }, // the query was invalid as the rate limit parameters can "never" accommodate the number of cells queried for. Err(_) => return Err(Error::new( ErrorKind::RateLimited, "InsufficientCapacity due to burst size being smaller than the request size", )), } } } fn close(&mut self) -> Result { self.inner.close() } } opendal-0.52.0/src/layers/timeout.rs000064400000000000000000000424441046102023000154640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use std::sync::Arc; use std::time::Duration; use crate::raw::*; use crate::*; /// Add timeout for every operation to avoid slow or unexpected hang operations. /// /// For example, a dead connection could hang a databases sql query. TimeoutLayer /// will break this connection and returns an error so users can handle it by /// retrying or print to users. /// /// # Notes /// /// `TimeoutLayer` treats all operations in two kinds: /// /// - Non IO Operation like `stat`, `delete` they operate on a single file. We control /// them by setting `timeout`. /// - IO Operation like `read`, `Reader::read` and `Writer::write`, they operate on data directly, we /// control them by setting `io_timeout`. /// /// # Default /// /// - timeout: 60 seconds /// - io_timeout: 10 seconds /// /// # Panics /// /// TimeoutLayer will drop the future if the timeout is reached. This might cause the internal state /// of the future to be broken. If underlying future moves ownership into the future, it will be /// dropped and will neven return back. /// /// For example, while using `TimeoutLayer` with `RetryLayer` at the same time, please make sure /// timeout layer showed up before retry layer. /// /// ```no_run /// # use std::time::Duration; /// /// # use opendal::layers::RetryLayer; /// # use opendal::layers::TimeoutLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # fn main() -> Result<()> { /// let op = Operator::new(services::Memory::default())? /// // This is fine, since timeout happen during retry. /// .layer(TimeoutLayer::new().with_io_timeout(Duration::from_nanos(1))) /// .layer(RetryLayer::new()) /// // This is wrong. Since timeout layer will drop future, leaving retry layer in a bad state. /// .layer(TimeoutLayer::new().with_io_timeout(Duration::from_nanos(1))) /// .finish(); /// Ok(()) /// # } /// ``` /// /// # Examples /// /// The following examples will create a timeout layer with 10 seconds timeout for all non-io /// operations, 3 seconds timeout for all io operations. /// /// ```no_run /// # use std::time::Duration; /// /// # use opendal::layers::TimeoutLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// # use opendal::Scheme; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer( /// TimeoutLayer::default() /// .with_timeout(Duration::from_secs(10)) /// .with_io_timeout(Duration::from_secs(3)), /// ) /// .finish(); /// Ok(()) /// # } /// ``` /// /// # Implementation Notes /// /// TimeoutLayer is using [`tokio::time::timeout`] to implement timeout for operations. And IO /// Operations insides `reader`, `writer` will use `Pin>` to track the /// timeout. /// /// This might introduce a bit overhead for IO operations, but it's the only way to implement /// timeout correctly. We used to implement timeout layer in zero cost way that only stores /// a [`std::time::Instant`] and check the timeout by comparing the instant with current time. /// However, it doesn't work for all cases. /// /// For examples, users TCP connection could be in [Busy ESTAB](https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die) state. In this state, no IO event will be emitted. The runtime /// will never poll our future again. From the application side, this future is hanging forever /// until this TCP connection is closed for reaching the linux [net.ipv4.tcp_retries2](https://man7.org/linux/man-pages/man7/tcp.7.html) times. #[derive(Clone)] pub struct TimeoutLayer { timeout: Duration, io_timeout: Duration, } impl Default for TimeoutLayer { fn default() -> Self { Self { timeout: Duration::from_secs(60), io_timeout: Duration::from_secs(10), } } } impl TimeoutLayer { /// Create a new `TimeoutLayer` with default settings. pub fn new() -> Self { Self::default() } /// Set timeout for TimeoutLayer with given value. /// /// This timeout is for all non-io operations like `stat`, `delete`. pub fn with_timeout(mut self, timeout: Duration) -> Self { self.timeout = timeout; self } /// Set io timeout for TimeoutLayer with given value. /// /// This timeout is for all io operations like `read`, `Reader::read` and `Writer::write`. pub fn with_io_timeout(mut self, timeout: Duration) -> Self { self.io_timeout = timeout; self } /// Set speed for TimeoutLayer with given value. /// /// # Notes /// /// The speed should be the lower bound of the IO speed. Set this value too /// large could result in all write operations failing. /// /// # Panics /// /// This function will panic if speed is 0. #[deprecated(note = "with speed is not supported anymore, please use with_io_timeout instead")] pub fn with_speed(self, _: u64) -> Self { self } } impl Layer for TimeoutLayer { type LayeredAccess = TimeoutAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { TimeoutAccessor { inner, timeout: self.timeout, io_timeout: self.io_timeout, } } } #[derive(Debug, Clone)] pub struct TimeoutAccessor { inner: A, timeout: Duration, io_timeout: Duration, } impl TimeoutAccessor { async fn timeout>, T>(&self, op: Operation, fut: F) -> Result { tokio::time::timeout(self.timeout, fut).await.map_err(|_| { Error::new(ErrorKind::Unexpected, "operation timeout reached") .with_operation(op) .with_context("timeout", self.timeout.as_secs_f64().to_string()) .set_temporary() })? } async fn io_timeout>, T>( &self, op: Operation, fut: F, ) -> Result { tokio::time::timeout(self.io_timeout, fut) .await .map_err(|_| { Error::new(ErrorKind::Unexpected, "io timeout reached") .with_operation(op) .with_context("timeout", self.io_timeout.as_secs_f64().to_string()) .set_temporary() })? } } impl LayeredAccess for TimeoutAccessor { type Inner = A; type Reader = TimeoutWrapper; type BlockingReader = A::BlockingReader; type Writer = TimeoutWrapper; type BlockingWriter = A::BlockingWriter; type Lister = TimeoutWrapper; type BlockingLister = A::BlockingLister; type Deleter = TimeoutWrapper; type BlockingDeleter = A::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.timeout(Operation::CreateDir, self.inner.create_dir(path, args)) .await } async fn read(&self, path: &str, mut args: OpRead) -> Result<(RpRead, Self::Reader)> { if let Some(exec) = args.executor().cloned() { args = args.with_executor(Executor::with(TimeoutExecutor::new( exec.into_inner(), self.io_timeout, ))); } self.io_timeout(Operation::Read, self.inner.read(path, args)) .await .map(|(rp, r)| (rp, TimeoutWrapper::new(r, self.io_timeout))) } async fn write(&self, path: &str, mut args: OpWrite) -> Result<(RpWrite, Self::Writer)> { if let Some(exec) = args.executor().cloned() { args = args.with_executor(Executor::with(TimeoutExecutor::new( exec.into_inner(), self.io_timeout, ))); } self.io_timeout(Operation::Write, self.inner.write(path, args)) .await .map(|(rp, r)| (rp, TimeoutWrapper::new(r, self.io_timeout))) } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.timeout(Operation::Copy, self.inner.copy(from, to, args)) .await } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.timeout(Operation::Rename, self.inner.rename(from, to, args)) .await } async fn stat(&self, path: &str, args: OpStat) -> Result { self.timeout(Operation::Stat, self.inner.stat(path, args)) .await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.timeout(Operation::Delete, self.inner.delete()) .await .map(|(rp, r)| (rp, TimeoutWrapper::new(r, self.io_timeout))) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.io_timeout(Operation::List, self.inner.list(path, args)) .await .map(|(rp, r)| (rp, TimeoutWrapper::new(r, self.io_timeout))) } async fn presign(&self, path: &str, args: OpPresign) -> Result { self.timeout(Operation::Presign, self.inner.presign(path, args)) .await } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner.blocking_read(path, args) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner.blocking_write(path, args) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner.blocking_list(path, args) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner.blocking_delete() } } pub struct TimeoutExecutor { exec: Arc, timeout: Duration, } impl TimeoutExecutor { pub fn new(exec: Arc, timeout: Duration) -> Self { Self { exec, timeout } } } impl Execute for TimeoutExecutor { fn execute(&self, f: BoxedStaticFuture<()>) { self.exec.execute(f) } fn timeout(&self) -> Option> { Some(Box::pin(tokio::time::sleep(self.timeout))) } } pub struct TimeoutWrapper { inner: R, timeout: Duration, } impl TimeoutWrapper { fn new(inner: R, timeout: Duration) -> Self { Self { inner, timeout } } #[inline] async fn io_timeout>, T>( timeout: Duration, op: &'static str, fut: F, ) -> Result { tokio::time::timeout(timeout, fut).await.map_err(|_| { Error::new(ErrorKind::Unexpected, "io operation timeout reached") .with_operation(op) .with_context("timeout", timeout.as_secs_f64().to_string()) .set_temporary() })? } } impl oio::Read for TimeoutWrapper { async fn read(&mut self) -> Result { let fut = self.inner.read(); Self::io_timeout(self.timeout, Operation::ReaderRead.into_static(), fut).await } } impl oio::Write for TimeoutWrapper { async fn write(&mut self, bs: Buffer) -> Result<()> { let fut = self.inner.write(bs); Self::io_timeout(self.timeout, Operation::WriterWrite.into_static(), fut).await } async fn close(&mut self) -> Result { let fut = self.inner.close(); Self::io_timeout(self.timeout, Operation::WriterClose.into_static(), fut).await } async fn abort(&mut self) -> Result<()> { let fut = self.inner.abort(); Self::io_timeout(self.timeout, Operation::WriterAbort.into_static(), fut).await } } impl oio::List for TimeoutWrapper { async fn next(&mut self) -> Result> { let fut = self.inner.next(); Self::io_timeout(self.timeout, Operation::ListerNext.into_static(), fut).await } } impl oio::Delete for TimeoutWrapper { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } async fn flush(&mut self) -> Result { let fut = self.inner.flush(); Self::io_timeout(self.timeout, Operation::DeleterFlush.into_static(), fut).await } } #[cfg(test)] mod tests { use std::future::pending; use std::future::Future; use std::sync::Arc; use std::time::Duration; use futures::StreamExt; use tokio::time::sleep; use tokio::time::timeout; use crate::layers::TimeoutLayer; use crate::layers::TypeEraseLayer; use crate::raw::*; use crate::*; #[derive(Debug, Clone, Default)] struct MockService; impl Access for MockService { type Reader = MockReader; type Writer = (); type Lister = MockLister; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type Deleter = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_native_capability(Capability { read: true, delete: true, ..Default::default() }); am.into() } /// This function will build a reader that always return pending. async fn read(&self, _: &str, _: OpRead) -> Result<(RpRead, Self::Reader)> { Ok((RpRead::new(), MockReader)) } /// This function will never return. async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { sleep(Duration::from_secs(u64::MAX)).await; Ok((RpDelete::default(), ())) } async fn list(&self, _: &str, _: OpList) -> Result<(RpList, Self::Lister)> { Ok((RpList::default(), MockLister)) } } #[derive(Debug, Clone, Default)] struct MockReader; impl oio::Read for MockReader { fn read(&mut self) -> impl Future> { pending() } } #[derive(Debug, Clone, Default)] struct MockLister; impl oio::List for MockLister { fn next(&mut self) -> impl Future>> { pending() } } #[tokio::test] async fn test_operation_timeout() { let acc = Arc::new(TypeEraseLayer.layer(MockService)) as Accessor; let op = Operator::from_inner(acc) .layer(TimeoutLayer::new().with_timeout(Duration::from_secs(1))); let fut = async { let res = op.delete("test").await; assert!(res.is_err()); let err = res.unwrap_err(); assert_eq!(err.kind(), ErrorKind::Unexpected); assert!(err.to_string().contains("timeout")) }; timeout(Duration::from_secs(2), fut) .await .expect("this test should not exceed 2 seconds") } #[tokio::test] async fn test_io_timeout() { let acc = Arc::new(TypeEraseLayer.layer(MockService)) as Accessor; let op = Operator::from_inner(acc) .layer(TimeoutLayer::new().with_io_timeout(Duration::from_secs(1))); let reader = op.reader("test").await.unwrap(); let res = reader.read(0..4).await; assert!(res.is_err()); let err = res.unwrap_err(); assert_eq!(err.kind(), ErrorKind::Unexpected); assert!(err.to_string().contains("timeout")) } #[tokio::test] async fn test_list_timeout() { let acc = Arc::new(TypeEraseLayer.layer(MockService)) as Accessor; let op = Operator::from_inner(acc).layer( TimeoutLayer::new() .with_timeout(Duration::from_secs(1)) .with_io_timeout(Duration::from_secs(1)), ); let mut lister = op.lister("test").await.unwrap(); let res = lister.next().await.unwrap(); assert!(res.is_err()); let err = res.unwrap_err(); assert_eq!(err.kind(), ErrorKind::Unexpected); assert!(err.to_string().contains("timeout")) } #[tokio::test] async fn test_list_timeout_raw() { use oio::List; let acc = MockService; let timeout_layer = TimeoutLayer::new() .with_timeout(Duration::from_secs(1)) .with_io_timeout(Duration::from_secs(1)); let timeout_acc = timeout_layer.layer(acc); let (_, mut lister) = Access::list(&timeout_acc, "test", OpList::default()) .await .unwrap(); let res = lister.next().await; assert!(res.is_err()); let err = res.unwrap_err(); assert_eq!(err.kind(), ErrorKind::Unexpected); assert!(err.to_string().contains("timeout")); } } opendal-0.52.0/src/layers/tracing.rs000064400000000000000000000305311046102023000154170ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::future::Future; use std::sync::Arc; use tracing::Span; use crate::raw::*; use crate::*; /// Add [tracing](https://docs.rs/tracing/) for every operation. /// /// # Examples /// /// ## Basic Setup /// /// ```no_run /// # use opendal::layers::TracingLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opendal::Result; /// /// # fn main() -> Result<()> { /// let _ = Operator::new(services::Memory::default())? /// .layer(TracingLayer) /// .finish(); /// Ok(()) /// # } /// ``` /// /// ## Real usage /// /// ```no_run /// # use anyhow::Result; /// # use opendal::layers::TracingLayer; /// # use opendal::services; /// # use opendal::Operator; /// # use opentelemetry::KeyValue; /// # use opentelemetry_sdk::trace; /// # use opentelemetry_sdk::Resource; /// # use tracing_subscriber::prelude::*; /// # use tracing_subscriber::EnvFilter; /// /// # fn main() -> Result<()> { /// use opentelemetry::trace::TracerProvider; /// let tracer_provider = opentelemetry_sdk::trace::SdkTracerProvider::builder() /// .with_simple_exporter(opentelemetry_otlp::SpanExporter::builder().with_tonic().build()?) /// .with_resource(Resource::builder().with_attributes(vec![ /// KeyValue::new("service.name", "opendal_example"), /// ]).build()) /// .build(); /// let tracer = tracer_provider.tracer("opendal_tracer"); /// let opentelemetry = tracing_opentelemetry::layer().with_tracer(tracer); /// /// tracing_subscriber::registry() /// .with(EnvFilter::from_default_env()) /// .with(opentelemetry) /// .try_init()?; /// /// { /// let runtime = tokio::runtime::Runtime::new()?; /// runtime.block_on(async { /// let root = tracing::span!(tracing::Level::INFO, "app_start", work_units = 2); /// let _enter = root.enter(); /// /// let _ = dotenvy::dotenv(); /// let op = Operator::new(services::Memory::default())? /// .layer(TracingLayer) /// .finish(); /// /// op.write("test", "0".repeat(16 * 1024 * 1024).into_bytes()) /// .await?; /// op.stat("test").await?; /// op.read("test").await?; /// Ok::<(), opendal::Error>(()) /// })?; /// } /// /// // Shut down the current tracer provider. /// // This will invoke the shutdown method on all span processors. /// // span processors should export remaining spans before return. /// tracer_provider.shutdown()?; /// /// Ok(()) /// # } /// ``` /// /// # Output /// /// OpenDAL is using [`tracing`](https://docs.rs/tracing/latest/tracing/) for tracing internally. /// /// To enable tracing output, please init one of the subscribers that `tracing` supports. /// /// For example: /// /// ```no_run /// # use tracing::dispatcher; /// # use tracing::Event; /// # use tracing::Metadata; /// # use tracing::span::Attributes; /// # use tracing::span::Id; /// # use tracing::span::Record; /// # use tracing::subscriber::Subscriber; /// /// # pub struct FooSubscriber; /// # impl Subscriber for FooSubscriber { /// # fn enabled(&self, _: &Metadata) -> bool { false } /// # fn new_span(&self, _: &Attributes) -> Id { Id::from_u64(0) } /// # fn record(&self, _: &Id, _: &Record) {} /// # fn record_follows_from(&self, _: &Id, _: &Id) {} /// # fn event(&self, _: &Event) {} /// # fn enter(&self, _: &Id) {} /// # fn exit(&self, _: &Id) {} /// # } /// # impl FooSubscriber { fn new() -> Self { FooSubscriber } } /// /// let my_subscriber = FooSubscriber::new(); /// tracing::subscriber::set_global_default(my_subscriber).expect("setting tracing default failed"); /// ``` /// /// For real-world usage, please take a look at [`tracing-opentelemetry`](https://crates.io/crates/tracing-opentelemetry). pub struct TracingLayer; impl Layer for TracingLayer { type LayeredAccess = TracingAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { TracingAccessor { inner } } } #[derive(Debug)] pub struct TracingAccessor { inner: A, } impl LayeredAccess for TracingAccessor { type Inner = A; type Reader = TracingWrapper; type Writer = TracingWrapper; type Lister = TracingWrapper; type Deleter = TracingWrapper; type BlockingReader = TracingWrapper; type BlockingWriter = TracingWrapper; type BlockingLister = TracingWrapper; type BlockingDeleter = TracingWrapper; fn inner(&self) -> &Self::Inner { &self.inner } #[tracing::instrument(level = "debug")] fn info(&self) -> Arc { self.inner.info() } #[tracing::instrument(level = "debug", skip(self))] async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.inner.create_dir(path, args).await } #[tracing::instrument(level = "debug", skip(self))] async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner .read(path, args) .await .map(|(rp, r)| (rp, TracingWrapper::new(Span::current(), r))) } #[tracing::instrument(level = "debug", skip(self))] async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner .write(path, args) .await .map(|(rp, r)| (rp, TracingWrapper::new(Span::current(), r))) } #[tracing::instrument(level = "debug", skip(self))] async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner().copy(from, to, args).await } #[tracing::instrument(level = "debug", skip(self))] async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner().rename(from, to, args).await } #[tracing::instrument(level = "debug", skip(self))] async fn stat(&self, path: &str, args: OpStat) -> Result { self.inner.stat(path, args).await } #[tracing::instrument(level = "debug", skip(self))] async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner .delete() .await .map(|(rp, r)| (rp, TracingWrapper::new(Span::current(), r))) } #[tracing::instrument(level = "debug", skip(self))] async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner .list(path, args) .await .map(|(rp, s)| (rp, TracingWrapper::new(Span::current(), s))) } #[tracing::instrument(level = "debug", skip(self))] async fn presign(&self, path: &str, args: OpPresign) -> Result { self.inner.presign(path, args).await } #[tracing::instrument(level = "debug", skip(self))] fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.inner.blocking_create_dir(path, args) } #[tracing::instrument(level = "debug", skip(self))] fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, TracingWrapper::new(Span::current(), r))) } #[tracing::instrument(level = "debug", skip(self))] fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner .blocking_write(path, args) .map(|(rp, r)| (rp, TracingWrapper::new(Span::current(), r))) } #[tracing::instrument(level = "debug", skip(self))] fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner().blocking_copy(from, to, args) } #[tracing::instrument(level = "debug", skip(self))] fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner().blocking_rename(from, to, args) } #[tracing::instrument(level = "debug", skip(self))] fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.inner.blocking_stat(path, args) } #[tracing::instrument(level = "debug", skip(self))] fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner .blocking_delete() .map(|(rp, r)| (rp, TracingWrapper::new(Span::current(), r))) } #[tracing::instrument(level = "debug", skip(self))] fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner .blocking_list(path, args) .map(|(rp, it)| (rp, TracingWrapper::new(Span::current(), it))) } } pub struct TracingWrapper { span: Span, inner: R, } impl TracingWrapper { fn new(span: Span, inner: R) -> Self { Self { span, inner } } } impl oio::Read for TracingWrapper { #[tracing::instrument( parent = &self.span, level = "trace", skip_all)] async fn read(&mut self) -> Result { self.inner.read().await } } impl oio::BlockingRead for TracingWrapper { #[tracing::instrument( parent = &self.span, level = "trace", skip_all)] fn read(&mut self) -> Result { self.inner.read() } } impl oio::Write for TracingWrapper { #[tracing::instrument( parent = &self.span, level = "trace", skip_all)] fn write(&mut self, bs: Buffer) -> impl Future> + MaybeSend { self.inner.write(bs) } #[tracing::instrument( parent = &self.span, level = "trace", skip_all)] fn abort(&mut self) -> impl Future> + MaybeSend { self.inner.abort() } #[tracing::instrument( parent = &self.span, level = "trace", skip_all)] fn close(&mut self) -> impl Future> + MaybeSend { self.inner.close() } } impl oio::BlockingWrite for TracingWrapper { #[tracing::instrument( parent = &self.span, level = "trace", skip_all)] fn write(&mut self, bs: Buffer) -> Result<()> { self.inner.write(bs) } #[tracing::instrument( parent = &self.span, level = "trace", skip_all)] fn close(&mut self) -> Result { self.inner.close() } } impl oio::List for TracingWrapper { #[tracing::instrument(parent = &self.span, level = "debug", skip_all)] async fn next(&mut self) -> Result> { self.inner.next().await } } impl oio::BlockingList for TracingWrapper { #[tracing::instrument(parent = &self.span, level = "debug", skip_all)] fn next(&mut self) -> Result> { self.inner.next() } } impl oio::Delete for TracingWrapper { #[tracing::instrument(parent = &self.span, level = "debug", skip_all)] fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } #[tracing::instrument(parent = &self.span, level = "debug", skip_all)] async fn flush(&mut self) -> Result { self.inner.flush().await } } impl oio::BlockingDelete for TracingWrapper { #[tracing::instrument(parent = &self.span, level = "debug", skip_all)] fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.inner.delete(path, args) } #[tracing::instrument(parent = &self.span, level = "debug", skip_all)] fn flush(&mut self) -> Result { self.inner.flush() } } opendal-0.52.0/src/layers/type_eraser.rs000064400000000000000000000074651046102023000163240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; use std::fmt::Debug; use std::fmt::Formatter; /// TypeEraseLayer will erase the types on internal accessor. /// /// For example, we will erase `Self::Reader` to `oio::Reader` (`Box`). /// /// # Notes /// /// TypeEraseLayer is not a public accessible layer that can be used by /// external users. We use this layer to erase any generic types. pub struct TypeEraseLayer; impl Layer for TypeEraseLayer { type LayeredAccess = TypeEraseAccessor; fn layer(&self, inner: A) -> Self::LayeredAccess { TypeEraseAccessor { inner } } } /// Provide reader wrapper for backend. pub struct TypeEraseAccessor { inner: A, } impl Debug for TypeEraseAccessor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { self.inner.fmt(f) } } impl LayeredAccess for TypeEraseAccessor { type Inner = A; type Reader = oio::Reader; type Writer = oio::Writer; type Lister = oio::Lister; type Deleter = oio::Deleter; type BlockingReader = oio::BlockingReader; type BlockingWriter = oio::BlockingWriter; type BlockingLister = oio::BlockingLister; type BlockingDeleter = oio::BlockingDeleter; fn inner(&self) -> &Self::Inner { &self.inner } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.inner .read(path, args) .await .map(|(rp, r)| (rp, Box::new(r) as oio::Reader)) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.inner .write(path, args) .await .map(|(rp, w)| (rp, Box::new(w) as oio::Writer)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.inner .delete() .await .map(|(rp, p)| (rp, Box::new(p) as oio::Deleter)) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.inner .list(path, args) .await .map(|(rp, p)| (rp, Box::new(p) as oio::Lister)) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.inner .blocking_read(path, args) .map(|(rp, r)| (rp, Box::new(r) as oio::BlockingReader)) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.inner .blocking_write(path, args) .map(|(rp, w)| (rp, Box::new(w) as oio::BlockingWriter)) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.inner .blocking_delete() .map(|(rp, p)| (rp, Box::new(p) as oio::BlockingDeleter)) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.inner .blocking_list(path, args) .map(|(rp, p)| (rp, Box::new(p) as oio::BlockingLister)) } } opendal-0.52.0/src/lib.rs000064400000000000000000000135251046102023000132430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #![doc( html_logo_url = "https://raw.githubusercontent.com/apache/opendal/main/website/static/img/logo.svg" )] #![cfg_attr(docsrs, feature(doc_auto_cfg))] //! Apache OpenDAL™ is an Open Data Access Layer that enables seamless interaction with diverse storage services. //! //! OpenDAL's development is guided by its vision of **One Layer, All Storage** and its core principles: **Open Community**, **Solid Foundation**, **Fast Access**, **Object Storage First**, and **Extensible Architecture**. Read the explained vision at [OpenDAL Vision](https://opendal.apache.org/vision). //! //! # Quick Start //! //! OpenDAL's API entry points are [`Operator`] and [`BlockingOperator`]. All //! public APIs are accessible through the operator. To utilize OpenDAL, you //! need to: //! //! - [Init a service](#init-a-service) //! - [Compose layers](#compose-layers) //! - [Use operator](#use-operator) //! //! ## Init a service //! //! The first step is to pick a service and init it with a builder. All supported //! services could be found at [`services`]. //! //! Let's take [`services::S3`] as an example: //! //! ```no_run //! use opendal::services; //! use opendal::Operator; //! use opendal::Result; //! //! fn main() -> Result<()> { //! // Pick a builder and configure it. //! let mut builder = services::S3::default().bucket("test"); //! //! // Init an operator //! let op = Operator::new(builder)?.finish(); //! Ok(()) //! } //! ``` //! //! ## Compose layers //! //! The next setup is to compose layers. Layers are modules that provide extra //! features for every operation. All builtin layers could be found at [`layers`]. //! //! Let's use [`layers::LoggingLayer`] as an example; this layer adds logging to //! every operation that OpenDAL performs. //! //! ```no_run //! use opendal::layers::LoggingLayer; //! use opendal::services; //! use opendal::Operator; //! use opendal::Result; //! //! #[tokio::main] //! async fn main() -> Result<()> { //! // Pick a builder and configure it. //! let mut builder = services::S3::default().bucket("test"); //! //! // Init an operator //! let op = Operator::new(builder)? //! // Init with logging layer enabled. //! .layer(LoggingLayer::default()) //! .finish(); //! //! Ok(()) //! } //! ``` //! //! ## Use operator //! //! The final step is to use the operator. OpenDAL supports both async [`Operator`] //! and blocking [`BlockingOperator`]. Please pick the one that fits your use case. //! //! Every Operator API follows the same pattern, take `read` as an example: //! //! - `read`: Execute a read operation. //! - `read_with`: Execute a read operation with additional options, like `range` and `if_match`. //! - `reader`: Create a reader for streaming data, enabling flexible access. //! - `reader_with`: Create a reader with advanced options. //! //! ```no_run //! use opendal::layers::LoggingLayer; //! use opendal::services; //! use opendal::Operator; //! use opendal::Result; //! //! #[tokio::main] //! async fn main() -> Result<()> { //! // Pick a builder and configure it. //! let mut builder = services::S3::default().bucket("test"); //! //! // Init an operator //! let op = Operator::new(builder)? //! // Init with logging layer enabled. //! .layer(LoggingLayer::default()) //! .finish(); //! //! // Fetch this file's metadata //! let meta = op.stat("hello.txt").await?; //! let length = meta.content_length(); //! //! // Read data from `hello.txt` with range `0..1024`. //! let bs = op.read_with("hello.txt").range(0..1024).await?; //! //! Ok(()) //! } //! ``` // Make sure all our public APIs have docs. #![warn(missing_docs)] // Private module with public types, they will be accessed via `opendal::Xxxx` mod types; pub use types::*; // Public modules, they will be accessed like `opendal::layers::Xxxx` #[cfg(docsrs)] pub mod docs; pub mod layers; pub mod raw; pub mod services; #[cfg(test)] mod tests { use std::mem::size_of; use super::*; /// This is not a real test case. /// /// We assert our public structs here to make sure we don't introduce /// unexpected struct/enum size change. #[test] fn assert_size() { assert_eq!(32, size_of::()); assert_eq!(320, size_of::()); assert_eq!(296, size_of::()); assert_eq!(1, size_of::()); assert_eq!(24, size_of::()); } trait AssertSendSync: Send + Sync {} impl AssertSendSync for Entry {} impl AssertSendSync for Capability {} impl AssertSendSync for Error {} impl AssertSendSync for Reader {} impl AssertSendSync for Writer {} impl AssertSendSync for Lister {} impl AssertSendSync for Operator {} impl AssertSendSync for BlockingReader {} impl AssertSendSync for BlockingWriter {} impl AssertSendSync for BlockingLister {} impl AssertSendSync for BlockingOperator {} /// This is used to make sure our public API implement Send + Sync #[test] fn test_trait() { let _: Box = Box::new(Capability::default()); } } opendal-0.52.0/src/raw/accessor.rs000064400000000000000000000672741046102023000151020ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::future::ready; use std::sync::Arc; use futures::Future; use crate::raw::*; use crate::*; /// Underlying trait of all backends for implementers. /// /// The actual data access of storage service happens in Accessor layer. /// Every storage supported by OpenDAL must implement [`Access`] but not all /// methods of [`Access`] will be implemented according to how the storage service is. /// /// For example, user can not modify the content from one HTTP file server directly. /// So [`Http`][crate::services::Http] implements and provides only read related actions. /// /// [`Access`] gives default implementation for all methods which will raise [`ErrorKind::Unsupported`] error. /// And what action this [`Access`] supports will be pointed out in [`AccessorInfo`]. /// /// # Note /// /// Visit [`internals`][crate::docs::internals] for more tutorials. /// /// # Operations /// /// - Path in args will all be normalized into the same style, services /// should handle them based on services' requirement. /// - Path that ends with `/` means it's Dir, otherwise, it's File. /// - Root dir is `/` /// - Path will never be empty. /// - Operations without capability requirement like `metadata`, `create` are /// basic operations. /// - All services must implement them. /// - Use `unimplemented!()` if not implemented or can't implement. /// - Operations with capability requirement like `presign` are optional operations. /// - Services can implement them based on services capabilities. /// - The default implementation should return [`ErrorKind::Unsupported`]. pub trait Access: Send + Sync + Debug + Unpin + 'static { /// Reader is the associated reader returned in `read` operation. type Reader: oio::Read; /// Writer is the associated writer returned in `write` operation. type Writer: oio::Write; /// Lister is the associated lister returned in `list` operation. type Lister: oio::List; /// Deleter is the associated deleter returned in `delete` operation. type Deleter: oio::Delete; /// BlockingReader is the associated reader returned `blocking_read` operation. type BlockingReader: oio::BlockingRead; /// BlockingWriter is the associated writer returned `blocking_write` operation. type BlockingWriter: oio::BlockingWrite; /// BlockingLister is the associated lister returned `blocking_list` operation. type BlockingLister: oio::BlockingList; /// BlockingDeleter is the associated deleter returned `blocking_delete` operation. type BlockingDeleter: oio::BlockingDelete; /// Invoke the `info` operation to get metadata of accessor. /// /// # Notes /// /// This function is required to be implemented. /// /// By returning AccessorInfo, underlying services can declare /// some useful information about itself. /// /// - scheme: declare the scheme of backend. /// - capabilities: declare the capabilities of current backend. fn info(&self) -> Arc; /// Invoke the `create` operation on the specified path /// /// Require [`Capability::create_dir`] /// /// # Behavior /// /// - Input path MUST match with EntryMode, DON'T NEED to check mode. /// - Create on existing dir SHOULD succeed. fn create_dir( &self, path: &str, args: OpCreateDir, ) -> impl Future> + MaybeSend { let (_, _) = (path, args); ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `stat` operation on the specified path. /// /// Require [`Capability::stat`] /// /// # Behavior /// /// - `stat` empty path means stat backend's root path. /// - `stat` a path endswith "/" means stating a dir. /// - `mode` and `content_length` must be set. fn stat(&self, path: &str, args: OpStat) -> impl Future> + MaybeSend { let (_, _) = (path, args); ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `read` operation on the specified path, returns a /// [`Reader`][crate::Reader] if operate successful. /// /// Require [`Capability::read`] /// /// # Behavior /// /// - Input path MUST be file path, DON'T NEED to check mode. /// - The returning content length may be smaller than the range specified. fn read( &self, path: &str, args: OpRead, ) -> impl Future> + MaybeSend { let (_, _) = (path, args); ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `write` operation on the specified path, returns a /// written size if operate successful. /// /// Require [`Capability::write`] /// /// # Behavior /// /// - Input path MUST be file path, DON'T NEED to check mode. fn write( &self, path: &str, args: OpWrite, ) -> impl Future> + MaybeSend { let (_, _) = (path, args); ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `delete` operation on the specified path. /// /// Require [`Capability::delete`] /// /// # Behavior /// /// - `delete` is an idempotent operation, it's safe to call `Delete` on the same path multiple times. /// - `delete` SHOULD return `Ok(())` if the path is deleted successfully or not exist. fn delete(&self) -> impl Future> + MaybeSend { ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `list` operation on the specified path. /// /// Require [`Capability::list`] /// /// # Behavior /// /// - Input path MUST be dir path, DON'T NEED to check mode. /// - List non-exist dir should return Empty. fn list( &self, path: &str, args: OpList, ) -> impl Future> + MaybeSend { let (_, _) = (path, args); ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `copy` operation on the specified `from` path and `to` path. /// /// Require [Capability::copy] /// /// # Behaviour /// /// - `from` and `to` MUST be file path, DON'T NEED to check mode. /// - Copy on existing file SHOULD succeed. /// - Copy on existing file SHOULD overwrite and truncate. fn copy( &self, from: &str, to: &str, args: OpCopy, ) -> impl Future> + MaybeSend { let (_, _, _) = (from, to, args); ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `rename` operation on the specified `from` path and `to` path. /// /// Require [Capability::rename] fn rename( &self, from: &str, to: &str, args: OpRename, ) -> impl Future> + MaybeSend { let (_, _, _) = (from, to, args); ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `presign` operation on the specified path. /// /// Require [`Capability::presign`] /// /// # Behavior /// /// - This API is optional, return [`std::io::ErrorKind::Unsupported`] if not supported. fn presign( &self, path: &str, args: OpPresign, ) -> impl Future> + MaybeSend { let (_, _) = (path, args); ready(Err(Error::new( ErrorKind::Unsupported, "operation is not supported", ))) } /// Invoke the `blocking_create` operation on the specified path. /// /// This operation is the blocking version of [`Accessor::create_dir`] /// /// Require [`Capability::create_dir`] and [`Capability::blocking`] fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { let (_, _) = (path, args); Err(Error::new( ErrorKind::Unsupported, "operation is not supported", )) } /// Invoke the `blocking_stat` operation on the specified path. /// /// This operation is the blocking version of [`Accessor::stat`] /// /// Require [`Capability::stat`] and [`Capability::blocking`] fn blocking_stat(&self, path: &str, args: OpStat) -> Result { let (_, _) = (path, args); Err(Error::new( ErrorKind::Unsupported, "operation is not supported", )) } /// Invoke the `blocking_read` operation on the specified path. /// /// This operation is the blocking version of [`Accessor::read`] /// /// Require [`Capability::read`] and [`Capability::blocking`] fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let (_, _) = (path, args); Err(Error::new( ErrorKind::Unsupported, "operation is not supported", )) } /// Invoke the `blocking_write` operation on the specified path. /// /// This operation is the blocking version of [`Accessor::write`] /// /// Require [`Capability::write`] and [`Capability::blocking`] fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let (_, _) = (path, args); Err(Error::new( ErrorKind::Unsupported, "operation is not supported", )) } /// Invoke the `blocking_delete` operation on the specified path. /// /// This operation is the blocking version of [`Accessor::delete`] /// /// Require [`Capability::write`] and [`Capability::blocking`] fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { Err(Error::new( ErrorKind::Unsupported, "operation is not supported", )) } /// Invoke the `blocking_list` operation on the specified path. /// /// This operation is the blocking version of [`Accessor::list`] /// /// Require [`Capability::list`] and [`Capability::blocking`] /// /// # Behavior /// /// - List non-exist dir should return Empty. fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { let (_, _) = (path, args); Err(Error::new( ErrorKind::Unsupported, "operation is not supported", )) } /// Invoke the `blocking_copy` operation on the specified `from` path and `to` path. /// /// This operation is the blocking version of [`Accessor::copy`] /// /// Require [`Capability::copy`] and [`Capability::blocking`] fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { let (_, _, _) = (from, to, args); Err(Error::new( ErrorKind::Unsupported, "operation is not supported", )) } /// Invoke the `blocking_rename` operation on the specified `from` path and `to` path. /// /// This operation is the blocking version of [`Accessor::rename`] /// /// Require [`Capability::rename`] and [`Capability::blocking`] fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { let (_, _, _) = (from, to, args); Err(Error::new( ErrorKind::Unsupported, "operation is not supported", )) } } /// `AccessDyn` is the dyn version of [`Access`] make it possible to use as /// `Box`. pub trait AccessDyn: Send + Sync + Debug + Unpin { /// Dyn version of [`Accessor::info`] fn info_dyn(&self) -> Arc; /// Dyn version of [`Accessor::create_dir`] fn create_dir_dyn<'a>( &'a self, path: &'a str, args: OpCreateDir, ) -> BoxedFuture<'a, Result>; /// Dyn version of [`Accessor::stat`] fn stat_dyn<'a>(&'a self, path: &'a str, args: OpStat) -> BoxedFuture<'a, Result>; /// Dyn version of [`Accessor::read`] fn read_dyn<'a>( &'a self, path: &'a str, args: OpRead, ) -> BoxedFuture<'a, Result<(RpRead, oio::Reader)>>; /// Dyn version of [`Accessor::write`] fn write_dyn<'a>( &'a self, path: &'a str, args: OpWrite, ) -> BoxedFuture<'a, Result<(RpWrite, oio::Writer)>>; /// Dyn version of [`Accessor::delete`] fn delete_dyn(&self) -> BoxedFuture>; /// Dyn version of [`Accessor::list`] fn list_dyn<'a>( &'a self, path: &'a str, args: OpList, ) -> BoxedFuture<'a, Result<(RpList, oio::Lister)>>; /// Dyn version of [`Accessor::copy`] fn copy_dyn<'a>( &'a self, from: &'a str, to: &'a str, args: OpCopy, ) -> BoxedFuture<'a, Result>; /// Dyn version of [`Accessor::rename`] fn rename_dyn<'a>( &'a self, from: &'a str, to: &'a str, args: OpRename, ) -> BoxedFuture<'a, Result>; /// Dyn version of [`Accessor::presign`] fn presign_dyn<'a>( &'a self, path: &'a str, args: OpPresign, ) -> BoxedFuture<'a, Result>; /// Dyn version of [`Accessor::blocking_create_dir`] fn blocking_create_dir_dyn(&self, path: &str, args: OpCreateDir) -> Result; /// Dyn version of [`Accessor::blocking_stat`] fn blocking_stat_dyn(&self, path: &str, args: OpStat) -> Result; /// Dyn version of [`Accessor::blocking_read`] fn blocking_read_dyn(&self, path: &str, args: OpRead) -> Result<(RpRead, oio::BlockingReader)>; /// Dyn version of [`Accessor::blocking_write`] fn blocking_write_dyn( &self, path: &str, args: OpWrite, ) -> Result<(RpWrite, oio::BlockingWriter)>; /// Dyn version of [`Accessor::blocking_delete`] fn blocking_delete_dyn(&self) -> Result<(RpDelete, oio::BlockingDeleter)>; /// Dyn version of [`Accessor::blocking_list`] fn blocking_list_dyn(&self, path: &str, args: OpList) -> Result<(RpList, oio::BlockingLister)>; /// Dyn version of [`Accessor::blocking_copy`] fn blocking_copy_dyn(&self, from: &str, to: &str, args: OpCopy) -> Result; /// Dyn version of [`Accessor::blocking_rename`] fn blocking_rename_dyn(&self, from: &str, to: &str, args: OpRename) -> Result; } impl AccessDyn for A where A: Access< Reader = oio::Reader, BlockingReader = oio::BlockingReader, Writer = oio::Writer, BlockingWriter = oio::BlockingWriter, Lister = oio::Lister, BlockingLister = oio::BlockingLister, Deleter = oio::Deleter, BlockingDeleter = oio::BlockingDeleter, >, { fn info_dyn(&self) -> Arc { self.info() } fn create_dir_dyn<'a>( &'a self, path: &'a str, args: OpCreateDir, ) -> BoxedFuture<'a, Result> { Box::pin(self.create_dir(path, args)) } fn stat_dyn<'a>(&'a self, path: &'a str, args: OpStat) -> BoxedFuture<'a, Result> { Box::pin(self.stat(path, args)) } fn read_dyn<'a>( &'a self, path: &'a str, args: OpRead, ) -> BoxedFuture<'a, Result<(RpRead, oio::Reader)>> { Box::pin(self.read(path, args)) } fn write_dyn<'a>( &'a self, path: &'a str, args: OpWrite, ) -> BoxedFuture<'a, Result<(RpWrite, oio::Writer)>> { Box::pin(self.write(path, args)) } fn delete_dyn(&self) -> BoxedFuture> { Box::pin(self.delete()) } fn list_dyn<'a>( &'a self, path: &'a str, args: OpList, ) -> BoxedFuture<'a, Result<(RpList, oio::Lister)>> { Box::pin(self.list(path, args)) } fn copy_dyn<'a>( &'a self, from: &'a str, to: &'a str, args: OpCopy, ) -> BoxedFuture<'a, Result> { Box::pin(self.copy(from, to, args)) } fn rename_dyn<'a>( &'a self, from: &'a str, to: &'a str, args: OpRename, ) -> BoxedFuture<'a, Result> { Box::pin(self.rename(from, to, args)) } fn presign_dyn<'a>( &'a self, path: &'a str, args: OpPresign, ) -> BoxedFuture<'a, Result> { Box::pin(self.presign(path, args)) } fn blocking_create_dir_dyn(&self, path: &str, args: OpCreateDir) -> Result { self.blocking_create_dir(path, args) } fn blocking_stat_dyn(&self, path: &str, args: OpStat) -> Result { self.blocking_stat(path, args) } fn blocking_read_dyn(&self, path: &str, args: OpRead) -> Result<(RpRead, oio::BlockingReader)> { self.blocking_read(path, args) } fn blocking_write_dyn( &self, path: &str, args: OpWrite, ) -> Result<(RpWrite, oio::BlockingWriter)> { self.blocking_write(path, args) } fn blocking_delete_dyn(&self) -> Result<(RpDelete, oio::BlockingDeleter)> { self.blocking_delete() } fn blocking_list_dyn(&self, path: &str, args: OpList) -> Result<(RpList, oio::BlockingLister)> { self.blocking_list(path, args) } fn blocking_copy_dyn(&self, from: &str, to: &str, args: OpCopy) -> Result { self.blocking_copy(from, to, args) } fn blocking_rename_dyn(&self, from: &str, to: &str, args: OpRename) -> Result { self.blocking_rename(from, to, args) } } impl Access for dyn AccessDyn { type Reader = oio::Reader; type BlockingReader = oio::BlockingReader; type Writer = oio::Writer; type Deleter = oio::Deleter; type BlockingWriter = oio::BlockingWriter; type Lister = oio::Lister; type BlockingLister = oio::BlockingLister; type BlockingDeleter = oio::BlockingDeleter; fn info(&self) -> Arc { self.info_dyn() } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.create_dir_dyn(path, args).await } async fn stat(&self, path: &str, args: OpStat) -> Result { self.stat_dyn(path, args).await } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { self.read_dyn(path, args).await } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.write_dyn(path, args).await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { self.delete_dyn().await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { self.list_dyn(path, args).await } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.copy_dyn(from, to, args).await } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.rename_dyn(from, to, args).await } async fn presign(&self, path: &str, args: OpPresign) -> Result { self.presign_dyn(path, args).await } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.blocking_create_dir_dyn(path, args) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.blocking_read_dyn(path, args) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.blocking_stat_dyn(path, args) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.blocking_write_dyn(path, args) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.blocking_delete_dyn() } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.blocking_list_dyn(path, args) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.blocking_copy_dyn(from, to, args) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.blocking_rename_dyn(from, to, args) } } /// Dummy implementation of accessor. impl Access for () { type Reader = (); type Writer = (); type Lister = (); type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { AccessorInfo { scheme: Scheme::Custom("dummy"), root: "".to_string(), name: "dummy".to_string(), native_capability: Capability::default(), full_capability: Capability::default(), } .into() } } /// All functions in `Accessor` only requires `&self`, so it's safe to implement /// `Accessor` for `Arc`. // If we use async fn directly, some weird higher rank trait bound error (`Send`/`Accessor` impl not general enough) will happen. // Probably related to https://github.com/rust-lang/rust/issues/96865 #[allow(clippy::manual_async_fn)] impl Access for Arc { type Reader = T::Reader; type Writer = T::Writer; type Lister = T::Lister; type Deleter = T::Deleter; type BlockingReader = T::BlockingReader; type BlockingWriter = T::BlockingWriter; type BlockingLister = T::BlockingLister; type BlockingDeleter = T::BlockingDeleter; fn info(&self) -> Arc { self.as_ref().info() } fn create_dir( &self, path: &str, args: OpCreateDir, ) -> impl Future> + MaybeSend { async move { self.as_ref().create_dir(path, args).await } } fn stat(&self, path: &str, args: OpStat) -> impl Future> + MaybeSend { async move { self.as_ref().stat(path, args).await } } fn read( &self, path: &str, args: OpRead, ) -> impl Future> + MaybeSend { async move { self.as_ref().read(path, args).await } } fn write( &self, path: &str, args: OpWrite, ) -> impl Future> + MaybeSend { async move { self.as_ref().write(path, args).await } } fn delete(&self) -> impl Future> + MaybeSend { async move { self.as_ref().delete().await } } fn list( &self, path: &str, args: OpList, ) -> impl Future> + MaybeSend { async move { self.as_ref().list(path, args).await } } fn copy( &self, from: &str, to: &str, args: OpCopy, ) -> impl Future> + MaybeSend { async move { self.as_ref().copy(from, to, args).await } } fn rename( &self, from: &str, to: &str, args: OpRename, ) -> impl Future> + MaybeSend { async move { self.as_ref().rename(from, to, args).await } } fn presign( &self, path: &str, args: OpPresign, ) -> impl Future> + MaybeSend { async move { self.as_ref().presign(path, args).await } } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.as_ref().blocking_create_dir(path, args) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.as_ref().blocking_stat(path, args) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { self.as_ref().blocking_read(path, args) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { self.as_ref().blocking_write(path, args) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { self.as_ref().blocking_delete() } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { self.as_ref().blocking_list(path, args) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.as_ref().blocking_copy(from, to, args) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.as_ref().blocking_rename(from, to, args) } } /// Accessor is the type erased accessor with `Arc`. pub type Accessor = Arc; /// Metadata for accessor, users can use this metadata to get information of underlying backend. #[derive(Clone, Debug, Default)] pub struct AccessorInfo { scheme: Scheme, root: String, name: String, native_capability: Capability, full_capability: Capability, } impl AccessorInfo { /// [`Scheme`] of backend. pub fn scheme(&self) -> Scheme { self.scheme } /// Set [`Scheme`] for backend. pub fn set_scheme(&mut self, scheme: Scheme) -> &mut Self { self.scheme = scheme; self } /// Root of backend, will be in format like `/path/to/dir/` pub fn root(&self) -> &str { &self.root } /// Set root for backend. /// /// Note: input root must be normalized. pub fn set_root(&mut self, root: &str) -> &mut Self { self.root = root.to_string(); self } /// Name of backend, could be empty if underlying backend doesn't have namespace concept. /// /// For example: /// /// - name for `s3` => bucket name /// - name for `azblob` => container name pub fn name(&self) -> &str { &self.name } /// Set name of this backend. pub fn set_name(&mut self, name: &str) -> &mut Self { self.name = name.to_string(); self } /// Get backend's native capabilities. pub fn native_capability(&self) -> Capability { self.native_capability } /// Set native capabilities for service. /// /// # NOTES /// /// Set native capability will also flush the full capability. The only way to change /// full_capability is via `full_capability_mut`. pub fn set_native_capability(&mut self, capability: Capability) -> &mut Self { self.native_capability = capability; self.full_capability = capability; self } /// Get service's full capabilities. pub fn full_capability(&self) -> Capability { self.full_capability } /// Get service's full capabilities. pub fn full_capability_mut(&mut self) -> &mut Capability { &mut self.full_capability } } opendal-0.52.0/src/raw/adapters/kv/api.rs000064400000000000000000000147221046102023000162620ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::future::ready; use std::ops::DerefMut; use futures::Future; use crate::raw::*; use crate::Capability; use crate::Scheme; use crate::*; /// Scan is the async iterator returned by `Adapter::scan`. pub trait Scan: Send + Sync + Unpin { /// Fetch the next key in the current key prefix /// /// `Ok(None)` means no further key will be returned fn next(&mut self) -> impl Future>> + MaybeSend; } /// A noop implementation of Scan impl Scan for () { async fn next(&mut self) -> Result> { Ok(None) } } /// A Scan implementation for all trivial non-async iterators pub struct ScanStdIter(I); #[cfg(any( feature = "services-cloudflare-kv", feature = "services-etcd", feature = "services-nebula-graph", feature = "services-rocksdb", feature = "services-sled" ))] impl ScanStdIter where I: Iterator> + Unpin + Send + Sync, { /// Create a new ScanStdIter from an Iterator pub(crate) fn new(inner: I) -> Self { Self(inner) } } impl Scan for ScanStdIter where I: Iterator> + Unpin + Send + Sync, { async fn next(&mut self) -> Result> { self.0.next().transpose() } } /// A type-erased wrapper of Scan pub type Scanner = Box; pub trait ScanDyn: Unpin + Send + Sync { fn next_dyn(&mut self) -> BoxedFuture>>; } impl ScanDyn for T { fn next_dyn(&mut self) -> BoxedFuture>> { Box::pin(self.next()) } } impl Scan for Box { async fn next(&mut self) -> Result> { self.deref_mut().next_dyn().await } } /// KvAdapter is the adapter to underlying kv services. /// /// By implement this trait, any kv service can work as an OpenDAL Service. pub trait Adapter: Send + Sync + Debug + Unpin + 'static { /// TODO: use default associate type `= ()` after stabilized type Scanner: Scan; /// Return the info of this key value accessor. fn info(&self) -> Info; /// Get a key from service. /// /// - return `Ok(None)` if this key is not exist. fn get(&self, path: &str) -> impl Future>> + MaybeSend; /// The blocking version of get. fn blocking_get(&self, path: &str) -> Result> { let _ = path; Err(Error::new( ErrorKind::Unsupported, "kv adapter doesn't support this operation", ) .with_operation("kv::Adapter::blocking_get")) } /// Set a key into service. fn set(&self, path: &str, value: Buffer) -> impl Future> + MaybeSend; /// The blocking version of set. fn blocking_set(&self, path: &str, value: Buffer) -> Result<()> { let _ = (path, value); Err(Error::new( ErrorKind::Unsupported, "kv adapter doesn't support this operation", ) .with_operation("kv::Adapter::blocking_set")) } /// Delete a key from service. /// /// - return `Ok(())` even if this key is not exist. fn delete(&self, path: &str) -> impl Future> + MaybeSend; /// Delete a key from service in blocking way. /// /// - return `Ok(())` even if this key is not exist. fn blocking_delete(&self, path: &str) -> Result<()> { let _ = path; Err(Error::new( ErrorKind::Unsupported, "kv adapter doesn't support this operation", ) .with_operation("kv::Adapter::blocking_delete")) } /// Scan a key prefix to get all keys that start with this key. fn scan(&self, path: &str) -> impl Future> + MaybeSend { let _ = path; ready(Err(Error::new( ErrorKind::Unsupported, "kv adapter doesn't support this operation", ) .with_operation("kv::Adapter::scan"))) } /// Scan a key prefix to get all keys that start with this key /// in blocking way. fn blocking_scan(&self, path: &str) -> Result> { let _ = path; Err(Error::new( ErrorKind::Unsupported, "kv adapter doesn't support this operation", ) .with_operation("kv::Adapter::blocking_scan")) } /// Append a key into service fn append(&self, path: &str, value: &[u8]) -> impl Future> + MaybeSend { let _ = path; let _ = value; ready(Err(Error::new( ErrorKind::Unsupported, "kv adapter doesn't support this operation", ) .with_operation("kv::Adapter::append"))) } /// Append a key into service /// in blocking way. fn blocking_append(&self, path: &str, value: &[u8]) -> Result<()> { let _ = path; let _ = value; Err(Error::new( ErrorKind::Unsupported, "kv adapter doesn't support this operation", ) .with_operation("kv::Adapter::blocking_append")) } } /// Info for this key value accessor. pub struct Info { scheme: Scheme, name: String, capabilities: Capability, } impl Info { /// Create a new KeyValueAccessorInfo. pub fn new(scheme: Scheme, name: &str, capabilities: Capability) -> Self { Self { scheme, name: name.to_string(), capabilities, } } /// Get the scheme. pub fn scheme(&self) -> Scheme { self.scheme } /// Get the name. pub fn name(&self) -> &str { &self.name } /// Get the capabilities. pub fn capabilities(&self) -> Capability { self.capabilities } } opendal-0.52.0/src/raw/adapters/kv/backend.rs000064400000000000000000000245631046102023000171040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use std::vec::IntoIter; use super::{Adapter, Scan}; use crate::raw::oio::HierarchyLister; use crate::raw::oio::QueueBuf; use crate::raw::*; use crate::*; /// Backend of kv service. If the storage service is one k-v-like service, it should implement this kv [`Backend`] by right. /// /// `Backend` implements one general logic on how to read, write, scan the data from one kv store efficiently. /// And the [`Adapter`] held by `Backend` will handle how to communicate with one k-v-like service really and provides /// a series of basic operation for this service. /// /// OpenDAL developer can implement one new k-v store backend easily with help of this Backend. #[derive(Debug, Clone)] pub struct Backend { kv: Arc, root: String, } impl Backend where S: Adapter, { /// Create a new kv backend. pub fn new(kv: S) -> Self { Self { kv: Arc::new(kv), root: "/".to_string(), } } /// Configure root within this backend. pub fn with_root(self, root: &str) -> Self { self.with_normalized_root(normalize_root(root)) } /// Configure root within this backend. /// /// This method assumes root is normalized. pub(crate) fn with_normalized_root(mut self, root: String) -> Self { self.root = root; self } } impl Access for Backend { type Reader = Buffer; type Writer = KvWriter; type Lister = HierarchyLister>; type Deleter = oio::OneShotDeleter>; type BlockingReader = Buffer; type BlockingWriter = KvWriter; type BlockingLister = HierarchyLister; type BlockingDeleter = oio::OneShotDeleter>; fn info(&self) -> Arc { let kv_info = self.kv.info(); let mut am: AccessorInfo = AccessorInfo::default(); am.set_root(&self.root); am.set_scheme(kv_info.scheme()); am.set_name(kv_info.name()); let mut cap = kv_info.capabilities(); if cap.read { cap.stat = true; } if cap.write { cap.write_can_empty = true; cap.delete = true; } if cap.list { cap.list_with_recursive = true; } am.set_native_capability(cap); am.into() } async fn stat(&self, path: &str, _: OpStat) -> Result { let p = build_abs_path(&self.root, path); if p == build_abs_path(&self.root, "") { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } else { let bs = self.kv.get(&p).await?; match bs { Some(bs) => Ok(RpStat::new( Metadata::new(EntryMode::FILE).with_content_length(bs.len() as u64), )), None => Err(Error::new(ErrorKind::NotFound, "kv doesn't have this path")), } } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let p = build_abs_path(&self.root, path); let bs = match self.kv.get(&p).await? { Some(bs) => bs, None => return Err(Error::new(ErrorKind::NotFound, "kv doesn't have this path")), }; Ok((RpRead::new(), bs.slice(args.range().to_range_as_usize()))) } async fn write(&self, path: &str, _: OpWrite) -> Result<(RpWrite, Self::Writer)> { let p = build_abs_path(&self.root, path); Ok((RpWrite::new(), KvWriter::new(self.kv.clone(), p))) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(KvDeleter::new(self.kv.clone(), self.root.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let p = build_abs_path(&self.root, path); let res = self.kv.scan(&p).await?; let lister = KvLister::new(&self.root, res); let lister = HierarchyLister::new(lister, path, args.recursive()); Ok((RpList::default(), lister)) } fn blocking_stat(&self, path: &str, _: OpStat) -> Result { let p = build_abs_path(&self.root, path); if p == build_abs_path(&self.root, "") { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } else { let bs = self.kv.blocking_get(&p)?; match bs { Some(bs) => Ok(RpStat::new( Metadata::new(EntryMode::FILE).with_content_length(bs.len() as u64), )), None => Err(Error::new(ErrorKind::NotFound, "kv doesn't have this path")), } } } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let p = build_abs_path(&self.root, path); let bs = match self.kv.blocking_get(&p)? { Some(bs) => bs, None => return Err(Error::new(ErrorKind::NotFound, "kv doesn't have this path")), }; Ok((RpRead::new(), bs.slice(args.range().to_range_as_usize()))) } fn blocking_write(&self, path: &str, _: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let p = build_abs_path(&self.root, path); Ok((RpWrite::new(), KvWriter::new(self.kv.clone(), p))) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(KvDeleter::new(self.kv.clone(), self.root.clone())), )) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { let p = build_abs_path(&self.root, path); let res = self.kv.blocking_scan(&p)?; let lister = BlockingKvLister::new(&self.root, res); let lister = HierarchyLister::new(lister, path, args.recursive()); Ok((RpList::default(), lister)) } } pub struct KvLister { root: String, inner: Iter, } impl KvLister where Iter: Scan, { fn new(root: &str, inner: Iter) -> Self { Self { root: root.to_string(), inner, } } async fn inner_next(&mut self) -> Result> { Ok(self.inner.next().await?.map(|v| { let mode = if v.ends_with('/') { EntryMode::DIR } else { EntryMode::FILE }; let mut path = build_rel_path(&self.root, &v); if path.is_empty() { path = "/".to_string(); } oio::Entry::new(&path, Metadata::new(mode)) })) } } impl oio::List for KvLister where Iter: Scan, { async fn next(&mut self) -> Result> { self.inner_next().await } } pub struct BlockingKvLister { root: String, inner: IntoIter, } impl BlockingKvLister { fn new(root: &str, inner: Vec) -> Self { Self { root: root.to_string(), inner: inner.into_iter(), } } fn inner_next(&mut self) -> Option { self.inner.next().map(|v| { let mode = if v.ends_with('/') { EntryMode::DIR } else { EntryMode::FILE }; let mut path = build_rel_path(&self.root, &v); if path.is_empty() { path = "/".to_string(); } oio::Entry::new(&path, Metadata::new(mode)) }) } } impl oio::BlockingList for BlockingKvLister { fn next(&mut self) -> Result> { Ok(self.inner_next()) } } pub struct KvWriter { kv: Arc, path: String, buffer: QueueBuf, } impl KvWriter { fn new(kv: Arc, path: String) -> Self { KvWriter { kv, path, buffer: QueueBuf::new(), } } } /// # Safety /// /// We will only take `&mut Self` reference for KvWriter. unsafe impl Sync for KvWriter {} impl oio::Write for KvWriter { async fn write(&mut self, bs: Buffer) -> Result<()> { self.buffer.push(bs); Ok(()) } async fn close(&mut self) -> Result { let buf = self.buffer.clone().collect(); let length = buf.len() as u64; self.kv.set(&self.path, buf).await?; let meta = Metadata::new(EntryMode::from_path(&self.path)).with_content_length(length); Ok(meta) } async fn abort(&mut self) -> Result<()> { self.buffer.clear(); Ok(()) } } impl oio::BlockingWrite for KvWriter { fn write(&mut self, bs: Buffer) -> Result<()> { self.buffer.push(bs); Ok(()) } fn close(&mut self) -> Result { let buf = self.buffer.clone().collect(); let length = buf.len() as u64; self.kv.blocking_set(&self.path, buf)?; let meta = Metadata::new(EntryMode::from_path(&self.path)).with_content_length(length); Ok(meta) } } pub struct KvDeleter { kv: Arc, root: String, } impl KvDeleter { fn new(kv: Arc, root: String) -> Self { KvDeleter { kv, root } } } impl oio::OneShotDelete for KvDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = build_abs_path(&self.root, &path); self.kv.delete(&p).await?; Ok(()) } } impl oio::BlockingOneShotDelete for KvDeleter { fn blocking_delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = build_abs_path(&self.root, &path); self.kv.blocking_delete(&p)?; Ok(()) } } opendal-0.52.0/src/raw/adapters/kv/mod.rs000064400000000000000000000023771046102023000162730ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Providing Key Value Adapter for OpenDAL. //! //! Any services that implement `Adapter` can be used an OpenDAL Service. mod api; pub use api::Adapter; pub use api::Info; pub use api::Scan; #[cfg(any( feature = "services-cloudflare-kv", feature = "services-etcd", feature = "services-nebula-graph", feature = "services-rocksdb", feature = "services-sled" ))] pub(crate) use api::ScanStdIter; pub use api::Scanner; mod backend; pub use backend::Backend; opendal-0.52.0/src/raw/adapters/mod.rs000064400000000000000000000031271046102023000156450ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Providing adapters and its implementations. //! //! Adapters in OpenDAL means services that shares similar behaviors. We use //! adapter to make those services been implemented more easily. For example, //! with [`kv::Adapter`], users only need to implement `get`, `set` for a service. //! //! # Notes //! //! Please import the module instead of its type. //! //! For example, use the following: //! //! ```ignore //! use opendal::adapters::kv; //! //! impl kv::Adapter for MyType {} //! ``` //! //! Instead of: //! //! ```ignore //! use opendal::adapters::kv::Adapter; //! //! impl Adapter for MyType {} //! ``` //! //! # Available Adapters //! //! - [`kv::Adapter`]: Adapter for Key Value Services like `redis`. //! - [`typed_kv::Adapter`]: Adapter key value services that in-memory. pub mod kv; pub mod typed_kv; opendal-0.52.0/src/raw/adapters/typed_kv/api.rs000064400000000000000000000130011046102023000174540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::future::ready; use std::future::Future; use std::mem::size_of; use chrono::Utc; use crate::raw::MaybeSend; use crate::Buffer; use crate::EntryMode; use crate::Error; use crate::ErrorKind; use crate::Metadata; use crate::Result; use crate::Scheme; /// Adapter is the typed adapter to underlying kv services. /// /// By implement this trait, any kv service can work as an OpenDAL Service. /// /// # Notes /// /// `typed_kv::Adapter` is the typed version of `kv::Adapter`. It's more /// efficient if the underlying kv service can store data with its type. For /// example, we can store `Bytes` along with its metadata so that we don't /// need to serialize/deserialize it when we get it from the service. /// /// Ideally, we should use `typed_kv::Adapter` instead of `kv::Adapter` for /// in-memory rust libs like moka and dashmap. pub trait Adapter: Send + Sync + Debug + Unpin + 'static { /// Return the info of this key value accessor. fn info(&self) -> Info; /// Get a value from adapter. fn get(&self, path: &str) -> impl Future>> + MaybeSend; /// Get a value from adapter. fn blocking_get(&self, path: &str) -> Result>; /// Set a value into adapter. fn set(&self, path: &str, value: Value) -> impl Future> + MaybeSend; /// Set a value into adapter. fn blocking_set(&self, path: &str, value: Value) -> Result<()>; /// Delete a value from adapter. fn delete(&self, path: &str) -> impl Future> + MaybeSend; /// Delete a value from adapter. fn blocking_delete(&self, path: &str) -> Result<()>; /// Scan a key prefix to get all keys that start with this key. fn scan(&self, path: &str) -> impl Future>> + MaybeSend { let _ = path; ready(Err(Error::new( ErrorKind::Unsupported, "typed_kv adapter doesn't support this operation", ) .with_operation("typed_kv::Adapter::scan"))) } /// Scan a key prefix to get all keys that start with this key /// in blocking way. fn blocking_scan(&self, path: &str) -> Result> { let _ = path; Err(Error::new( ErrorKind::Unsupported, "typed_kv adapter doesn't support this operation", ) .with_operation("typed_kv::Adapter::blocking_scan")) } } /// Value is the typed value stored in adapter. /// /// It's cheap to clone so that users can read data without extra copy. #[derive(Debug, Clone)] pub struct Value { /// Metadata of this value. pub metadata: Metadata, /// The corresponding content of this value. pub value: Buffer, } impl Value { /// Create a new dir of value. pub fn new_dir() -> Self { Self { metadata: Metadata::new(EntryMode::DIR) .with_content_length(0) .with_last_modified(Utc::now()), value: Buffer::new(), } } /// Size returns the in-memory size of Value. pub fn size(&self) -> usize { size_of::() + self.value.len() } } /// Capability is used to describe what operations are supported /// by Typed KV Operator. #[derive(Copy, Clone, Default)] pub struct Capability { /// If typed_kv operator supports get natively. pub get: bool, /// If typed_kv operator supports set natively. pub set: bool, /// If typed_kv operator supports delete natively. pub delete: bool, /// If typed_kv operator supports scan natively. pub scan: bool, /// If typed_kv operator supports shared access. pub shared: bool, } impl Debug for Capability { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut s = vec![]; if self.get { s.push("Get") } if self.set { s.push("Set"); } if self.delete { s.push("Delete"); } if self.scan { s.push("Scan"); } if self.shared { s.push("Shared"); } write!(f, "{{ {} }}", s.join(" | ")) } } /// Info for this key value accessor. pub struct Info { scheme: Scheme, name: String, capabilities: Capability, } impl Info { /// Create a new KeyValueAccessorInfo. pub fn new(scheme: Scheme, name: &str, capabilities: Capability) -> Self { Self { scheme, name: name.to_string(), capabilities, } } /// Get the scheme. pub fn scheme(&self) -> Scheme { self.scheme } /// Get the name. pub fn name(&self) -> &str { &self.name } /// Get the capabilities. pub fn capabilities(&self) -> Capability { self.capabilities } } opendal-0.52.0/src/raw/adapters/typed_kv/backend.rs000064400000000000000000000244201046102023000203010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use std::vec::IntoIter; use super::Adapter; use super::Value; use crate::raw::oio::HierarchyLister; use crate::raw::oio::QueueBuf; use crate::raw::*; use crate::*; /// The typed kv backend which implements Accessor for typed kv adapter. #[derive(Debug, Clone)] pub struct Backend { kv: Arc, root: String, } impl Backend where S: Adapter, { /// Create a new kv backend. pub fn new(kv: S) -> Self { Self { kv: Arc::new(kv), root: "/".to_string(), } } /// Configure root within this backend. pub fn with_root(mut self, root: &str) -> Self { self.root = normalize_root(root); self } } impl Access for Backend { type Reader = Buffer; type Writer = KvWriter; type Lister = HierarchyLister; type Deleter = oio::OneShotDeleter>; type BlockingReader = Buffer; type BlockingWriter = KvWriter; type BlockingLister = HierarchyLister; type BlockingDeleter = oio::OneShotDeleter>; fn info(&self) -> Arc { let kv_info = self.kv.info(); let mut am: AccessorInfo = AccessorInfo::default(); am.set_root(&self.root); am.set_scheme(kv_info.scheme()); am.set_name(kv_info.name()); let kv_cap = kv_info.capabilities(); let mut cap = Capability::default(); if kv_cap.get { cap.read = true; cap.stat = true; } if kv_cap.set { cap.write = true; cap.write_can_empty = true; } if kv_cap.delete { cap.delete = true; } if kv_cap.scan { cap.list = true; cap.list_with_recursive = true; } if kv_cap.shared { cap.shared = true; } cap.blocking = true; am.set_native_capability(cap); am.into() } async fn stat(&self, path: &str, _: OpStat) -> Result { let p = build_abs_path(&self.root, path); if p == build_abs_path(&self.root, "") { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } else { let bs = self.kv.get(&p).await?; match bs { Some(bs) => Ok(RpStat::new(bs.metadata)), None => Err(Error::new(ErrorKind::NotFound, "kv doesn't have this path")), } } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let p = build_abs_path(&self.root, path); let bs = match self.kv.get(&p).await? { // TODO: we can reuse the metadata in value to build content range. Some(bs) => bs.value, None => return Err(Error::new(ErrorKind::NotFound, "kv doesn't have this path")), }; Ok((RpRead::new(), bs.slice(args.range().to_range_as_usize()))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let p = build_abs_path(&self.root, path); Ok((RpWrite::new(), KvWriter::new(self.kv.clone(), p, args))) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(KvDeleter::new(self.kv.clone(), self.root.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let p = build_abs_path(&self.root, path); let res = self.kv.scan(&p).await?; let lister = KvLister::new(&self.root, res); let lister = HierarchyLister::new(lister, path, args.recursive()); Ok((RpList::default(), lister)) } fn blocking_stat(&self, path: &str, _: OpStat) -> Result { let p = build_abs_path(&self.root, path); if p == build_abs_path(&self.root, "") { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } else { let bs = self.kv.blocking_get(&p)?; match bs { Some(bs) => Ok(RpStat::new(bs.metadata)), None => Err(Error::new(ErrorKind::NotFound, "kv doesn't have this path")), } } } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let p = build_abs_path(&self.root, path); let bs = match self.kv.blocking_get(&p)? { // TODO: we can reuse the metadata in value to build content range. Some(bs) => bs.value, None => return Err(Error::new(ErrorKind::NotFound, "kv doesn't have this path")), }; Ok((RpRead::new(), bs.slice(args.range().to_range_as_usize()))) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let p = build_abs_path(&self.root, path); Ok((RpWrite::new(), KvWriter::new(self.kv.clone(), p, args))) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(KvDeleter::new(self.kv.clone(), self.root.clone())), )) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { let p = build_abs_path(&self.root, path); let res = self.kv.blocking_scan(&p)?; let lister = KvLister::new(&self.root, res); let lister = HierarchyLister::new(lister, path, args.recursive()); Ok((RpList::default(), lister)) } } pub struct KvLister { root: String, inner: IntoIter, } impl KvLister { fn new(root: &str, inner: Vec) -> Self { Self { root: root.to_string(), inner: inner.into_iter(), } } fn inner_next(&mut self) -> Option { self.inner.next().map(|v| { let mode = if v.ends_with('/') { EntryMode::DIR } else { EntryMode::FILE }; let mut path = build_rel_path(&self.root, &v); if path.is_empty() { path = "/".to_string(); } oio::Entry::new(&path, Metadata::new(mode)) }) } } impl oio::List for KvLister { async fn next(&mut self) -> Result> { Ok(self.inner_next()) } } impl oio::BlockingList for KvLister { fn next(&mut self) -> Result> { Ok(self.inner_next()) } } pub struct KvWriter { kv: Arc, path: String, op: OpWrite, buf: Option, value: Option, } /// # Safety /// /// We will only take `&mut Self` reference for KvWriter. unsafe impl Sync for KvWriter {} impl KvWriter { fn new(kv: Arc, path: String, op: OpWrite) -> Self { KvWriter { kv, path, op, buf: None, value: None, } } fn build(&mut self) -> Value { let value = self.buf.take().map(QueueBuf::collect).unwrap_or_default(); let mut metadata = Metadata::new(EntryMode::FILE); metadata.set_content_length(value.len() as u64); if let Some(v) = self.op.cache_control() { metadata.set_cache_control(v); } if let Some(v) = self.op.content_disposition() { metadata.set_content_disposition(v); } if let Some(v) = self.op.content_type() { metadata.set_content_type(v); } Value { metadata, value } } } impl oio::Write for KvWriter { async fn write(&mut self, bs: Buffer) -> Result<()> { let mut buf = self.buf.take().unwrap_or_default(); buf.push(bs); self.buf = Some(buf); Ok(()) } async fn close(&mut self) -> Result { let value = match &self.value { Some(value) => value.clone(), None => { let value = self.build(); self.value = Some(value.clone()); value } }; let meta = value.metadata.clone(); self.kv.set(&self.path, value).await?; Ok(meta) } async fn abort(&mut self) -> Result<()> { self.buf = None; Ok(()) } } impl oio::BlockingWrite for KvWriter { fn write(&mut self, bs: Buffer) -> Result<()> { let mut buf = self.buf.take().unwrap_or_default(); buf.push(bs); self.buf = Some(buf); Ok(()) } fn close(&mut self) -> Result { let kv = self.kv.clone(); let value = match &self.value { Some(value) => value.clone(), None => { let value = self.build(); self.value = Some(value.clone()); value } }; let meta = value.metadata.clone(); kv.blocking_set(&self.path, value)?; Ok(meta) } } pub struct KvDeleter { kv: Arc, root: String, } impl KvDeleter { fn new(kv: Arc, root: String) -> Self { KvDeleter { kv, root } } } impl oio::OneShotDelete for KvDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = build_abs_path(&self.root, &path); self.kv.delete(&p).await?; Ok(()) } } impl oio::BlockingOneShotDelete for KvDeleter { fn blocking_delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = build_abs_path(&self.root, &path); self.kv.blocking_delete(&p)?; Ok(()) } } opendal-0.52.0/src/raw/adapters/typed_kv/mod.rs000064400000000000000000000020531046102023000174670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Providing Typed Key Value Adapter for OpenDAL. //! //! Any services that implement `Adapter` can be used an OpenDAL Service. mod api; pub use api::Adapter; pub use api::Capability; pub use api::Info; pub use api::Value; mod backend; pub use backend::Backend; opendal-0.52.0/src/raw/atomic_util.rs000064400000000000000000000035621046102023000155770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; /// AtomicContentLength is a wrapper of AtomicU64 that used to store content length. /// /// It provides a way to store and load content length in atomic way, so caller don't need to /// use `Mutex` or `RwLock` to protect the content length. /// /// We use value `u64::MAX` to represent unknown size, it's impossible for us to /// handle a file that has `u64::MAX` bytes. #[derive(Debug)] pub struct AtomicContentLength(AtomicU64); impl Default for AtomicContentLength { fn default() -> Self { Self::new() } } impl AtomicContentLength { /// Create a new AtomicContentLength. pub const fn new() -> Self { Self(AtomicU64::new(u64::MAX)) } /// Load content length from AtomicU64. #[inline] pub fn load(&self) -> Option { match self.0.load(Ordering::Relaxed) { u64::MAX => None, v => Some(v), } } /// Store content length to AtomicU64. #[inline] pub fn store(&self, v: u64) { self.0.store(v, Ordering::Relaxed) } } opendal-0.52.0/src/raw/chrono_util.rs000064400000000000000000000054241046102023000156120ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::time::Duration; use std::time::UNIX_EPOCH; use chrono::DateTime; use chrono::Utc; use crate::*; /// Parse datetime from rfc2822. /// /// For example: `Fri, 28 Nov 2014 21:00:09 +0900` pub fn parse_datetime_from_rfc2822(s: &str) -> Result> { DateTime::parse_from_rfc2822(s) .map(|v| v.into()) .map_err(|e| { Error::new(ErrorKind::Unexpected, "parse datetime from rfc2822 failed").set_source(e) }) } /// Parse datetime from rfc3339. /// /// For example: `2014-11-28T21:00:09+09:00` pub fn parse_datetime_from_rfc3339(s: &str) -> Result> { DateTime::parse_from_rfc3339(s) .map(|v| v.into()) .map_err(|e| { Error::new(ErrorKind::Unexpected, "parse datetime from rfc3339 failed").set_source(e) }) } /// parse datetime from given timestamp_millis pub fn parse_datetime_from_from_timestamp_millis(s: i64) -> Result> { let st = UNIX_EPOCH .checked_add(Duration::from_millis(s as u64)) .ok_or_else(|| Error::new(ErrorKind::Unexpected, "input timestamp overflow"))?; Ok(st.into()) } /// parse datetime from given timestamp pub fn parse_datetime_from_from_timestamp(s: i64) -> Result> { let st = UNIX_EPOCH .checked_add(Duration::from_secs(s as u64)) .ok_or_else(|| Error::new(ErrorKind::Unexpected, "input timestamp overflow"))?; Ok(st.into()) } /// format datetime into http date, this format is required by: /// https://httpwg.org/specs/rfc9110.html#field.if-modified-since pub fn format_datetime_into_http_date(s: DateTime) -> String { s.format("%a, %d %b %Y %H:%M:%S GMT").to_string() } #[cfg(test)] mod tests { use super::*; #[test] fn test_format_datetime_into_http_date() { let s = "Sat, 29 Oct 1994 19:43:31 +0000"; let v = parse_datetime_from_rfc2822(s).unwrap(); assert_eq!( format_datetime_into_http_date(v), "Sat, 29 Oct 1994 19:43:31 GMT" ); } } opendal-0.52.0/src/raw/enum_utils.rs000064400000000000000000000160021046102023000154430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! [`type_alias_impl_trait`](https://github.com/rust-lang/rust/issues/63063) is not stable yet. //! So we can't write the following code: //! //! ```txt //! impl Access for S3Backend { //! type Writer = impl oio::Write; //! } //! ``` //! //! Which means we have to write the type directly like: //! //! ```txt //! impl Access for OssBackend { //! type Writer = raw::TwoWays< //! oio::MultipartWriter, //! oio::AppendWriter, //! >; //! } //! ``` //! //! This module is used to provide some enums for the above code. We should remove this module once //! type_alias_impl_trait has been stabilized. use crate::raw::*; use crate::*; /// TwoWays is used to implement traits that based on two ways. /// /// Users can wrap two different trait types together. pub enum TwoWays { /// The first type for the [`TwoWays`]. One(ONE), /// The second type for the [`TwoWays`]. Two(TWO), } impl oio::Read for TwoWays { async fn read(&mut self) -> Result { match self { TwoWays::One(v) => v.read().await, TwoWays::Two(v) => v.read().await, } } } impl oio::BlockingRead for TwoWays { fn read(&mut self) -> Result { match self { Self::One(v) => v.read(), Self::Two(v) => v.read(), } } } impl oio::Write for TwoWays { async fn write(&mut self, bs: Buffer) -> Result<()> { match self { Self::One(v) => v.write(bs).await, Self::Two(v) => v.write(bs).await, } } async fn close(&mut self) -> Result { match self { Self::One(v) => v.close().await, Self::Two(v) => v.close().await, } } async fn abort(&mut self) -> Result<()> { match self { Self::One(v) => v.abort().await, Self::Two(v) => v.abort().await, } } } impl oio::List for TwoWays { async fn next(&mut self) -> Result> { match self { Self::One(v) => v.next().await, Self::Two(v) => v.next().await, } } } /// ThreeWays is used to implement traits that based on three ways. /// /// Users can wrap three different trait types together. pub enum ThreeWays { /// The first type for the [`ThreeWays`]. One(ONE), /// The second type for the [`ThreeWays`]. Two(TWO), /// The third type for the [`ThreeWays`]. Three(THREE), } impl oio::Read for ThreeWays { async fn read(&mut self) -> Result { match self { ThreeWays::One(v) => v.read().await, ThreeWays::Two(v) => v.read().await, ThreeWays::Three(v) => v.read().await, } } } impl oio::BlockingRead for ThreeWays { fn read(&mut self) -> Result { match self { Self::One(v) => v.read(), Self::Two(v) => v.read(), Self::Three(v) => v.read(), } } } impl oio::Write for ThreeWays { async fn write(&mut self, bs: Buffer) -> Result<()> { match self { Self::One(v) => v.write(bs).await, Self::Two(v) => v.write(bs).await, Self::Three(v) => v.write(bs).await, } } async fn close(&mut self) -> Result { match self { Self::One(v) => v.close().await, Self::Two(v) => v.close().await, Self::Three(v) => v.close().await, } } async fn abort(&mut self) -> Result<()> { match self { Self::One(v) => v.abort().await, Self::Two(v) => v.abort().await, Self::Three(v) => v.abort().await, } } } /// FourWays is used to implement traits that based on four ways. /// /// Users can wrap four different trait types together. pub enum FourWays { /// The first type for the [`FourWays`]. One(ONE), /// The second type for the [`FourWays`]. Two(TWO), /// The third type for the [`FourWays`]. Three(THREE), /// The fourth type for the [`FourWays`]. Four(FOUR), } impl oio::Read for FourWays where ONE: oio::Read, TWO: oio::Read, THREE: oio::Read, FOUR: oio::Read, { async fn read(&mut self) -> Result { match self { FourWays::One(v) => v.read().await, FourWays::Two(v) => v.read().await, FourWays::Three(v) => v.read().await, FourWays::Four(v) => v.read().await, } } } impl oio::BlockingRead for FourWays where ONE: oio::BlockingRead, TWO: oio::BlockingRead, THREE: oio::BlockingRead, FOUR: oio::BlockingRead, { fn read(&mut self) -> Result { match self { Self::One(v) => v.read(), Self::Two(v) => v.read(), Self::Three(v) => v.read(), Self::Four(v) => v.read(), } } } impl oio::List for FourWays where ONE: oio::List, TWO: oio::List, THREE: oio::List, FOUR: oio::List, { async fn next(&mut self) -> Result> { match self { Self::One(v) => v.next().await, Self::Two(v) => v.next().await, Self::Three(v) => v.next().await, Self::Four(v) => v.next().await, } } } impl oio::BlockingList for FourWays where ONE: oio::BlockingList, TWO: oio::BlockingList, THREE: oio::BlockingList, FOUR: oio::BlockingList, { fn next(&mut self) -> Result> { match self { Self::One(v) => v.next(), Self::Two(v) => v.next(), Self::Three(v) => v.next(), Self::Four(v) => v.next(), } } } opendal-0.52.0/src/raw/futures_util.rs000064400000000000000000000446111046102023000160200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::VecDeque; use std::future::Future; use std::pin::Pin; use std::task::Context; use std::task::Poll; use futures::poll; use futures::stream::FuturesOrdered; use futures::FutureExt; use futures::StreamExt; use crate::*; /// BoxedFuture is the type alias of [`futures::future::BoxFuture`]. /// /// We will switch to [`futures::future::LocalBoxFuture`] on wasm32 target. #[cfg(not(target_arch = "wasm32"))] pub type BoxedFuture<'a, T> = futures::future::BoxFuture<'a, T>; #[cfg(target_arch = "wasm32")] pub type BoxedFuture<'a, T> = futures::future::LocalBoxFuture<'a, T>; /// BoxedStaticFuture is the type alias of [`futures::future::BoxFuture`]. /// /// We will switch to [`futures::future::LocalBoxFuture`] on wasm32 target. #[cfg(not(target_arch = "wasm32"))] pub type BoxedStaticFuture = futures::future::BoxFuture<'static, T>; #[cfg(target_arch = "wasm32")] pub type BoxedStaticFuture = futures::future::LocalBoxFuture<'static, T>; /// MaybeSend is a marker to determine whether a type is `Send` or not. /// We use this trait to wrap the `Send` requirement for wasm32 target. /// /// # Safety /// /// [`MaybeSend`] is equivalent to `Send` on non-wasm32 target. /// And it's empty trait on wasm32 target to indicate that a type is not `Send`. #[cfg(not(target_arch = "wasm32"))] pub trait MaybeSend: Send {} #[cfg(target_arch = "wasm32")] pub trait MaybeSend {} #[cfg(not(target_arch = "wasm32"))] impl MaybeSend for T {} #[cfg(target_arch = "wasm32")] impl MaybeSend for T {} /// ConcurrentTasks is used to execute tasks concurrently. /// /// ConcurrentTasks has two generic types: /// /// - `I` represents the input type of the task. /// - `O` represents the output type of the task. pub struct ConcurrentTasks { /// The executor to execute the tasks. /// /// If user doesn't provide an executor, the tasks will be executed with the default executor. executor: Executor, /// The factory to create the task. /// /// Caller of ConcurrentTasks must provides a factory to create the task for executing. /// /// The factory must accept an input and return a future that resolves to a tuple of input and /// output result. If the given result is error, the error will be returned to users and the /// task will be retried. factory: fn(I) -> BoxedStaticFuture<(I, Result)>, /// `tasks` holds the ongoing tasks. /// /// Please keep in mind that all tasks are running in the background by `Executor`. We only need /// to poll the tasks to see if they are ready. /// /// Dropping task without `await` it will cancel the task. tasks: VecDeque)>>, /// `results` stores the successful results. results: VecDeque, /// hitting the last unrecoverable error. /// /// If concurrent tasks hit an unrecoverable error, it will stop executing new tasks and return /// an unrecoverable error to users. errored: bool, } impl ConcurrentTasks { /// Create a new concurrent tasks with given executor, concurrent and factory. /// /// The factory is a function pointer that shouldn't capture any context. pub fn new( executor: Executor, concurrent: usize, factory: fn(I) -> BoxedStaticFuture<(I, Result)>, ) -> Self { Self { executor, factory, tasks: VecDeque::with_capacity(concurrent), results: VecDeque::with_capacity(concurrent), errored: false, } } /// Return true if the tasks are running concurrently. #[inline] fn is_concurrent(&self) -> bool { self.tasks.capacity() > 1 } /// Clear all tasks and results. /// /// All ongoing tasks will be canceled. pub fn clear(&mut self) { self.tasks.clear(); self.results.clear(); } /// Check if there are remaining space to push new tasks. #[inline] pub fn has_remaining(&self) -> bool { self.tasks.len() < self.tasks.capacity() } /// Chunk if there are remaining results to fetch. #[inline] pub fn has_result(&self) -> bool { !self.results.is_empty() } /// Execute the task with given input. /// /// - Execute the task in the current thread if is not concurrent. /// - Execute the task in the background if there are available slots. /// - Await the first task in the queue if there is no available slots. pub async fn execute(&mut self, input: I) -> Result<()> { if self.errored { return Err(Error::new( ErrorKind::Unexpected, "concurrent tasks met an unrecoverable error", )); } // Short path for non-concurrent case. if !self.is_concurrent() { let (_, o) = (self.factory)(input).await; return match o { Ok(o) => { self.results.push_back(o); Ok(()) } // We don't need to rebuild the future if it's not concurrent. Err(err) => Err(err), }; } loop { // Try poll once to see if there is any ready task. if let Some(task) = self.tasks.front_mut() { if let Poll::Ready((i, o)) = poll!(task) { match o { Ok(o) => { let _ = self.tasks.pop_front(); self.results.push_back(o) } Err(err) => { // Retry this task if the error is temporary if err.is_temporary() { self.tasks .front_mut() .expect("tasks must have at least one task") .replace(self.executor.execute((self.factory)(i))); } else { self.clear(); self.errored = true; } return Err(err); } } } } // Try to push new task if there are available space. if self.tasks.len() < self.tasks.capacity() { self.tasks .push_back(self.executor.execute((self.factory)(input))); return Ok(()); } // Wait for the next task to be ready. let task = self .tasks .front_mut() .expect("tasks must have at least one task"); let (i, o) = task.await; match o { Ok(o) => { let _ = self.tasks.pop_front(); self.results.push_back(o); continue; } Err(err) => { // Retry this task if the error is temporary if err.is_temporary() { self.tasks .front_mut() .expect("tasks must have at least one task") .replace(self.executor.execute((self.factory)(i))); } else { self.clear(); self.errored = true; } return Err(err); } } } } /// Fetch the successful result from the result queue. pub async fn next(&mut self) -> Option> { if self.errored { return Some(Err(Error::new( ErrorKind::Unexpected, "concurrent tasks met an unrecoverable error", ))); } if let Some(result) = self.results.pop_front() { return Some(Ok(result)); } if let Some(task) = self.tasks.front_mut() { let (i, o) = task.await; return match o { Ok(o) => { let _ = self.tasks.pop_front(); Some(Ok(o)) } Err(err) => { // Retry this task if the error is temporary if err.is_temporary() { self.tasks .front_mut() .expect("tasks must have at least one task") .replace(self.executor.execute((self.factory)(i))); } else { self.clear(); self.errored = true; } Some(Err(err)) } }; } None } } /// CONCURRENT_LARGE_THRESHOLD is the threshold to determine whether to use /// [`FuturesOrdered`] or not. /// /// The value of `8` is picked by random, no strict benchmark is done. /// Please raise an issue if you found the value is not good enough or you want to configure /// this value at runtime. const CONCURRENT_LARGE_THRESHOLD: usize = 8; /// ConcurrentFutures is a stream that can hold a stream of concurrent futures. /// /// - the order of the futures is the same. /// - the number of concurrent futures is limited by concurrent. /// - optimized for small number of concurrent futures. /// - zero cost for non-concurrent futures cases (concurrent == 1). pub struct ConcurrentFutures { tasks: Tasks, concurrent: usize, } /// Tasks is used to hold the entire task queue. enum Tasks { /// The special case for concurrent == 1. /// /// It works exactly the same like `Option` in a struct. Once(Option), /// The special cases for concurrent is small. /// /// At this case, the cost to loop poll is lower than using `FuturesOrdered`. /// /// We will replace the future by `TaskResult::Ready` once it's ready to avoid consume it again. Small(VecDeque>), /// The general cases for large concurrent. /// /// We use `FuturesOrdered` to avoid huge amount of poll on futures. Large(FuturesOrdered), } impl Unpin for Tasks {} enum TaskResult { Polling(F), Ready(F::Output), } impl ConcurrentFutures where F: Future + Unpin + 'static, { /// Create a new ConcurrentFutures by specifying the number of concurrent futures. pub fn new(concurrent: usize) -> Self { if (0..2).contains(&concurrent) { Self { tasks: Tasks::Once(None), concurrent, } } else if (2..=CONCURRENT_LARGE_THRESHOLD).contains(&concurrent) { Self { tasks: Tasks::Small(VecDeque::with_capacity(concurrent)), concurrent, } } else { Self { tasks: Tasks::Large(FuturesOrdered::new()), concurrent, } } } /// Drop all tasks. pub fn clear(&mut self) { match &mut self.tasks { Tasks::Once(fut) => *fut = None, Tasks::Small(tasks) => tasks.clear(), Tasks::Large(tasks) => *tasks = FuturesOrdered::new(), } } /// Return the length of current concurrent futures (both ongoing and ready). pub fn len(&self) -> usize { match &self.tasks { Tasks::Once(fut) => fut.is_some() as usize, Tasks::Small(v) => v.len(), Tasks::Large(v) => v.len(), } } /// Return true if there is no futures in the queue. pub fn is_empty(&self) -> bool { self.len() == 0 } /// Return the number of remaining space to push new futures. pub fn remaining(&self) -> usize { self.concurrent - self.len() } /// Return true if there is remaining space to push new futures. pub fn has_remaining(&self) -> bool { self.remaining() > 0 } /// Push new future into the end of queue. pub fn push_back(&mut self, f: F) { debug_assert!( self.has_remaining(), "concurrent futures must have remaining space" ); match &mut self.tasks { Tasks::Once(fut) => { *fut = Some(f); } Tasks::Small(v) => v.push_back(TaskResult::Polling(f)), Tasks::Large(v) => v.push_back(f), } } /// Push new future into the start of queue, this task will be exactly the next to poll. pub fn push_front(&mut self, f: F) { debug_assert!( self.has_remaining(), "concurrent futures must have remaining space" ); match &mut self.tasks { Tasks::Once(fut) => { *fut = Some(f); } Tasks::Small(v) => v.push_front(TaskResult::Polling(f)), Tasks::Large(v) => v.push_front(f), } } } impl futures::Stream for ConcurrentFutures where F: Future + Unpin + 'static, { type Item = F::Output; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { match &mut self.get_mut().tasks { Tasks::Once(fut) => match fut { Some(x) => x.poll_unpin(cx).map(|v| { *fut = None; Some(v) }), None => Poll::Ready(None), }, Tasks::Small(v) => { // Poll all tasks together. for task in v.iter_mut() { if let TaskResult::Polling(f) = task { match f.poll_unpin(cx) { Poll::Pending => {} Poll::Ready(res) => { // Replace with ready value if this future has been resolved. *task = TaskResult::Ready(res); } } } } // Pick the first one to check. match v.front_mut() { // Return pending if the first one is still polling. Some(TaskResult::Polling(_)) => Poll::Pending, Some(TaskResult::Ready(_)) => { let res = v.pop_front().unwrap(); match res { TaskResult::Polling(_) => unreachable!(), TaskResult::Ready(res) => Poll::Ready(Some(res)), } } None => Poll::Ready(None), } } Tasks::Large(v) => v.poll_next_unpin(cx), } } } #[cfg(test)] mod tests { use std::task::ready; use std::time::Duration; use futures::future::BoxFuture; use futures::Stream; use rand::Rng; use tokio::time::sleep; use super::*; struct Lister { size: usize, idx: usize, concurrent: usize, tasks: ConcurrentFutures>, } impl Stream for Lister { type Item = usize; fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { // Randomly sleep for a while, simulate some io operations that up to 100 microseconds. let timeout = Duration::from_micros(rand::thread_rng().gen_range(0..100)); let idx = self.idx; if self.tasks.len() < self.concurrent && self.idx < self.size { let fut = async move { tokio::time::sleep(timeout).await; idx }; self.idx += 1; self.tasks.push_back(Box::pin(fut)); } if let Some(v) = ready!(self.tasks.poll_next_unpin(cx)) { Poll::Ready(Some(v)) } else { Poll::Ready(None) } } } #[tokio::test] async fn test_concurrent_futures() { let cases = vec![ ("once", 1), ("small", CONCURRENT_LARGE_THRESHOLD - 1), ("large", CONCURRENT_LARGE_THRESHOLD + 1), ]; for (name, concurrent) in cases { let lister = Lister { size: 1000, idx: 0, concurrent, tasks: ConcurrentFutures::new(concurrent), }; let expected: Vec = (0..1000).collect(); let result: Vec = lister.collect().await; assert_eq!(expected, result, "concurrent futures failed: {}", name); } } #[tokio::test] async fn test_concurrent_tasks() { let executor = Executor::new(); let mut tasks = ConcurrentTasks::new(executor, 16, |(i, dur)| { Box::pin(async move { sleep(dur).await; // 5% rate to fail. if rand::thread_rng().gen_range(0..100) > 90 { return ( (i, dur), Err(Error::new(ErrorKind::Unexpected, "I'm lucky").set_temporary()), ); } ((i, dur), Ok(i)) }) }); let mut ans = vec![]; for i in 0..10240 { // Sleep up to 10ms let dur = Duration::from_millis(rand::thread_rng().gen_range(0..10)); loop { let res = tasks.execute((i, dur)).await; if res.is_ok() { break; } } } loop { match tasks.next().await.transpose() { Ok(Some(i)) => ans.push(i), Ok(None) => break, Err(_) => continue, } } assert_eq!(ans, (0..10240).collect::>()) } } opendal-0.52.0/src/raw/http_util/body.rs000064400000000000000000000070471046102023000162410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::cmp::Ordering; use futures::Stream; use futures::StreamExt; use oio::Read; use crate::raw::*; use crate::*; /// HttpBody is the streaming body that opendal's HttpClient returned. /// /// It implements the `oio::Read` trait, service implementors can return it as /// `Access::Read`. pub struct HttpBody { #[cfg(not(target_arch = "wasm32"))] stream: Box> + Send + Sync + Unpin + 'static>, #[cfg(target_arch = "wasm32")] stream: Box> + Unpin + 'static>, size: Option, consumed: u64, } /// # Safety /// /// HttpBody is `Send` on non wasm32 targets. unsafe impl Send for HttpBody {} /// # Safety /// /// HttpBody is sync on non wasm32 targets. unsafe impl Sync for HttpBody {} impl HttpBody { /// Create a new `HttpBody` with given stream and optional size. #[cfg(not(target_arch = "wasm32"))] pub fn new(stream: S, size: Option) -> Self where S: Stream> + Send + Sync + Unpin + 'static, { HttpBody { stream: Box::new(stream), size, consumed: 0, } } /// Create a new `HttpBody` with given stream and optional size. #[cfg(target_arch = "wasm32")] pub fn new(stream: S, size: Option) -> Self where S: Stream> + Unpin + 'static, { HttpBody { stream: Box::new(stream), size, consumed: 0, } } /// Check if the consumed data is equal to the expected content length. #[inline] fn check(&self) -> Result<()> { let Some(expect) = self.size else { return Ok(()); }; let actual = self.consumed; match actual.cmp(&expect) { Ordering::Equal => Ok(()), Ordering::Less => Err(Error::new( ErrorKind::Unexpected, format!("http response got too little data, expect: {expect}, actual: {actual}"), ) .set_temporary()), Ordering::Greater => Err(Error::new( ErrorKind::Unexpected, format!("http response got too much data, expect: {expect}, actual: {actual}"), ) .set_temporary()), } } /// Read all data from the stream. pub async fn to_buffer(&mut self) -> Result { self.read_all().await } } impl oio::Read for HttpBody { async fn read(&mut self) -> Result { match self.stream.next().await.transpose()? { Some(buf) => { self.consumed += buf.len() as u64; Ok(buf) } None => { self.check()?; Ok(Buffer::new()) } } } } opendal-0.52.0/src/raw/http_util/bytes_content_range.rs000064400000000000000000000172421046102023000213360ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Display; use std::fmt::Formatter; use std::ops::Range; use std::ops::RangeInclusive; use std::str::FromStr; use crate::*; /// BytesContentRange is the content range of bytes. /// /// `` should always be `bytes`. /// /// ```text /// Content-Range: bytes -/ /// Content-Range: bytes -/* /// Content-Range: bytes */ /// ``` /// /// # Notes /// /// ## Usage of the default. /// /// `BytesContentRange::default` is not a valid content range. /// Please make sure their comes up with `with_range` or `with_size` call. /// /// ## Allow clippy::len_without_is_empty /// /// BytesContentRange implements `len()` but not `is_empty()` because it's useless. /// - When BytesContentRange's range is known, it must be non-empty. /// - When BytesContentRange's range is no known, we don't know whether it's empty. #[allow(clippy::len_without_is_empty)] #[derive(Debug, Default, Clone, Copy, Eq, PartialEq, serde::Serialize, serde::Deserialize)] pub struct BytesContentRange( /// Start position of the range. `None` means unknown. Option, /// End position of the range. `None` means unknown. Option, /// Size of the whole content. `None` means unknown. Option, ); impl BytesContentRange { /// Update BytesContentRange with range. /// /// The range is inclusive: `[start..=end]` as described in `content-range`. pub fn with_range(mut self, start: u64, end: u64) -> Self { self.0 = Some(start); self.1 = Some(end); self } /// Update BytesContentRange with size. pub fn with_size(mut self, size: u64) -> Self { self.2 = Some(size); self } /// Get the length that specified by this BytesContentRange, return `None` if range is not known. pub fn len(&self) -> Option { if let (Some(start), Some(end)) = (self.0, self.1) { Some(end - start + 1) } else { None } } /// Get the size of this BytesContentRange, return `None` if size is not known. pub fn size(&self) -> Option { self.2 } /// Get the range inclusive of this BytesContentRange, return `None` if range is not known. pub fn range(&self) -> Option> { if let (Some(start), Some(end)) = (self.0, self.1) { Some(start..end + 1) } else { None } } /// Get the range inclusive of this BytesContentRange, return `None` if range is not known. pub fn range_inclusive(&self) -> Option> { if let (Some(start), Some(end)) = (self.0, self.1) { Some(start..=end) } else { None } } /// Convert bytes content range into Content-Range header. pub fn to_header(&self) -> String { format!("bytes {self}") } } impl Display for BytesContentRange { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { match (self.0, self.1, self.2) { (Some(start), Some(end), Some(size)) => write!(f, "{start}-{end}/{size}"), (Some(start), Some(end), None) => write!(f, "{start}-{end}/*"), (None, None, Some(size)) => write!(f, "*/{size}"), _ => unreachable!("invalid bytes range: {:?}", self), } } } impl FromStr for BytesContentRange { type Err = Error; fn from_str(value: &str) -> Result { let s = value.strip_prefix("bytes ").ok_or_else(|| { Error::new(ErrorKind::Unexpected, "header content range is invalid") .with_operation("BytesContentRange::from_str") .with_context("value", value) })?; let parse_int_error = |e: std::num::ParseIntError| { Error::new(ErrorKind::Unexpected, "header content range is invalid") .with_operation("BytesContentRange::from_str") .with_context("value", value) .set_source(e) }; if let Some(size) = s.strip_prefix("*/") { return Ok( BytesContentRange::default().with_size(size.parse().map_err(parse_int_error)?) ); } let s: Vec<_> = s.split('/').collect(); if s.len() != 2 { return Err( Error::new(ErrorKind::Unexpected, "header content range is invalid") .with_operation("BytesContentRange::from_str") .with_context("value", value), ); } let v: Vec<_> = s[0].split('-').collect(); if v.len() != 2 { return Err( Error::new(ErrorKind::Unexpected, "header content range is invalid") .with_operation("BytesContentRange::from_str") .with_context("value", value), ); } let start: u64 = v[0].parse().map_err(parse_int_error)?; let end: u64 = v[1].parse().map_err(parse_int_error)?; let mut bcr = BytesContentRange::default().with_range(start, end); // Handle size part first. if s[1] != "*" { bcr = bcr.with_size(s[1].parse().map_err(parse_int_error)?) }; Ok(bcr) } } #[cfg(test)] mod tests { use super::*; #[test] fn test_bytes_content_range_from_str() -> Result<()> { let cases = vec![ ( "range start with unknown size", "bytes 123-123/*", BytesContentRange::default().with_range(123, 123), ), ( "range start with known size", "bytes 123-123/1024", BytesContentRange::default() .with_range(123, 123) .with_size(1024), ), ( "only have size", "bytes */1024", BytesContentRange::default().with_size(1024), ), ]; for (name, input, expected) in cases { let actual = input.parse()?; assert_eq!(expected, actual, "{name}") } Ok(()) } #[test] fn test_bytes_content_range_to_string() { let h = BytesContentRange::default().with_size(1024); assert_eq!(h.to_string(), "*/1024"); let h = BytesContentRange::default().with_range(0, 1023); assert_eq!(h.to_string(), "0-1023/*"); let h = BytesContentRange::default() .with_range(0, 1023) .with_size(1024); assert_eq!(h.to_string(), "0-1023/1024"); } #[test] fn test_bytes_content_range_to_header() { let h = BytesContentRange::default().with_size(1024); assert_eq!(h.to_header(), "bytes */1024"); let h = BytesContentRange::default().with_range(0, 1023); assert_eq!(h.to_header(), "bytes 0-1023/*"); let h = BytesContentRange::default() .with_range(0, 1023) .with_size(1024); assert_eq!(h.to_header(), "bytes 0-1023/1024"); } } opendal-0.52.0/src/raw/http_util/bytes_range.rs000064400000000000000000000174571046102023000176140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Display; use std::fmt::Formatter; use std::ops::Bound; use std::ops::RangeBounds; use std::str::FromStr; use crate::*; /// BytesRange(offset, size) carries a range of content. /// /// BytesRange implements `ToString` which can be used as `Range` HTTP header directly. /// /// `` should always be `bytes`. /// /// ```text /// Range: bytes=- /// Range: bytes=- /// ``` /// /// # Notes /// /// We don't support tailing read like `Range: bytes=-` #[derive(Default, Debug, Clone, Copy, Eq, PartialEq)] pub struct BytesRange( /// Offset of the range. u64, /// Size of the range. Option, ); impl BytesRange { /// Create a new `BytesRange` /// /// It better to use `BytesRange::from(1024..2048)` to construct. /// /// # Note /// /// The behavior for `None` and `Some(0)` is different. /// /// - offset=None => `bytes=-`, read `` bytes from end. /// - offset=Some(0) => `bytes=0-`, read `` bytes from start. pub fn new(offset: u64, size: Option) -> Self { BytesRange(offset, size) } /// Get offset of BytesRange. pub fn offset(&self) -> u64 { self.0 } /// Get size of BytesRange. pub fn size(&self) -> Option { self.1 } /// Advance the range by `n` bytes. /// /// # Panics /// /// Panic if input `n` is larger than the size of the range. pub fn advance(&mut self, n: u64) { self.0 += n; self.1 = self.1.map(|size| size - n); } /// Check if this range is full of this content. /// /// If this range is full, we don't need to specify it in http request. pub fn is_full(&self) -> bool { self.0 == 0 && self.1.is_none() } /// Convert bytes range into Range header. pub fn to_header(&self) -> String { format!("bytes={self}") } /// Convert bytes range into rust range. pub fn to_range(&self) -> impl RangeBounds { ( Bound::Included(self.0), match self.1 { Some(size) => Bound::Excluded(self.0 + size), None => Bound::Unbounded, }, ) } /// Convert bytes range into rust range with usize. pub(crate) fn to_range_as_usize(self) -> impl RangeBounds { ( Bound::Included(self.0 as usize), match self.1 { Some(size) => Bound::Excluded((self.0 + size) as usize), None => Bound::Unbounded, }, ) } } impl Display for BytesRange { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { match self.1 { None => write!(f, "{}-", self.0), Some(size) => write!(f, "{}-{}", self.0, self.0 + size - 1), } } } impl FromStr for BytesRange { type Err = Error; fn from_str(value: &str) -> Result { let s = value.strip_prefix("bytes=").ok_or_else(|| { Error::new(ErrorKind::Unexpected, "header range is invalid") .with_operation("BytesRange::from_str") .with_context("value", value) })?; if s.contains(',') { return Err(Error::new(ErrorKind::Unexpected, "header range is invalid") .with_operation("BytesRange::from_str") .with_context("value", value)); } let v = s.split('-').collect::>(); if v.len() != 2 { return Err(Error::new(ErrorKind::Unexpected, "header range is invalid") .with_operation("BytesRange::from_str") .with_context("value", value)); } let parse_int_error = |e: std::num::ParseIntError| { Error::new(ErrorKind::Unexpected, "header range is invalid") .with_operation("BytesRange::from_str") .with_context("value", value) .set_source(e) }; if v[1].is_empty() { // - Ok(BytesRange::new( v[0].parse().map_err(parse_int_error)?, None, )) } else if v[0].is_empty() { // - Err(Error::new( ErrorKind::Unexpected, "header range with tailing is not supported", ) .with_operation("BytesRange::from_str") .with_context("value", value)) } else { // - let start: u64 = v[0].parse().map_err(parse_int_error)?; let end: u64 = v[1].parse().map_err(parse_int_error)?; Ok(BytesRange::new(start, Some(end - start + 1))) } } } impl From for BytesRange where T: RangeBounds, { fn from(range: T) -> Self { let offset = match range.start_bound().cloned() { Bound::Included(n) => n, Bound::Excluded(n) => n + 1, Bound::Unbounded => 0, }; let size = match range.end_bound().cloned() { Bound::Included(n) => Some(n + 1 - offset), Bound::Excluded(n) => Some(n - offset), Bound::Unbounded => None, }; BytesRange(offset, size) } } #[cfg(test)] mod tests { use super::*; #[test] fn test_bytes_range_to_string() { let h = BytesRange::new(0, Some(1024)); assert_eq!(h.to_string(), "0-1023"); let h = BytesRange::new(1024, None); assert_eq!(h.to_string(), "1024-"); let h = BytesRange::new(1024, Some(1024)); assert_eq!(h.to_string(), "1024-2047"); } #[test] fn test_bytes_range_to_header() { let h = BytesRange::new(0, Some(1024)); assert_eq!(h.to_header(), "bytes=0-1023"); let h = BytesRange::new(1024, None); assert_eq!(h.to_header(), "bytes=1024-"); let h = BytesRange::new(1024, Some(1024)); assert_eq!(h.to_header(), "bytes=1024-2047"); } #[test] fn test_bytes_range_from_range_bounds() { assert_eq!(BytesRange::new(0, None), BytesRange::from(..)); assert_eq!(BytesRange::new(10, None), BytesRange::from(10..)); assert_eq!(BytesRange::new(0, Some(11)), BytesRange::from(..=10)); assert_eq!(BytesRange::new(0, Some(10)), BytesRange::from(..10)); assert_eq!(BytesRange::new(10, Some(10)), BytesRange::from(10..20)); assert_eq!(BytesRange::new(10, Some(11)), BytesRange::from(10..=20)); } #[test] fn test_bytes_range_from_str() -> Result<()> { let cases = vec![ ("range-start", "bytes=123-", BytesRange::new(123, None)), ("range", "bytes=123-124", BytesRange::new(123, Some(2))), ("one byte", "bytes=0-0", BytesRange::new(0, Some(1))), ( "lower case header", "bytes=0-0", BytesRange::new(0, Some(1)), ), ]; for (name, input, expected) in cases { let actual = input.parse()?; assert_eq!(expected, actual, "{name}") } Ok(()) } } opendal-0.52.0/src/raw/http_util/client.rs000064400000000000000000000172131046102023000165560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::future; use std::mem; use std::ops::Deref; use std::str::FromStr; use std::sync::Arc; use futures::Future; use futures::TryStreamExt; use http::Request; use http::Response; use once_cell::sync::Lazy; use raw::oio::Read; use super::parse_content_encoding; use super::parse_content_length; use super::HttpBody; use crate::raw::*; use crate::*; /// Http client used across opendal for loading credentials. /// This is merely a temporary solution because reqsign requires a reqwest client to be passed. /// We will remove it after the next major version of reqsign, which will enable users to provide their own client. #[allow(dead_code)] pub(crate) static GLOBAL_REQWEST_CLIENT: Lazy = Lazy::new(reqwest::Client::new); /// HttpFetcher is a type erased [`HttpFetch`]. pub type HttpFetcher = Arc; /// HttpClient that used across opendal. #[derive(Clone)] pub struct HttpClient { fetcher: HttpFetcher, } /// We don't want users to know details about our clients. impl Debug for HttpClient { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("HttpClient").finish() } } impl HttpClient { /// Create a new http client in async context. pub fn new() -> Result { let fetcher = Arc::new(reqwest::Client::new()); Ok(Self { fetcher }) } /// Construct `Self` with given [`reqwest::Client`] pub fn with(client: impl HttpFetch) -> Self { let fetcher = Arc::new(client); Self { fetcher } } /// Build a new http client in async context. #[deprecated] pub fn build(builder: reqwest::ClientBuilder) -> Result { let client = builder.build().map_err(|err| { Error::new(ErrorKind::Unexpected, "http client build failed").set_source(err) })?; let fetcher = Arc::new(client); Ok(Self { fetcher }) } /// Send a request in async way. pub async fn send(&self, req: Request) -> Result> { let (parts, mut body) = self.fetch(req).await?.into_parts(); let buffer = body.read_all().await?; Ok(Response::from_parts(parts, buffer)) } /// Fetch a request in async way. pub async fn fetch(&self, req: Request) -> Result> { self.fetcher.fetch(req).await } } /// HttpFetch is the trait to fetch a request in async way. /// User should implement this trait to provide their own http client. pub trait HttpFetch: Send + Sync + Unpin + 'static { /// Fetch a request in async way. fn fetch( &self, req: Request, ) -> impl Future>> + MaybeSend; } /// HttpFetchDyn is the dyn version of [`HttpFetch`] /// which make it possible to use as `Arc`. /// User should never implement this trait, but use `HttpFetch` instead. pub trait HttpFetchDyn: Send + Sync + Unpin + 'static { /// The dyn version of [`HttpFetch::fetch`]. /// /// This function returns a boxed future to make it object safe. fn fetch_dyn(&self, req: Request) -> BoxedFuture>>; } impl HttpFetchDyn for T { fn fetch_dyn(&self, req: Request) -> BoxedFuture>> { Box::pin(self.fetch(req)) } } impl HttpFetch for Arc { async fn fetch(&self, req: Request) -> Result> { self.deref().fetch_dyn(req).await } } impl HttpFetch for reqwest::Client { async fn fetch(&self, req: Request) -> Result> { // Uri stores all string alike data in `Bytes` which means // the clone here is cheap. let uri = req.uri().clone(); let is_head = req.method() == http::Method::HEAD; let (parts, body) = req.into_parts(); let mut req_builder = self .request( parts.method, reqwest::Url::from_str(&uri.to_string()).expect("input request url must be valid"), ) .headers(parts.headers); // Client under wasm doesn't support set version. #[cfg(not(target_arch = "wasm32"))] { req_builder = req_builder.version(parts.version); } // Don't set body if body is empty. if !body.is_empty() { #[cfg(not(target_arch = "wasm32"))] { req_builder = req_builder.body(reqwest::Body::wrap_stream(body)) } #[cfg(target_arch = "wasm32")] { req_builder = req_builder.body(reqwest::Body::from(body.to_bytes())) } } let mut resp = req_builder.send().await.map_err(|err| { Error::new(ErrorKind::Unexpected, "send http request") .with_operation("http_util::Client::send") .with_context("url", uri.to_string()) .with_temporary(is_temporary_error(&err)) .set_source(err) })?; // Get content length from header so that we can check it. // // - If the request method is HEAD, we will ignore content length. // - If response contains content_encoding, we should omit its content length. let content_length = if is_head || parse_content_encoding(resp.headers())?.is_some() { None } else { parse_content_length(resp.headers())? }; let mut hr = Response::builder() .status(resp.status()) // Insert uri into response extension so that we can fetch // it later. .extension(uri.clone()); // Response builder under wasm doesn't support set version. #[cfg(not(target_arch = "wasm32"))] { hr = hr.version(resp.version()); } // Swap headers directly instead of copy the entire map. mem::swap(hr.headers_mut().unwrap(), resp.headers_mut()); let bs = HttpBody::new( resp.bytes_stream() .try_filter(|v| future::ready(!v.is_empty())) .map_ok(Buffer::from) .map_err(move |err| { Error::new(ErrorKind::Unexpected, "read data from http response") .with_operation("http_util::Client::send") .with_context("url", uri.to_string()) .with_temporary(is_temporary_error(&err)) .set_source(err) }), content_length, ); let resp = hr.body(bs).expect("response must build succeed"); Ok(resp) } } #[inline] fn is_temporary_error(err: &reqwest::Error) -> bool { // error sending request err.is_request()|| // request or response body error err.is_body() || // error decoding response body, for example, connection reset. err.is_decode() } opendal-0.52.0/src/raw/http_util/error.rs000064400000000000000000000044701046102023000164320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::response::Parts; use http::Uri; use crate::Error; use crate::ErrorKind; /// Create a new error happened during building request. pub fn new_request_build_error(err: http::Error) -> Error { Error::new(ErrorKind::Unexpected, "building http request") .with_operation("http::Request::build") .set_source(err) } /// Create a new error happened during signing request. pub fn new_request_credential_error(err: anyhow::Error) -> Error { Error::new( ErrorKind::Unexpected, "loading credential to sign http request", ) .set_temporary() .with_operation("reqsign::LoadCredential") .set_source(err) } /// Create a new error happened during signing request. pub fn new_request_sign_error(err: anyhow::Error) -> Error { Error::new(ErrorKind::Unexpected, "signing http request") .with_operation("reqsign::Sign") .set_source(err) } /// Add response context to error. /// /// This helper function will: /// /// - remove sensitive or useless headers from parts. /// - fetch uri if parts extensions contains `Uri`. pub fn with_error_response_context(mut err: Error, mut parts: Parts) -> Error { if let Some(uri) = parts.extensions.get::() { err = err.with_context("uri", uri.to_string()); } // The following headers may contains sensitive information. parts.headers.remove("Set-Cookie"); parts.headers.remove("WWW-Authenticate"); parts.headers.remove("Proxy-Authenticate"); err = err.with_context("response", format!("{parts:?}")); err } opendal-0.52.0/src/raw/http_util/header.rs000064400000000000000000000245301046102023000165300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use base64::engine::general_purpose; use base64::Engine; use chrono::DateTime; use chrono::Utc; use http::header::CACHE_CONTROL; use http::header::CONTENT_DISPOSITION; use http::header::CONTENT_ENCODING; use http::header::CONTENT_LENGTH; use http::header::CONTENT_RANGE; use http::header::CONTENT_TYPE; use http::header::ETAG; use http::header::LAST_MODIFIED; use http::header::LOCATION; use http::HeaderMap; use http::HeaderName; use http::HeaderValue; use md5::Digest; use crate::raw::*; use crate::EntryMode; use crate::Error; use crate::ErrorKind; use crate::Metadata; use crate::Result; /// Parse redirect location from header map /// /// # Note /// The returned value maybe a relative path, like `/index.html`, `/robots.txt`, etc. pub fn parse_location(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, LOCATION) } /// Parse cache control from header map. /// /// # Note /// /// The returned value is the raw string of `cache-control` header, /// maybe `no-cache`, `max-age=3600`, etc. pub fn parse_cache_control(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, CACHE_CONTROL) } /// Parse content length from header map. pub fn parse_content_length(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, CONTENT_LENGTH)? .map(|v| { v.parse::().map_err(|e| { Error::new(ErrorKind::Unexpected, "header value is not valid integer").set_source(e) }) }) .transpose() } /// Parse content md5 from header map. pub fn parse_content_md5(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, "content-md5") } /// Parse content type from header map. pub fn parse_content_type(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, CONTENT_TYPE) } /// Parse content encoding from header map. pub fn parse_content_encoding(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, CONTENT_ENCODING) } /// Parse content range from header map. pub fn parse_content_range(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, CONTENT_RANGE)? .map(|v| v.parse()) .transpose() } /// Parse last modified from header map. pub fn parse_last_modified(headers: &HeaderMap) -> Result>> { parse_header_to_str(headers, LAST_MODIFIED)? .map(parse_datetime_from_rfc2822) .transpose() } /// Parse etag from header map. pub fn parse_etag(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, ETAG) } /// Parse Content-Disposition for header map pub fn parse_content_disposition(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, CONTENT_DISPOSITION) } /// Parse multipart boundary from header map. pub fn parse_multipart_boundary(headers: &HeaderMap) -> Result> { parse_header_to_str(headers, CONTENT_TYPE).map(|v| v.and_then(|v| v.split("boundary=").nth(1))) } /// Parse header value to string according to name. #[inline] pub fn parse_header_to_str(headers: &HeaderMap, name: K) -> Result> where HeaderName: TryFrom, { let name = HeaderName::try_from(name).map_err(|_| { Error::new( ErrorKind::Unexpected, "header name must be valid http header name but not", ) .with_operation("http_util::parse_header_to_str") })?; let value = if let Some(v) = headers.get(&name) { v } else { return Ok(None); }; Ok(Some(value.to_str().map_err(|e| { Error::new( ErrorKind::Unexpected, "header value must be valid utf-8 string but not", ) .with_operation("http_util::parse_header_to_str") .with_context("header_name", name.as_str()) .set_source(e) })?)) } /// parse_into_metadata will parse standards http headers into Metadata. /// /// # Notes /// /// parse_into_metadata only handles the standard behavior of http /// headers. If services have their own logic, they should update the parsed /// metadata on demand. pub fn parse_into_metadata(path: &str, headers: &HeaderMap) -> Result { let mode = if path.ends_with('/') { EntryMode::DIR } else { EntryMode::FILE }; let mut m = Metadata::new(mode); if let Some(v) = parse_cache_control(headers)? { m.set_cache_control(v); } if let Some(v) = parse_content_length(headers)? { m.set_content_length(v); } if let Some(v) = parse_content_type(headers)? { m.set_content_type(v); } if let Some(v) = parse_content_encoding(headers)? { m.set_content_encoding(v); } if let Some(v) = parse_content_range(headers)? { m.set_content_range(v); } if let Some(v) = parse_etag(headers)? { m.set_etag(v); } if let Some(v) = parse_content_md5(headers)? { m.set_content_md5(v); } if let Some(v) = parse_last_modified(headers)? { m.set_last_modified(v); } if let Some(v) = parse_content_disposition(headers)? { m.set_content_disposition(v); } Ok(m) } /// Parse prefixed headers and return a map with the prefix of each header removed. pub fn parse_prefixed_headers(headers: &HeaderMap, prefix: &str) -> HashMap { headers .iter() .filter_map(|(name, value)| { name.as_str().strip_prefix(prefix).and_then(|stripped_key| { value .to_str() .ok() .map(|parsed_value| (stripped_key.to_string(), parsed_value.to_string())) }) }) .collect() } /// format content md5 header by given input. pub fn format_content_md5(bs: &[u8]) -> String { let mut hasher = md5::Md5::new(); hasher.update(bs); general_purpose::STANDARD.encode(hasher.finalize()) } /// format authorization header by basic auth. /// /// # Errors /// /// If input username is empty, function will return an unexpected error. pub fn format_authorization_by_basic(username: &str, password: &str) -> Result { if username.is_empty() { return Err(Error::new( ErrorKind::Unexpected, "can't build authorization header with empty username", )); } let value = general_purpose::STANDARD.encode(format!("{username}:{password}")); Ok(format!("Basic {value}")) } /// format authorization header by bearer token. /// /// # Errors /// /// If input token is empty, function will return an unexpected error. pub fn format_authorization_by_bearer(token: &str) -> Result { if token.is_empty() { return Err(Error::new( ErrorKind::Unexpected, "can't build authorization header with empty token", )); } Ok(format!("Bearer {token}")) } /// Build header value from given string. pub fn build_header_value(v: &str) -> Result { HeaderValue::from_str(v).map_err(|e| { Error::new( ErrorKind::ConfigInvalid, "header value contains invalid characters", ) .with_operation("http_util::build_header_value") .set_source(e) }) } #[cfg(test)] mod tests { use super::*; /// Test cases is from https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html #[test] fn test_format_content_md5() { let cases = vec![( r#" sample1.txt sample2.txt "#, "WOctCY1SS662e7ziElh4cw==", )]; for (input, expected) in cases { let actual = format_content_md5(input.as_bytes()); assert_eq!(actual, expected) } } /// Test cases is borrowed from /// /// - RFC2617: https://datatracker.ietf.org/doc/html/rfc2617#section-2 /// - MDN: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization #[test] fn test_format_authorization_by_basic() { let cases = vec![ ("aladdin", "opensesame", "Basic YWxhZGRpbjpvcGVuc2VzYW1l"), ("aladdin", "", "Basic YWxhZGRpbjo="), ( "Aladdin", "open sesame", "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==", ), ("Aladdin", "", "Basic QWxhZGRpbjo="), ]; for (username, password, expected) in cases { let actual = format_authorization_by_basic(username, password).expect("format must success"); assert_eq!(actual, expected) } } /// Test cases is borrowed from /// /// - RFC6750: https://datatracker.ietf.org/doc/html/rfc6750 #[test] fn test_format_authorization_by_bearer() { let cases = vec![("mF_9.B5f-4.1JqM", "Bearer mF_9.B5f-4.1JqM")]; for (token, expected) in cases { let actual = format_authorization_by_bearer(token).expect("format must success"); assert_eq!(actual, expected) } } #[test] fn test_parse_multipart_boundary() { let cases = vec![ ( "multipart/mixed; boundary=gc0p4Jq0M2Yt08jU534c0p", Some("gc0p4Jq0M2Yt08jU534c0p"), ), ("multipart/mixed", None), ]; for (input, expected) in cases { let mut headers = HeaderMap::new(); headers.insert(CONTENT_TYPE, HeaderValue::from_str(input).unwrap()); let actual = parse_multipart_boundary(&headers).expect("parse must success"); assert_eq!(actual, expected) } } } opendal-0.52.0/src/raw/http_util/mod.rs000064400000000000000000000046611046102023000160620ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! http_util contains the util types and functions that used across OpenDAL. //! //! # NOTE //! //! This mod is not a part of OpenDAL's public API. We expose them out to make //! it easier to develop services and layers outside opendal. mod client; pub use client::HttpClient; pub use client::HttpFetch; /// temporary client used by several features #[allow(unused_imports)] pub(crate) use client::GLOBAL_REQWEST_CLIENT; mod body; pub use body::HttpBody; mod header; pub use header::build_header_value; pub use header::format_authorization_by_basic; pub use header::format_authorization_by_bearer; pub use header::format_content_md5; pub use header::parse_content_disposition; pub use header::parse_content_encoding; pub use header::parse_content_length; pub use header::parse_content_md5; pub use header::parse_content_range; pub use header::parse_content_type; pub use header::parse_etag; pub use header::parse_header_to_str; pub use header::parse_into_metadata; pub use header::parse_last_modified; pub use header::parse_location; pub use header::parse_multipart_boundary; pub use header::parse_prefixed_headers; mod uri; pub use uri::new_http_uri_invalid_error; pub use uri::percent_decode_path; pub use uri::percent_encode_path; mod error; pub use error::new_request_build_error; pub use error::new_request_credential_error; pub use error::new_request_sign_error; pub use error::with_error_response_context; mod bytes_range; pub use bytes_range::BytesRange; mod bytes_content_range; pub use bytes_content_range::BytesContentRange; mod multipart; pub use multipart::FormDataPart; pub use multipart::MixedPart; pub use multipart::Multipart; pub use multipart::Part; opendal-0.52.0/src/raw/http_util/multipart.rs000064400000000000000000001057201046102023000173220ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::mem; use std::str::FromStr; use bytes::Bytes; use bytes::BytesMut; use http::header::CONTENT_DISPOSITION; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::uri::PathAndQuery; use http::HeaderMap; use http::HeaderName; use http::HeaderValue; use http::Method; use http::Request; use http::Response; use http::StatusCode; use http::Uri; use http::Version; use super::new_request_build_error; use crate::*; /// Multipart is a builder for multipart/form-data. #[derive(Debug)] pub struct Multipart { boundary: String, parts: Vec, } impl Default for Multipart { fn default() -> Self { Self::new() } } impl Multipart { /// Create a new multipart with random boundary. pub fn new() -> Self { Multipart { boundary: format!("opendal-{}", uuid::Uuid::new_v4()), parts: Vec::default(), } } /// Set the boundary with given string. pub fn with_boundary(mut self, boundary: &str) -> Self { self.boundary = boundary.to_string(); self } /// Insert a part into multipart. pub fn part(mut self, part: T) -> Self { self.parts.push(part); self } /// Into parts. pub fn into_parts(self) -> Vec { self.parts } /// Parse a response with multipart body into Multipart. pub fn parse(mut self, bs: Bytes) -> Result { let s = String::from_utf8(bs.to_vec()).map_err(|err| { Error::new( ErrorKind::Unexpected, "multipart response contains invalid utf-8 chars", ) .set_source(err) })?; let parts = s .split(format!("--{}", self.boundary).as_str()) .collect::>(); for part in parts { if part.is_empty() || part.starts_with("--") { continue; } self.parts.push(T::parse(part)?); } Ok(self) } pub(crate) fn build(self) -> Buffer { let mut bufs = Vec::with_capacity(self.parts.len() + 2); // Build pre part. let mut bs = BytesMut::new(); bs.extend_from_slice(b"--"); bs.extend_from_slice(self.boundary.as_bytes()); bs.extend_from_slice(b"\r\n"); let pre_part = Buffer::from(bs.freeze()); // Write all parts. for part in self.parts { bufs.push(pre_part.clone()); bufs.push(part.format()); } // Write the last boundary let mut bs = BytesMut::new(); bs.extend_from_slice(b"--"); bs.extend_from_slice(self.boundary.as_bytes()); bs.extend_from_slice(b"--"); bs.extend_from_slice(b"\r\n"); // Build final part. bufs.push(Buffer::from(bs.freeze())); bufs.into_iter().flatten().collect() } /// Consume the input and generate a request with multipart body. /// /// This function will make sure content_type and content_length set correctly. pub fn apply(self, mut builder: http::request::Builder) -> Result> { let boundary = self.boundary.clone(); let buf = self.build(); let content_length = buf.len(); // Insert content type with correct boundary. builder = builder.header( CONTENT_TYPE, format!("multipart/{}; boundary={}", T::TYPE, boundary).as_str(), ); // Insert content length with calculated size. builder = builder.header(CONTENT_LENGTH, content_length); builder.body(buf).map_err(new_request_build_error) } } /// Part is a trait for multipart part. pub trait Part: Sized + 'static { /// TYPE is the type of multipart. /// /// Current available types are: `form-data` and `mixed` const TYPE: &'static str; /// format will generates the bytes. fn format(self) -> Buffer; /// parse will parse the bytes into a part. fn parse(s: &str) -> Result; } /// FormDataPart is a builder for multipart/form-data part. pub struct FormDataPart { headers: HeaderMap, content: Buffer, } impl FormDataPart { /// Create a new part builder /// /// # Panics /// /// Input name must be percent encoded. pub fn new(name: &str) -> Self { // Insert content disposition header for part. let mut headers = HeaderMap::new(); headers.insert( CONTENT_DISPOSITION, format!("form-data; name=\"{}\"", name).parse().unwrap(), ); Self { headers, content: Buffer::new(), } } /// Insert a header into part. pub fn header(mut self, key: HeaderName, value: HeaderValue) -> Self { self.headers.insert(key, value); self } /// Set the content for this part. pub fn content(mut self, content: impl Into) -> Self { self.content = content.into(); self } } impl Part for FormDataPart { const TYPE: &'static str = "form-data"; fn format(self) -> Buffer { let mut bufs = Vec::with_capacity(3); // Building pre-content. let mut bs = BytesMut::new(); for (k, v) in self.headers.iter() { // Trick! // // Seafile could not recognize header names like `content-disposition` // and requires to use `Content-Disposition`. So we hardcode the part // headers name here. match k.as_str() { "content-disposition" => { bs.extend_from_slice("Content-Disposition".as_bytes()); } _ => { bs.extend_from_slice(k.as_str().as_bytes()); } } bs.extend_from_slice(b": "); bs.extend_from_slice(v.as_bytes()); bs.extend_from_slice(b"\r\n"); } bs.extend_from_slice(b"\r\n"); bufs.push(Buffer::from(bs.freeze())); // Building content. bufs.push(self.content); // Building post-content. bufs.push(Buffer::from("\r\n")); bufs.into_iter().flatten().collect() } fn parse(_: &str) -> Result { Err(Error::new( ErrorKind::Unsupported, "parse of form-data is not supported", )) } } /// MixedPart is a builder for multipart/mixed part. pub struct MixedPart { part_headers: HeaderMap, /// Common version: Version, headers: HeaderMap, content: Buffer, /// Request only method: Option, uri: Option, /// Response only status_code: Option, } impl MixedPart { /// Create a new mixed part with given uri. pub fn new(uri: &str) -> Self { let mut part_headers = HeaderMap::new(); part_headers.insert(CONTENT_TYPE, "application/http".parse().unwrap()); part_headers.insert("content-transfer-encoding", "binary".parse().unwrap()); let uri = Uri::from_str(uri).expect("the uri used to build a mixed part must be valid"); Self { part_headers, version: Version::HTTP_11, headers: HeaderMap::new(), content: Buffer::new(), uri: Some(uri), method: None, status_code: None, } } /// Build a mixed part from a request. pub fn from_request(req: Request) -> Self { let mut part_headers = HeaderMap::new(); part_headers.insert(CONTENT_TYPE, "application/http".parse().unwrap()); part_headers.insert("content-transfer-encoding", "binary".parse().unwrap()); let (parts, content) = req.into_parts(); Self { part_headers, uri: Some( Uri::from_str( parts .uri .path_and_query() .unwrap_or(&PathAndQuery::from_static("/")) .as_str(), ) .expect("the uri used to build a mixed part must be valid"), ), version: parts.version, headers: parts.headers, content, method: Some(parts.method), status_code: None, } } /// Consume a mixed part to build a response. pub fn into_response(mut self) -> Response { let mut builder = Response::builder(); builder = builder.status(self.status_code.unwrap_or(StatusCode::OK)); builder = builder.version(self.version); // Swap headers directly instead of copy the entire map. mem::swap(builder.headers_mut().unwrap(), &mut self.headers); builder .body(self.content) .expect("mixed part must be valid response") } /// Insert a part header into part. pub fn part_header(mut self, key: HeaderName, value: HeaderValue) -> Self { self.part_headers.insert(key, value); self } /// Set the method for request in this part. pub fn method(mut self, method: Method) -> Self { self.method = Some(method); self } /// Set the version for request in this part. pub fn version(mut self, version: Version) -> Self { self.version = version; self } /// Insert a header into part. pub fn header(mut self, key: HeaderName, value: HeaderValue) -> Self { self.headers.insert(key, value); self } /// Set the content for this part. pub fn content(mut self, content: impl Into) -> Self { self.content = content.into(); self } } impl Part for MixedPart { const TYPE: &'static str = "mixed"; fn format(self) -> Buffer { let mut bufs = Vec::with_capacity(3); // Write parts headers. let mut bs = BytesMut::new(); for (k, v) in self.part_headers.iter() { // Trick! // // Azblob could not recognize header names like `content-type` // and requires to use `Content-Type`. So we hardcode the part // headers name here. match k.as_str() { "content-type" => { bs.extend_from_slice("Content-Type".as_bytes()); } "content-id" => { bs.extend_from_slice("Content-ID".as_bytes()); } "content-transfer-encoding" => { bs.extend_from_slice("Content-Transfer-Encoding".as_bytes()); } _ => { bs.extend_from_slice(k.as_str().as_bytes()); } } bs.extend_from_slice(b": "); bs.extend_from_slice(v.as_bytes()); bs.extend_from_slice(b"\r\n"); } // Write request line: `DELETE /container0/blob0 HTTP/1.1` bs.extend_from_slice(b"\r\n"); bs.extend_from_slice( self.method .as_ref() .expect("mixed part must be a valid request that contains method") .as_str() .as_bytes(), ); bs.extend_from_slice(b" "); bs.extend_from_slice( self.uri .as_ref() .expect("mixed part must be a valid request that contains uri") .path() .as_bytes(), ); bs.extend_from_slice(b" "); bs.extend_from_slice(format!("{:?}", self.version).as_bytes()); bs.extend_from_slice(b"\r\n"); // Write request headers. for (k, v) in self.headers.iter() { bs.extend_from_slice(k.as_str().as_bytes()); bs.extend_from_slice(b": "); bs.extend_from_slice(v.as_bytes()); bs.extend_from_slice(b"\r\n"); } bs.extend_from_slice(b"\r\n"); bufs.push(Buffer::from(bs.freeze())); if !self.content.is_empty() { bufs.push(self.content); bufs.push(Buffer::from("\r\n")) } bufs.into_iter().flatten().collect() } /// TODO /// /// This is a simple implementation and have a lot of space to improve. fn parse(s: &str) -> Result { let parts = s.splitn(2, "\r\n\r\n").collect::>(); let part_headers_content = parts[0]; let http_response = parts.get(1).unwrap_or(&""); let mut part_headers = HeaderMap::new(); for line in part_headers_content.lines() { let parts = line.splitn(2, ": ").collect::>(); if parts.len() == 2 { let header_name = HeaderName::from_str(parts[0]).map_err(|err| { Error::new( ErrorKind::Unexpected, "multipart response contains invalid part header name", ) .set_source(err) })?; let header_value = parts[1].parse().map_err(|err| { Error::new( ErrorKind::Unexpected, "multipart response contains invalid part header value", ) .set_source(err) })?; part_headers.insert(header_name, header_value); } } let parts = http_response.split("\r\n\r\n").collect::>(); let headers_content = parts[0]; let body_content = parts.get(1).unwrap_or(&""); let body_bytes = Buffer::from(body_content.to_string()); let status_line = headers_content.lines().next().unwrap_or(""); let status_code = status_line .split_whitespace() .nth(1) .unwrap_or("") .parse::() .unwrap_or(200); let mut headers = HeaderMap::new(); for line in headers_content.lines().skip(1) { let parts = line.splitn(2, ": ").collect::>(); if parts.len() == 2 { let header_name = HeaderName::from_str(parts[0]).map_err(|err| { Error::new( ErrorKind::Unexpected, "multipart response contains invalid part header name", ) .set_source(err) })?; let header_value = parts[1].parse().map_err(|err| { Error::new( ErrorKind::Unexpected, "multipart response contains invalid part header value", ) .set_source(err) })?; headers.insert(header_name, header_value); } } Ok(Self { part_headers, version: Version::HTTP_11, headers, content: body_bytes, method: None, uri: None, status_code: Some(StatusCode::from_u16(status_code).map_err(|err| { Error::new( ErrorKind::Unexpected, "multipart response contains invalid status code", ) .set_source(err) })?), }) } } #[cfg(test)] mod tests { use http::header::CONTENT_TYPE; use pretty_assertions::assert_eq; use super::*; #[test] fn test_multipart_formdata_basic() -> Result<()> { let multipart = Multipart::new() .with_boundary("lalala") .part(FormDataPart::new("foo").content(Bytes::from("bar"))) .part(FormDataPart::new("hello").content(Bytes::from("world"))); let bs = multipart.build(); let expected = "--lalala\r\n\ Content-Disposition: form-data; name=\"foo\"\r\n\ \r\n\ bar\r\n\ --lalala\r\n\ Content-Disposition: form-data; name=\"hello\"\r\n\ \r\n\ world\r\n\ --lalala--\r\n"; assert_eq!(Bytes::from(expected), bs.to_bytes()); Ok(()) } /// This test is inspired by #[test] fn test_multipart_formdata_s3_form_upload() -> Result<()> { let multipart = Multipart::new() .with_boundary("9431149156168") .part(FormDataPart::new("key").content("user/eric/MyPicture.jpg")) .part(FormDataPart::new("acl").content("public-read")) .part(FormDataPart::new("success_action_redirect").content( "https://awsexamplebucket1.s3.us-west-1.amazonaws.com/successful_upload.html", )) .part(FormDataPart::new("content-type").content("image/jpeg")) .part(FormDataPart::new("x-amz-meta-uuid").content("14365123651274")) .part(FormDataPart::new("x-amz-meta-tag").content("Some,Tag,For,Picture")) .part(FormDataPart::new("AWSAccessKeyId").content("AKIAIOSFODNN7EXAMPLE")) .part(FormDataPart::new("Policy").content("eyAiZXhwaXJhdGlvbiI6ICIyMDA3LTEyLTAxVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiam9obnNtaXRoIn0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAidXNlci9lcmljLyJdLAogICAgeyJhY2wiOiAicHVibGljLXJlYWQifSwKICAgIHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiAiaHR0cDovL2pvaG5zbWl0aC5zMy5hbWF6b25hd3MuY29tL3N1Y2Nlc3NmdWxfdXBsb2FkLmh0bWwifSwKICAgIFsic3RhcnRzLXdpdGgiLCAiJENvbnRlbnQtVHlwZSIsICJpbWFnZS8iXSwKICAgIHsieC1hbXotbWV0YS11dWlkIjogIjE0MzY1MTIzNjUxMjc0In0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiR4LWFtei1tZXRhLXRhZyIsICIiXQogIF0KfQo=")) .part(FormDataPart::new("Signature").content("0RavWzkygo6QX9caELEqKi9kDbU=")) .part(FormDataPart::new("file").header(CONTENT_TYPE, "image/jpeg".parse().unwrap()).content("...file content...")).part(FormDataPart::new("submit").content("Upload to Amazon S3")); let bs = multipart.build(); let expected = r#"--9431149156168 Content-Disposition: form-data; name="key" user/eric/MyPicture.jpg --9431149156168 Content-Disposition: form-data; name="acl" public-read --9431149156168 Content-Disposition: form-data; name="success_action_redirect" https://awsexamplebucket1.s3.us-west-1.amazonaws.com/successful_upload.html --9431149156168 Content-Disposition: form-data; name="content-type" image/jpeg --9431149156168 Content-Disposition: form-data; name="x-amz-meta-uuid" 14365123651274 --9431149156168 Content-Disposition: form-data; name="x-amz-meta-tag" Some,Tag,For,Picture --9431149156168 Content-Disposition: form-data; name="AWSAccessKeyId" AKIAIOSFODNN7EXAMPLE --9431149156168 Content-Disposition: form-data; name="Policy" eyAiZXhwaXJhdGlvbiI6ICIyMDA3LTEyLTAxVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIjogWwogICAgeyJidWNrZXQiOiAiam9obnNtaXRoIn0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiRrZXkiLCAidXNlci9lcmljLyJdLAogICAgeyJhY2wiOiAicHVibGljLXJlYWQifSwKICAgIHsic3VjY2Vzc19hY3Rpb25fcmVkaXJlY3QiOiAiaHR0cDovL2pvaG5zbWl0aC5zMy5hbWF6b25hd3MuY29tL3N1Y2Nlc3NmdWxfdXBsb2FkLmh0bWwifSwKICAgIFsic3RhcnRzLXdpdGgiLCAiJENvbnRlbnQtVHlwZSIsICJpbWFnZS8iXSwKICAgIHsieC1hbXotbWV0YS11dWlkIjogIjE0MzY1MTIzNjUxMjc0In0sCiAgICBbInN0YXJ0cy13aXRoIiwgIiR4LWFtei1tZXRhLXRhZyIsICIiXQogIF0KfQo= --9431149156168 Content-Disposition: form-data; name="Signature" 0RavWzkygo6QX9caELEqKi9kDbU= --9431149156168 Content-Disposition: form-data; name="file" content-type: image/jpeg ...file content... --9431149156168 Content-Disposition: form-data; name="submit" Upload to Amazon S3 --9431149156168-- "#; assert_eq!( expected, // Rust can't represent `\r` in a string literal, so we // replace `\r\n` with `\n` for comparison String::from_utf8(bs.to_bytes().to_vec()) .unwrap() .replace("\r\n", "\n") ); Ok(()) } /// This test is inspired by #[test] fn test_multipart_mixed_gcs_batch_metadata() -> Result<()> { let multipart = Multipart::new() .with_boundary("===============7330845974216740156==") .part( MixedPart::new("/storage/v1/b/example-bucket/o/obj1") .method(Method::PATCH) .part_header( "content-id".parse().unwrap(), "".parse().unwrap(), ) .header( "content-type".parse().unwrap(), "application/json".parse().unwrap(), ) .header( "accept".parse().unwrap(), "application/json".parse().unwrap(), ) .header("content-length".parse().unwrap(), "31".parse().unwrap()) .content(r#"{"metadata": {"type": "tabby"}}"#), ) .part( MixedPart::new("/storage/v1/b/example-bucket/o/obj2") .method(Method::PATCH) .part_header( "content-id".parse().unwrap(), "".parse().unwrap(), ) .header( "content-type".parse().unwrap(), "application/json".parse().unwrap(), ) .header( "accept".parse().unwrap(), "application/json".parse().unwrap(), ) .header("content-length".parse().unwrap(), "32".parse().unwrap()) .content(r#"{"metadata": {"type": "tuxedo"}}"#), ) .part( MixedPart::new("/storage/v1/b/example-bucket/o/obj3") .method(Method::PATCH) .part_header( "content-id".parse().unwrap(), "".parse().unwrap(), ) .header( "content-type".parse().unwrap(), "application/json".parse().unwrap(), ) .header( "accept".parse().unwrap(), "application/json".parse().unwrap(), ) .header("content-length".parse().unwrap(), "32".parse().unwrap()) .content(r#"{"metadata": {"type": "calico"}}"#), ); let bs = multipart.build(); let expected = r#"--===============7330845974216740156== Content-Type: application/http Content-Transfer-Encoding: binary Content-ID: PATCH /storage/v1/b/example-bucket/o/obj1 HTTP/1.1 content-type: application/json accept: application/json content-length: 31 {"metadata": {"type": "tabby"}} --===============7330845974216740156== Content-Type: application/http Content-Transfer-Encoding: binary Content-ID: PATCH /storage/v1/b/example-bucket/o/obj2 HTTP/1.1 content-type: application/json accept: application/json content-length: 32 {"metadata": {"type": "tuxedo"}} --===============7330845974216740156== Content-Type: application/http Content-Transfer-Encoding: binary Content-ID: PATCH /storage/v1/b/example-bucket/o/obj3 HTTP/1.1 content-type: application/json accept: application/json content-length: 32 {"metadata": {"type": "calico"}} --===============7330845974216740156==-- "#; assert_eq!( expected, // Rust can't represent `\r` in a string literal, so we // replace `\r\n` with `\n` for comparison String::from_utf8(bs.to_bytes().to_vec()) .unwrap() .replace("\r\n", "\n") ); Ok(()) } /// This test is inspired by #[test] fn test_multipart_mixed_azblob_batch_delete() -> Result<()> { let multipart = Multipart::new() .with_boundary("batch_357de4f7-6d0b-4e02-8cd2-6361411a9525") .part( MixedPart::new("/container0/blob0") .method(Method::DELETE) .part_header("content-id".parse().unwrap(), "0".parse().unwrap()) .header( "x-ms-date".parse().unwrap(), "Thu, 14 Jun 2018 16:46:54 GMT".parse().unwrap(), ) .header( "authorization".parse().unwrap(), "SharedKey account:G4jjBXA7LI/RnWKIOQ8i9xH4p76pAQ+4Fs4R1VxasaE=" .parse() .unwrap(), ) .header("content-length".parse().unwrap(), "0".parse().unwrap()), ) .part( MixedPart::new("/container1/blob1") .method(Method::DELETE) .part_header("content-id".parse().unwrap(), "1".parse().unwrap()) .header( "x-ms-date".parse().unwrap(), "Thu, 14 Jun 2018 16:46:54 GMT".parse().unwrap(), ) .header( "authorization".parse().unwrap(), "SharedKey account:IvCoYDQ+0VcaA/hKFjUmQmIxXv2RT3XwwTsOTHL39HI=" .parse() .unwrap(), ) .header("content-length".parse().unwrap(), "0".parse().unwrap()), ) .part( MixedPart::new("/container2/blob2") .method(Method::DELETE) .part_header("content-id".parse().unwrap(), "2".parse().unwrap()) .header( "x-ms-date".parse().unwrap(), "Thu, 14 Jun 2018 16:46:54 GMT".parse().unwrap(), ) .header( "authorization".parse().unwrap(), "SharedKey account:S37N2JTjcmOQVLHLbDmp2johz+KpTJvKhbVc4M7+UqI=" .parse() .unwrap(), ) .header("content-length".parse().unwrap(), "0".parse().unwrap()), ); let bs = multipart.build(); let expected = r#"--batch_357de4f7-6d0b-4e02-8cd2-6361411a9525 Content-Type: application/http Content-Transfer-Encoding: binary Content-ID: 0 DELETE /container0/blob0 HTTP/1.1 x-ms-date: Thu, 14 Jun 2018 16:46:54 GMT authorization: SharedKey account:G4jjBXA7LI/RnWKIOQ8i9xH4p76pAQ+4Fs4R1VxasaE= content-length: 0 --batch_357de4f7-6d0b-4e02-8cd2-6361411a9525 Content-Type: application/http Content-Transfer-Encoding: binary Content-ID: 1 DELETE /container1/blob1 HTTP/1.1 x-ms-date: Thu, 14 Jun 2018 16:46:54 GMT authorization: SharedKey account:IvCoYDQ+0VcaA/hKFjUmQmIxXv2RT3XwwTsOTHL39HI= content-length: 0 --batch_357de4f7-6d0b-4e02-8cd2-6361411a9525 Content-Type: application/http Content-Transfer-Encoding: binary Content-ID: 2 DELETE /container2/blob2 HTTP/1.1 x-ms-date: Thu, 14 Jun 2018 16:46:54 GMT authorization: SharedKey account:S37N2JTjcmOQVLHLbDmp2johz+KpTJvKhbVc4M7+UqI= content-length: 0 --batch_357de4f7-6d0b-4e02-8cd2-6361411a9525-- "#; assert_eq!( expected, // Rust can't represent `\r` in a string literal, so we // replace `\r\n` with `\n` for comparison String::from_utf8(bs.to_bytes().to_vec()) .unwrap() .replace("\r\n", "\n") ); Ok(()) } /// This test is inspired by #[test] fn test_multipart_mixed_gcs_batch_metadata_response() { let response = r#"--batch_pK7JBAk73-E=_AA5eFwv4m2Q= Content-Type: application/http Content-ID: HTTP/1.1 200 OK ETag: "lGaP-E0memYDumK16YuUDM_6Gf0/V43j6azD55CPRGb9b6uytDYl61Y" Content-Type: application/json; charset=UTF-8 Date: Mon, 22 Jan 2018 18:56:00 GMT Expires: Mon, 22 Jan 2018 18:56:00 GMT Cache-Control: private, max-age=0 Content-Length: 846 {"kind": "storage#object","id": "example-bucket/obj1/1495822576643790","metadata": {"type": "tabby"}} --batch_pK7JBAk73-E=_AA5eFwv4m2Q= Content-Type: application/http Content-ID: HTTP/1.1 200 OK ETag: "lGaP-E0memYDumK16YuUDM_6Gf0/91POdd-sxSAkJnS8Dm7wMxBSDKk" Content-Type: application/json; charset=UTF-8 Date: Mon, 22 Jan 2018 18:56:00 GMT Expires: Mon, 22 Jan 2018 18:56:00 GMT Cache-Control: private, max-age=0 Content-Length: 846 {"kind": "storage#object","id": "example-bucket/obj2/1495822576643790","metadata": {"type": "tuxedo"}} --batch_pK7JBAk73-E=_AA5eFwv4m2Q= Content-Type: application/http Content-ID: HTTP/1.1 200 OK ETag: "lGaP-E0memYDumK16YuUDM_6Gf0/d2Z1F1_ZVbB1dC0YKM9rX5VAgIQ" Content-Type: application/json; charset=UTF-8 Date: Mon, 22 Jan 2018 18:56:00 GMT Expires: Mon, 22 Jan 2018 18:56:00 GMT Cache-Control: private, max-age=0 Content-Length: 846 {"kind": "storage#object","id": "example-bucket/obj3/1495822576643790","metadata": {"type": "calico"}} --batch_pK7JBAk73-E=_AA5eFwv4m2Q=--"#.replace('\n', "\r\n"); let multipart: Multipart = Multipart::new() .with_boundary("batch_pK7JBAk73-E=_AA5eFwv4m2Q=") .parse(Bytes::from(response)) .unwrap(); let part0_bs = Bytes::from_static( r#"{"kind": "storage#object","id": "example-bucket/obj1/1495822576643790","metadata": {"type": "tabby"}}"#.as_bytes()); let part1_bs = Bytes::from_static( r#"{"kind": "storage#object","id": "example-bucket/obj2/1495822576643790","metadata": {"type": "tuxedo"}}"# .as_bytes() ); let part2_bs = Bytes::from_static( r#"{"kind": "storage#object","id": "example-bucket/obj3/1495822576643790","metadata": {"type": "calico"}}"# .as_bytes() ); assert_eq!(multipart.parts.len(), 3); assert_eq!(multipart.parts[0].part_headers, { let mut h = HeaderMap::new(); h.insert("Content-Type", "application/http".parse().unwrap()); h.insert( "Content-ID", "" .parse() .unwrap(), ); h }); assert_eq!(multipart.parts[0].version, Version::HTTP_11); assert_eq!(multipart.parts[0].headers, { let mut h = HeaderMap::new(); h.insert( "ETag", "\"lGaP-E0memYDumK16YuUDM_6Gf0/V43j6azD55CPRGb9b6uytDYl61Y\"" .parse() .unwrap(), ); h.insert( "Content-Type", "application/json; charset=UTF-8".parse().unwrap(), ); h.insert("Date", "Mon, 22 Jan 2018 18:56:00 GMT".parse().unwrap()); h.insert("Expires", "Mon, 22 Jan 2018 18:56:00 GMT".parse().unwrap()); h.insert("Cache-Control", "private, max-age=0".parse().unwrap()); h.insert("Content-Length", "846".parse().unwrap()); h }); assert_eq!(multipart.parts[0].content.len(), part0_bs.len()); assert_eq!(multipart.parts[0].uri, None); assert_eq!(multipart.parts[0].method, None); assert_eq!( multipart.parts[0].status_code, Some(StatusCode::from_u16(200).unwrap()) ); assert_eq!(multipart.parts[1].part_headers, { let mut h = HeaderMap::new(); h.insert("Content-Type", "application/http".parse().unwrap()); h.insert( "Content-ID", "" .parse() .unwrap(), ); h }); assert_eq!(multipart.parts[1].version, Version::HTTP_11); assert_eq!(multipart.parts[1].headers, { let mut h = HeaderMap::new(); h.insert( "ETag", "\"lGaP-E0memYDumK16YuUDM_6Gf0/91POdd-sxSAkJnS8Dm7wMxBSDKk\"" .parse() .unwrap(), ); h.insert( "Content-Type", "application/json; charset=UTF-8".parse().unwrap(), ); h.insert("Date", "Mon, 22 Jan 2018 18:56:00 GMT".parse().unwrap()); h.insert("Expires", "Mon, 22 Jan 2018 18:56:00 GMT".parse().unwrap()); h.insert("Cache-Control", "private, max-age=0".parse().unwrap()); h.insert("Content-Length", "846".parse().unwrap()); h }); assert_eq!(multipart.parts[1].content.len(), part1_bs.len()); assert_eq!(multipart.parts[1].uri, None); assert_eq!(multipart.parts[1].method, None); assert_eq!( multipart.parts[1].status_code, Some(StatusCode::from_u16(200).unwrap()) ); assert_eq!(multipart.parts[2].part_headers, { let mut h = HeaderMap::new(); h.insert("Content-Type", "application/http".parse().unwrap()); h.insert( "Content-ID", "" .parse() .unwrap(), ); h }); assert_eq!(multipart.parts[2].version, Version::HTTP_11); assert_eq!(multipart.parts[2].headers, { let mut h = HeaderMap::new(); h.insert( "ETag", "\"lGaP-E0memYDumK16YuUDM_6Gf0/d2Z1F1_ZVbB1dC0YKM9rX5VAgIQ\"" .parse() .unwrap(), ); h.insert( "Content-Type", "application/json; charset=UTF-8".parse().unwrap(), ); h.insert("Date", "Mon, 22 Jan 2018 18:56:00 GMT".parse().unwrap()); h.insert("Expires", "Mon, 22 Jan 2018 18:56:00 GMT".parse().unwrap()); h.insert("Cache-Control", "private, max-age=0".parse().unwrap()); h.insert("Content-Length", "846".parse().unwrap()); h }); assert_eq!(multipart.parts[2].content.len(), part2_bs.len()); assert_eq!(multipart.parts[2].uri, None); assert_eq!(multipart.parts[2].method, None); assert_eq!( multipart.parts[2].status_code, Some(StatusCode::from_u16(200).unwrap()) ); } } opendal-0.52.0/src/raw/http_util/uri.rs000064400000000000000000000111771046102023000161020ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::*; use percent_encoding::percent_decode_str; use percent_encoding::utf8_percent_encode; use percent_encoding::AsciiSet; use percent_encoding::NON_ALPHANUMERIC; /// Parse http uri invalid error in to opendal::Error. pub fn new_http_uri_invalid_error(err: http::uri::InvalidUri) -> Error { Error::new(ErrorKind::Unexpected, "parse http uri").set_source(err) } /// PATH_ENCODE_SET is the encode set for http url path. /// /// This set follows [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) which will encode all non-ASCII characters except `A-Z a-z 0-9 - _ . ! ~ * ' ( )` /// /// There is a special case for `/` in path: we will allow `/` in path as /// required by storage services like s3. static PATH_ENCODE_SET: AsciiSet = NON_ALPHANUMERIC .remove(b'/') .remove(b'-') .remove(b'_') .remove(b'.') .remove(b'!') .remove(b'~') .remove(b'*') .remove(b'\'') .remove(b'(') .remove(b')'); /// percent_encode_path will do percent encoding for http encode path. /// /// Follows [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) which will encode all non-ASCII characters except `A-Z a-z 0-9 - _ . ! ~ * ' ( )` /// /// There is a special case for `/` in path: we will allow `/` in path as /// required by storage services like s3. pub fn percent_encode_path(path: &str) -> String { utf8_percent_encode(path, &PATH_ENCODE_SET).to_string() } /// percent_decode_path will do percent decoding for http decode path. /// /// If the input is not percent encoded or not valid utf8, return the input. pub fn percent_decode_path(path: &str) -> String { match percent_decode_str(path).decode_utf8() { Ok(v) => v.to_string(), Err(_) => path.to_string(), } } #[cfg(test)] mod tests { use super::*; #[test] fn test_percent_encode_path() { let cases = vec![ ( "Reserved Characters", ";,/?:@&=+$", "%3B%2C/%3F%3A%40%26%3D%2B%24", ), ("Unescaped Characters", "-_.!~*'()", "-_.!~*'()"), ("Number Sign", "#", "%23"), ( "Alphanumeric Characters + Space", "ABC abc 123", "ABC%20abc%20123", ), ( "Unicode", "你好，世界！❤", "%E4%BD%A0%E5%A5%BD%EF%BC%8C%E4%B8%96%E7%95%8C%EF%BC%81%E2%9D%A4", ), ]; for (name, input, expected) in cases { let actual = percent_encode_path(input); assert_eq!(actual, expected, "{name}"); } } #[test] fn test_percent_decode_path() { let cases = vec![ ( "Reserved Characters", "%3B%2C/%3F%3A%40%26%3D%2B%24", ";,/?:@&=+$", ), ("Unescaped Characters", "-_.!~*'()", "-_.!~*'()"), ("Number Sign", "%23", "#"), ( "Alphanumeric Characters + Space", "ABC%20abc%20123", "ABC abc 123", ), ( "Unicode Characters", "%E4%BD%A0%E5%A5%BD%EF%BC%8C%E4%B8%96%E7%95%8C%EF%BC%81%E2%9D%A4", "你好，世界！❤", ), ( "Double Encoded Characters", "Double%2520Encoded", "Double%20Encoded", ), ( "Not Percent Encoded Characters", "/not percent encoded/path;,/?:@&=+$-", "/not percent encoded/path;,/?:@&=+$-", ), ]; for (name, input, expected) in cases { let actual = percent_decode_path(input); assert_eq!(actual, expected, "{name}"); } } } opendal-0.52.0/src/raw/layer.rs000064400000000000000000000272001046102023000143750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::sync::Arc; use futures::Future; use crate::raw::*; use crate::*; /// Layer is used to intercept the operations on the underlying storage. /// /// Struct that implement this trait must accept input `Arc` as inner, /// and returns a new `Arc` as output. /// /// All functions in `Accessor` requires `&self`, so it's implementer's responsibility /// to maintain the internal mutability. Please also keep in mind that `Accessor` /// requires `Send` and `Sync`. /// /// # Notes /// /// ## Inner /// /// It's required to implement `fn inner() -> Option>` for layer's accessors. /// /// By implement this method, all API calls will be forwarded to inner accessor instead. /// /// # Examples /// /// ``` /// use std::sync::Arc; /// /// use opendal::raw::*; /// use opendal::*; /// /// /// Implement the real accessor logic here. /// #[derive(Debug)] /// struct TraceAccessor { /// inner: A, /// } /// /// impl LayeredAccess for TraceAccessor { /// type Inner = A; /// type Reader = A::Reader; /// type BlockingReader = A::BlockingReader; /// type Writer = A::Writer; /// type BlockingWriter = A::BlockingWriter; /// type Lister = A::Lister; /// type BlockingLister = A::BlockingLister; /// type Deleter = A::Deleter; /// type BlockingDeleter = A::BlockingDeleter; /// /// fn inner(&self) -> &Self::Inner { /// &self.inner /// } /// /// async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { /// self.inner.read(path, args).await /// } /// /// fn blocking_read( /// &self, /// path: &str, /// args: OpRead, /// ) -> Result<(RpRead, Self::BlockingReader)> { /// self.inner.blocking_read(path, args) /// } /// /// async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { /// self.inner.write(path, args).await /// } /// /// fn blocking_write( /// &self, /// path: &str, /// args: OpWrite, /// ) -> Result<(RpWrite, Self::BlockingWriter)> { /// self.inner.blocking_write(path, args) /// } /// /// async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { /// self.inner.list(path, args).await /// } /// /// fn blocking_list( /// &self, /// path: &str, /// args: OpList, /// ) -> Result<(RpList, Self::BlockingLister)> { /// self.inner.blocking_list(path, args) /// } /// /// async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { /// self.inner.delete().await /// } /// /// fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { /// self.inner.blocking_delete() /// } /// } /// /// /// The public struct that exposed to users. /// /// /// /// Will be used like `op.layer(TraceLayer)` /// struct TraceLayer; /// /// impl Layer for TraceLayer { /// type LayeredAccess = TraceAccessor; /// /// fn layer(&self, inner: A) -> Self::LayeredAccess { /// TraceAccessor { inner } /// } /// } /// ``` pub trait Layer { /// The layered accessor that returned by this layer. type LayeredAccess: Access; /// Intercept the operations on the underlying storage. fn layer(&self, inner: A) -> Self::LayeredAccess; } /// LayeredAccess is layered accessor that forward all not implemented /// method to inner. #[allow(missing_docs)] pub trait LayeredAccess: Send + Sync + Debug + Unpin + 'static { type Inner: Access; type Reader: oio::Read; type Writer: oio::Write; type Lister: oio::List; type Deleter: oio::Delete; type BlockingReader: oio::BlockingRead; type BlockingWriter: oio::BlockingWrite; type BlockingLister: oio::BlockingList; type BlockingDeleter: oio::BlockingDelete; fn inner(&self) -> &Self::Inner; fn info(&self) -> Arc { self.inner().info() } fn create_dir( &self, path: &str, args: OpCreateDir, ) -> impl Future> + MaybeSend { self.inner().create_dir(path, args) } fn read( &self, path: &str, args: OpRead, ) -> impl Future> + MaybeSend; fn write( &self, path: &str, args: OpWrite, ) -> impl Future> + MaybeSend; fn copy( &self, from: &str, to: &str, args: OpCopy, ) -> impl Future> + MaybeSend { self.inner().copy(from, to, args) } fn rename( &self, from: &str, to: &str, args: OpRename, ) -> impl Future> + MaybeSend { self.inner().rename(from, to, args) } fn stat(&self, path: &str, args: OpStat) -> impl Future> + MaybeSend { self.inner().stat(path, args) } fn delete(&self) -> impl Future> + MaybeSend; fn list( &self, path: &str, args: OpList, ) -> impl Future> + MaybeSend; fn presign( &self, path: &str, args: OpPresign, ) -> impl Future> + MaybeSend { self.inner().presign(path, args) } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { self.inner().blocking_create_dir(path, args) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)>; fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)>; fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { self.inner().blocking_copy(from, to, args) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { self.inner().blocking_rename(from, to, args) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { self.inner().blocking_stat(path, args) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)>; fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)>; } impl Access for L { type Reader = L::Reader; type Writer = L::Writer; type Lister = L::Lister; type Deleter = L::Deleter; type BlockingReader = L::BlockingReader; type BlockingWriter = L::BlockingWriter; type BlockingLister = L::BlockingLister; type BlockingDeleter = L::BlockingDeleter; fn info(&self) -> Arc { LayeredAccess::info(self) } async fn create_dir(&self, path: &str, args: OpCreateDir) -> Result { LayeredAccess::create_dir(self, path, args).await } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { LayeredAccess::read(self, path, args).await } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { LayeredAccess::write(self, path, args).await } async fn copy(&self, from: &str, to: &str, args: OpCopy) -> Result { LayeredAccess::copy(self, from, to, args).await } async fn rename(&self, from: &str, to: &str, args: OpRename) -> Result { LayeredAccess::rename(self, from, to, args).await } async fn stat(&self, path: &str, args: OpStat) -> Result { LayeredAccess::stat(self, path, args).await } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { LayeredAccess::delete(self).await } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { LayeredAccess::list(self, path, args).await } async fn presign(&self, path: &str, args: OpPresign) -> Result { LayeredAccess::presign(self, path, args).await } fn blocking_create_dir(&self, path: &str, args: OpCreateDir) -> Result { LayeredAccess::blocking_create_dir(self, path, args) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { LayeredAccess::blocking_read(self, path, args) } fn blocking_write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { LayeredAccess::blocking_write(self, path, args) } fn blocking_copy(&self, from: &str, to: &str, args: OpCopy) -> Result { LayeredAccess::blocking_copy(self, from, to, args) } fn blocking_rename(&self, from: &str, to: &str, args: OpRename) -> Result { LayeredAccess::blocking_rename(self, from, to, args) } fn blocking_stat(&self, path: &str, args: OpStat) -> Result { LayeredAccess::blocking_stat(self, path, args) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { LayeredAccess::blocking_delete(self) } fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingLister)> { LayeredAccess::blocking_list(self, path, args) } } #[cfg(test)] mod tests { use std::sync::Arc; use futures::lock::Mutex; use super::*; use crate::services::Memory; #[derive(Debug)] struct Test { #[allow(dead_code)] inner: Option, stated: Arc>, } impl Layer for &Test { type LayeredAccess = Test; fn layer(&self, inner: A) -> Self::LayeredAccess { Test { inner: Some(inner), stated: self.stated.clone(), } } } impl Access for Test { type Reader = (); type BlockingReader = (); type Writer = (); type BlockingWriter = (); type Lister = (); type BlockingLister = (); type Deleter = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Custom("test")); am.into() } async fn stat(&self, _: &str, _: OpStat) -> Result { let mut x = self.stated.lock().await; *x = true; assert!(self.inner.is_some()); // We will not call anything here to test the layer. Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } } #[tokio::test] async fn test_layer() { let test = Test { inner: None, stated: Arc::new(Mutex::new(false)), }; let op = Operator::new(Memory::default()) .unwrap() .layer(&test) .finish(); op.stat("xxxxx").await.unwrap(); assert!(*test.stated.clone().lock().await); } } opendal-0.52.0/src/raw/mod.rs000064400000000000000000000043001046102023000140340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Raw modules provide raw APIs that used by underlying services //! //! ## Notes //! //! - Only developers who want to develop new services or layers need to //! access raw APIs. //! - Raw APIs should only be accessed via `opendal::raw::Xxxx`, any public //! API should never expose raw API directly. //! - Raw APIs are far less stable than public API, please don't rely on //! them whenever possible. mod accessor; pub use accessor::*; mod layer; pub use layer::*; mod path; pub use path::*; #[cfg(feature = "internal-path-cache")] mod path_cache; #[cfg(feature = "internal-path-cache")] pub use path_cache::*; mod operation; pub use operation::*; mod version; pub use version::VERSION; mod rps; pub use rps::*; mod ops; pub use ops::*; mod http_util; pub use http_util::*; mod serde_util; pub use serde_util::*; mod chrono_util; pub use chrono_util::*; #[cfg(feature = "internal-tokio-rt")] mod tokio_util; #[cfg(feature = "internal-tokio-rt")] pub use tokio_util::*; mod std_io_util; pub use std_io_util::*; mod futures_util; pub use futures_util::BoxedFuture; pub use futures_util::BoxedStaticFuture; pub use futures_util::ConcurrentFutures; pub use futures_util::ConcurrentTasks; pub use futures_util::MaybeSend; mod enum_utils; pub use enum_utils::*; mod atomic_util; pub use atomic_util::*; // Expose as a pub mod to avoid confusing. pub mod adapters; pub mod oio; #[cfg(feature = "tests")] pub mod tests; opendal-0.52.0/src/raw/oio/buf/flex_buf.rs000064400000000000000000000066031046102023000164210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use bytes::BufMut; use bytes::Bytes; use bytes::BytesMut; /// FlexBuf is a buffer that support frozen bytes and reuse existing allocated memory. /// /// It's useful when we want to freeze the buffer and reuse the memory for the next buffer. pub struct FlexBuf { /// Already allocated memory size of `buf`. cap: usize, /// Already written bytes length inside `buf`. len: usize, buf: BytesMut, frozen: Option, } impl FlexBuf { /// Initializes a new `FlexBuf` with the given capacity. pub fn new(cap: usize) -> Self { FlexBuf { cap, len: 0, buf: BytesMut::with_capacity(cap), frozen: None, } } /// Put slice into flex buf. /// /// Return 0 means the buffer is frozen. pub fn put(&mut self, bs: &[u8]) -> usize { if self.frozen.is_some() { return 0; } let n = (self.cap - self.len).min(bs.len()); self.buf.put_slice(&bs[..n]); self.len += n; if self.len >= self.cap { let frozen = self.buf.split(); self.len = 0; self.frozen = Some(frozen.freeze()); } n } /// Freeze the buffer no matter it's full or not. /// /// It's a no-op if the buffer has already been frozen. pub fn freeze(&mut self) { if self.len == 0 { return; } let frozen = self.buf.split(); self.len = 0; self.frozen = Some(frozen.freeze()); } /// Get the frozen buffer. /// /// Return `None` if the buffer is not frozen. /// /// # Notes /// /// This operation did nothing to the buffer. We use `&mut self` just for make /// the API consistent with other APIs. pub fn get(&mut self) -> Option { self.frozen.clone() } // Advance the frozen buffer. /// /// # Panics /// /// Panic if the buffer is not frozen. pub fn advance(&mut self, cnt: usize) { debug_assert!(self.len == 0, "The buffer must be empty during advance"); let Some(bs) = self.frozen.as_mut() else { unreachable!("It must be a bug to advance on not frozen buffer") }; bs.advance(cnt); if bs.is_empty() { self.clean() } } /// Cleanup the buffer, reset to the initial state. #[inline] pub fn clean(&mut self) { self.frozen = None; // This reserve cloud be cheap since we can reuse already allocated memory. // (if all references to the frozen buffer are dropped) self.buf.reserve(self.cap); } } opendal-0.52.0/src/raw/oio/buf/mod.rs000064400000000000000000000016501046102023000154030ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod flex_buf; pub use flex_buf::FlexBuf; mod queue_buf; pub use queue_buf::QueueBuf; mod pooled_buf; pub use pooled_buf::PooledBuf; opendal-0.52.0/src/raw/oio/buf/pooled_buf.rs000064400000000000000000000071471046102023000167510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::VecDeque; use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::{self}; use std::sync::Mutex; use bytes::BytesMut; /// PooledBuf is a buffer pool that designed for reusing already allocated bufs. /// /// It works as best-effort that tries to reuse the buffer if possible. It /// won't block the thread if the pool is locked, just returning a new buffer /// or dropping existing buffer. pub struct PooledBuf { pool: Mutex>, size: usize, initial_capacity: usize, } impl Debug for PooledBuf { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { f.debug_struct("PooledBuf") .field("size", &self.size) .field("initial_capacity", &self.initial_capacity) .finish_non_exhaustive() } } impl PooledBuf { /// Create a new buffer pool with a given size. pub fn new(size: usize) -> Self { Self { pool: Mutex::new(VecDeque::with_capacity(size)), size, initial_capacity: 0, } } /// Set the initial capacity of the buffer. /// /// The default value is 0. pub fn with_initial_capacity(mut self, initial_capacity: usize) -> Self { self.initial_capacity = initial_capacity; self } /// Get a [`BytesMut`] from the pool. /// /// It's guaranteed that the buffer is empty. pub fn get(&self) -> BytesMut { // We don't want to block the thread if the pool is locked. // // Just returning a new buffer in this case. let Ok(mut pool) = self.pool.try_lock() else { return BytesMut::with_capacity(self.initial_capacity); }; if let Some(buf) = pool.pop_front() { buf } else { BytesMut::with_capacity(self.initial_capacity) } } /// Put a [`BytesMut`] back to the pool. pub fn put(&self, mut buf: BytesMut) { // We don't want to block the thread if the pool is locked. // // Just dropping the buffer in this case. let Ok(mut pool) = self.pool.try_lock() else { return; }; if pool.len() < self.size { // Clean the buffer before putting it back to the pool. buf.clear(); pool.push_back(buf); } } } #[cfg(test)] mod tests { use bytes::BufMut; use super::*; #[test] fn test_pooled_buf() { let pool = PooledBuf::new(2); let mut buf1 = pool.get(); buf1.put_slice(b"hello, world!"); let mut buf2 = pool.get(); buf2.reserve(1024); pool.put(buf1); pool.put(buf2); let buf3 = pool.get(); assert_eq!(buf3.len(), 0); assert_eq!(buf3.capacity(), 13); let buf4 = pool.get(); assert_eq!(buf4.len(), 0); assert_eq!(buf4.capacity(), 1024); } } opendal-0.52.0/src/raw/oio/buf/queue_buf.rs000064400000000000000000000066011046102023000166050ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::VecDeque; use std::mem; use bytes::Buf; use crate::*; /// QueueBuf is a queue of [`Buffer`]. /// /// It's designed to allow storing multiple buffers without copying underlying bytes and consume them /// in order. /// /// QueueBuf mainly provides the following operations: /// /// - `push`: Push a new buffer in the queue. /// - `collect`: Collect all buffer in the queue as a new [`Buffer`] /// - `advance`: Advance the queue by `cnt` bytes. #[derive(Clone, Default)] pub struct QueueBuf(VecDeque); impl QueueBuf { /// Create a new buffer queue. #[inline] pub fn new() -> Self { Self::default() } /// Push new [`Buffer`] into the queue. #[inline] pub fn push(&mut self, buf: Buffer) { if buf.is_empty() { return; } self.0.push_back(buf); } /// Total bytes size inside the buffer queue. #[inline] pub fn len(&self) -> usize { self.0.iter().map(|b| b.len()).sum() } /// Is the buffer queue empty. #[inline] pub fn is_empty(&self) -> bool { self.len() == 0 } /// Take the entire buffer queue and leave `self` in empty states. #[inline] pub fn take(&mut self) -> QueueBuf { mem::take(self) } /// Build a new [`Buffer`] from the queue. /// /// If the queue is empty, it will return an empty buffer. Otherwise, it will iterate over all /// buffers and collect them into a new buffer. /// /// # Notes /// /// There are allocation overheads when collecting multiple buffers into a new buffer. But /// most of them should be acceptable since we can expect the item length of buffers are slower /// than 4k. #[inline] pub fn collect(mut self) -> Buffer { if self.0.is_empty() { Buffer::new() } else if self.0.len() == 1 { self.0.pop_front().unwrap() } else { self.0.into_iter().flatten().collect() } } /// Advance the buffer queue by `cnt` bytes. #[inline] pub fn advance(&mut self, cnt: usize) { assert!(cnt <= self.len(), "cannot advance past {cnt} bytes"); let mut new_cnt = cnt; while new_cnt > 0 { let buf = self.0.front_mut().expect("buffer must be valid"); if new_cnt < buf.remaining() { buf.advance(new_cnt); break; } else { new_cnt -= buf.remaining(); self.0.pop_front(); } } } /// Clear the buffer queue. #[inline] pub fn clear(&mut self) { self.0.clear() } } opendal-0.52.0/src/raw/oio/delete/api.rs000064400000000000000000000124431046102023000160650ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::{BoxedFuture, MaybeSend, OpDelete}; use crate::*; use std::future::Future; use std::ops::DerefMut; /// Deleter is a type erased [`Delete`] pub type Deleter = Box; /// The Delete trait defines interfaces for performing deletion operations. pub trait Delete: Unpin + Send + Sync { /// Requests deletion of a resource at the specified path with optional arguments /// /// # Parameters /// - `path`: The path of the resource to delete /// - `args`: Additional arguments for the delete operation /// /// # Returns /// - `Ok(())`: The deletion request has been successfully queued (does not guarantee actual deletion) /// - `Err(err)`: An error occurred and the deletion request was not queued /// /// # Notes /// This method just queue the delete request. The actual deletion will be /// performed when `flush` is called. fn delete(&mut self, path: &str, args: OpDelete) -> Result<()>; /// Flushes the deletion queue to ensure queued deletions are executed /// /// # Returns /// - `Ok(0)`: All queued deletions have been processed or the queue is empty. /// - `Ok(count)`: The number of resources successfully deleted. Implementations should /// return an error if the queue is non-empty but no resources were deleted /// - `Err(err)`: An error occurred while performing the deletions /// /// # Notes /// - This method is asynchronous and will wait for queued deletions to complete fn flush(&mut self) -> impl Future> + MaybeSend; } impl Delete for () { fn delete(&mut self, _: &str, _: OpDelete) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "output deleter doesn't support delete", )) } async fn flush(&mut self) -> Result { Err(Error::new( ErrorKind::Unsupported, "output deleter doesn't support flush", )) } } /// The dyn version of [`Delete`] pub trait DeleteDyn: Unpin + Send + Sync { /// The dyn version of [`Delete::delete`] fn delete_dyn(&mut self, path: &str, args: OpDelete) -> Result<()>; /// The dyn version of [`Delete::flush`] fn flush_dyn(&mut self) -> BoxedFuture>; } impl DeleteDyn for T { fn delete_dyn(&mut self, path: &str, args: OpDelete) -> Result<()> { Delete::delete(self, path, args) } fn flush_dyn(&mut self) -> BoxedFuture> { Box::pin(self.flush()) } } impl Delete for Box { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.deref_mut().delete_dyn(path, args) } async fn flush(&mut self) -> Result { self.deref_mut().flush_dyn().await } } /// BlockingDeleter is a type erased [`BlockingDelete`] pub type BlockingDeleter = Box; /// BlockingDelete is the trait to perform delete operations. pub trait BlockingDelete: Send + Sync + 'static { /// Delete given path with optional arguments. /// /// # Behavior /// /// - `Ok(())` means the path has been queued for deletion. /// - `Err(err)` means error happens and no deletion has been queued. fn delete(&mut self, path: &str, args: OpDelete) -> Result<()>; /// Flushes the deletion queue to ensure queued deletions are executed /// /// # Returns /// - `Ok(0)`: All queued deletions have been processed or the queue is empty. /// - `Ok(count)`: The number of resources successfully deleted. Implementations should /// return an error if the queue is non-empty but no resources were deleted /// - `Err(err)`: An error occurred while performing the deletions fn flush(&mut self) -> Result; } impl BlockingDelete for () { fn delete(&mut self, _: &str, _: OpDelete) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "output deleter doesn't support delete", )) } fn flush(&mut self) -> Result { Err(Error::new( ErrorKind::Unsupported, "output deleter doesn't support flush", )) } } /// `Box` won't implement `BlockingDelete` automatically. /// /// To make BlockingWriter work as expected, we must add this impl. impl BlockingDelete for Box { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { (**self).delete(path, args) } fn flush(&mut self) -> Result { (**self).flush() } } opendal-0.52.0/src/raw/oio/delete/batch_delete.rs000064400000000000000000000105551046102023000177210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; use std::collections::HashSet; use std::future::Future; /// BatchDelete is used to implement [`oio::Delete`] based on batch delete operation. /// /// OneShotDeleter will perform delete operation while calling `flush`. pub trait BatchDelete: Send + Sync + Unpin + 'static { /// delete_once delete one path at once. /// /// Implementations should make sure that the data is deleted correctly at once. /// /// BatchDeleter may call this method while there are only one path to delete. fn delete_once( &self, path: String, args: OpDelete, ) -> impl Future> + MaybeSend; /// delete_batch delete multiple paths at once. /// /// - Implementations should make sure that the length of `batch` equals to the return result's length. /// - Implementations should return error no path is deleted. fn delete_batch( &self, batch: Vec<(String, OpDelete)>, ) -> impl Future> + MaybeSend; } /// BatchDeleteResult is the result of batch delete operation. #[derive(Default)] pub struct BatchDeleteResult { /// Collection of successful deletions, containing tuples of (path, args) pub succeeded: Vec<(String, OpDelete)>, /// Collection of failed deletions, containing tuples of (path, args, error) pub failed: Vec<(String, OpDelete, Error)>, } /// BatchDeleter is used to implement [`oio::Delete`] based on batch delete. pub struct BatchDeleter { inner: D, buffer: HashSet<(String, OpDelete)>, } impl BatchDeleter { /// Create a new batch deleter. pub fn new(inner: D) -> Self { Self { inner, buffer: HashSet::default(), } } } impl oio::Delete for BatchDeleter { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.buffer.insert((path.to_string(), args)); Ok(()) } async fn flush(&mut self) -> Result { if self.buffer.is_empty() { return Ok(0); } if self.buffer.len() == 1 { let (path, args) = self .buffer .iter() .next() .expect("the delete buffer size must be 1") .clone(); self.inner.delete_once(path, args).await?; self.buffer.clear(); return Ok(1); } let batch = self.buffer.iter().cloned().collect(); let result = self.inner.delete_batch(batch).await?; debug_assert!( !result.succeeded.is_empty(), "the number of succeeded operations must be greater than 0" ); debug_assert_eq!( result.succeeded.len() + result.failed.len(), self.buffer.len(), "the number of succeeded and failed operations must be equal to the input batch size" ); // Remove all succeeded operations from the buffer. let deleted = result.succeeded.len(); for i in result.succeeded { self.buffer.remove(&i); } // Return directly if there are non-temporary errors. for (path, op, err) in result.failed { if !err.is_temporary() { return Err(err .with_context("path", path) .with_context("version", op.version().unwrap_or(""))); } } // Return the number of succeeded operations to allow users to decide whether // to retry or push more delete operations. Ok(deleted) } } opendal-0.52.0/src/raw/oio/delete/mod.rs000064400000000000000000000023031046102023000160650ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod api; pub use api::BlockingDelete; pub use api::BlockingDeleter; pub use api::Delete; pub use api::DeleteDyn; pub use api::Deleter; mod batch_delete; pub use batch_delete::BatchDelete; pub use batch_delete::BatchDeleteResult; pub use batch_delete::BatchDeleter; mod one_shot_delete; pub use one_shot_delete::BlockingOneShotDelete; pub use one_shot_delete::OneShotDelete; pub use one_shot_delete::OneShotDeleter; opendal-0.52.0/src/raw/oio/delete/one_shot_delete.rs000064400000000000000000000064651046102023000204630ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; use std::future::Future; /// OneShotDelete is used to implement [`oio::Delete`] based on one shot operation. /// /// OneShotDeleter will perform delete operation while calling `flush`. pub trait OneShotDelete: Send + Sync + Unpin + 'static { /// delete_once delete one path at once. /// /// Implementations should make sure that the data is deleted correctly at once. fn delete_once( &self, path: String, args: OpDelete, ) -> impl Future> + MaybeSend; } /// BlockingOneShotDelete is used to implement [`oio::BlockingDelete`] based on one shot operation. /// /// BlockingOneShotDeleter will perform delete operation while calling `flush`. pub trait BlockingOneShotDelete: Send + Sync + 'static { /// delete_once delete one path at once. /// /// Implementations should make sure that the data is deleted correctly at once. fn blocking_delete_once(&self, path: String, args: OpDelete) -> Result<()>; } /// OneShotDelete is used to implement [`oio::Delete`] based on one shot. pub struct OneShotDeleter { inner: D, delete: Option<(String, OpDelete)>, } impl OneShotDeleter { /// Create a new one shot deleter. pub fn new(inner: D) -> Self { Self { inner, delete: None, } } fn delete_inner(&mut self, path: String, args: OpDelete) -> Result<()> { if self.delete.is_some() { return Err(Error::new( ErrorKind::Unsupported, "OneShotDeleter doesn't support batch delete", )); } self.delete = Some((path, args)); Ok(()) } } impl oio::Delete for OneShotDeleter { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.delete_inner(path.to_string(), args) } async fn flush(&mut self) -> Result { let Some((path, args)) = self.delete.clone() else { return Ok(0); }; self.inner.delete_once(path, args).await?; self.delete = None; Ok(1) } } impl oio::BlockingDelete for OneShotDeleter { fn delete(&mut self, path: &str, args: OpDelete) -> Result<()> { self.delete_inner(path.to_string(), args) } fn flush(&mut self) -> Result { let Some((path, args)) = self.delete.clone() else { return Ok(0); }; self.inner.blocking_delete_once(path, args)?; self.delete = None; Ok(1) } } opendal-0.52.0/src/raw/oio/entry.rs000064400000000000000000000051131046102023000152070ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::*; /// Entry is returned by `Page` or `BlockingPage` during list operations. /// /// # Notes /// /// Differences between `crate::Entry` and `oio::Entry`: /// /// - `crate::Entry` is the user's public API and have less public methods. /// - `oio::Entry` is the raw API and doesn't expose to users. #[derive(Debug, Clone, PartialEq, Eq)] pub struct Entry { path: String, meta: Metadata, } impl Entry { /// Create a new entry by its corresponding underlying storage. pub fn new(path: &str, meta: Metadata) -> Entry { Self::with(path.to_string(), meta) } /// Create a new entry with given value. pub fn with(mut path: String, meta: Metadata) -> Entry { // Normalize path as `/` if it's empty. if path.is_empty() { path = "/".to_string(); } debug_assert!( meta.mode().is_dir() == path.ends_with('/'), "mode {:?} not match with path {}", meta.mode(), path ); Entry { path, meta } } /// Set path for entry. pub fn set_path(&mut self, path: &str) -> &mut Self { self.path = path.to_string(); self } /// Get the path of entry. pub fn path(&self) -> &str { &self.path } /// Set mode for entry. /// /// # Note /// /// Please use this function carefully. pub fn set_mode(&mut self, mode: EntryMode) -> &mut Self { self.meta.set_mode(mode); self } /// Get entry's mode. pub fn mode(&self) -> EntryMode { self.meta.mode() } /// Consume self to convert into an Entry. /// /// NOTE: implement this by hand to avoid leaking raw entry to end-users. pub(crate) fn into_entry(self) -> crate::Entry { crate::Entry::new(self.path, self.meta) } } opendal-0.52.0/src/raw/oio/list/api.rs000064400000000000000000000055411046102023000155770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use std::ops::DerefMut; use crate::raw::oio::Entry; use crate::raw::*; use crate::*; /// The boxed version of [`List`] pub type Lister = Box; /// Page trait is used by [`raw::Accessor`] to implement `list` operation. pub trait List: Unpin + Send + Sync { /// Fetch a new page of [`Entry`] /// /// `Ok(None)` means all pages have been returned. Any following call /// to `next` will always get the same result. fn next(&mut self) -> impl Future>> + MaybeSend; } impl List for () { async fn next(&mut self) -> Result> { Ok(None) } } impl List for Option

{ async fn next(&mut self) -> Result> { match self { Some(p) => p.next().await, None => Ok(None), } } } pub trait ListDyn: Unpin + Send + Sync { fn next_dyn(&mut self) -> BoxedFuture>>; } impl ListDyn for T { fn next_dyn(&mut self) -> BoxedFuture>> { Box::pin(self.next()) } } impl List for Box { async fn next(&mut self) -> Result> { self.deref_mut().next_dyn().await } } /// BlockingList is the blocking version of [`List`]. pub trait BlockingList: Send { /// Fetch a new page of [`Entry`] /// /// `Ok(None)` means all pages have been returned. Any following call /// to `next` will always get the same result. fn next(&mut self) -> Result>; } /// BlockingLister is a boxed [`BlockingList`] pub type BlockingLister = Box; impl BlockingList for Box

{ fn next(&mut self) -> Result> { (**self).next() } } impl BlockingList for () { fn next(&mut self) -> Result> { Ok(None) } } impl BlockingList for Option

{ fn next(&mut self) -> Result> { match self { Some(p) => p.next(), None => Ok(None), } } } opendal-0.52.0/src/raw/oio/list/flat_list.rs000064400000000000000000000165351046102023000170140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; /// FlatLister will walk dir in bottom up way: /// /// - List nested dir first /// - Go back into parent dirs one by one /// /// Given the following file tree: /// /// ```txt /// . /// ├── dir_x/ /// │ ├── dir_y/ /// │ │ ├── dir_z/ /// │ │ └── file_c /// │ └── file_b /// └── file_a /// ``` /// /// ToFlatLister will output entries like: /// /// ```txt /// dir_x/dir_y/dir_z/file_c /// dir_x/dir_y/dir_z/ /// dir_x/dir_y/file_b /// dir_x/dir_y/ /// dir_x/file_a /// dir_x/ /// ``` /// /// # Note /// /// There is no guarantee about the order between files and dirs at the same level. /// We only make sure the nested dirs will show up before parent dirs. /// /// Especially, for storage services that can't return dirs first, ToFlatLister /// may output parent dirs' files before nested dirs, this is expected because files /// always output directly while listing. pub struct FlatLister { acc: A, next_dir: Option, active_lister: Vec<(Option, L)>, } /// # Safety /// /// wasm32 is a special target that we only have one event-loop for this FlatLister. unsafe impl Send for FlatLister {} /// # Safety /// /// We will only take `&mut Self` reference for FsLister. unsafe impl Sync for FlatLister {} impl FlatLister where A: Access, { /// Create a new flat lister pub fn new(acc: A, path: &str) -> FlatLister { FlatLister { acc, next_dir: Some(oio::Entry::new(path, Metadata::new(EntryMode::DIR))), active_lister: vec![], } } } impl oio::List for FlatLister where A: Access, L: oio::List, { async fn next(&mut self) -> Result> { loop { if let Some(de) = self.next_dir.take() { let (_, l) = self.acc.list(de.path(), OpList::new()).await?; self.active_lister.push((Some(de), l)); } let (de, lister) = match self.active_lister.last_mut() { Some((de, lister)) => (de, lister), None => return Ok(None), }; match lister.next().await? { Some(v) if v.mode().is_dir() => { // should not loop itself again if v.path() != de.as_ref().expect("de should not be none here").path() { self.next_dir = Some(v); continue; } } Some(v) => return Ok(Some(v)), None => match de.take() { Some(de) => { return Ok(Some(de)); } None => { let _ = self.active_lister.pop(); continue; } }, } } } } impl oio::BlockingList for FlatLister where A: Access, P: oio::BlockingList, { fn next(&mut self) -> Result> { loop { if let Some(de) = self.next_dir.take() { let (_, l) = self.acc.blocking_list(de.path(), OpList::new())?; self.active_lister.push((Some(de), l)) } let (de, lister) = match self.active_lister.last_mut() { Some((de, lister)) => (de, lister), None => return Ok(None), }; match lister.next()? { Some(v) if v.mode().is_dir() => { if v.path() != de.as_ref().expect("de should not be none here").path() { self.next_dir = Some(v); continue; } } Some(v) => return Ok(Some(v)), None => match de.take() { Some(de) => { return Ok(Some(de)); } None => { let _ = self.active_lister.pop(); continue; } }, } } } } #[cfg(test)] mod tests { use std::collections::HashMap; use std::sync::Arc; use std::vec; use std::vec::IntoIter; use log::debug; use oio::BlockingList; use super::*; #[derive(Debug)] struct MockService { map: HashMap<&'static str, Vec<&'static str>>, } impl MockService { fn new() -> Self { let mut map = HashMap::default(); map.insert("x/", vec!["x/x/"]); map.insert("x/x/", vec!["x/x/x/"]); map.insert("x/x/x/", vec!["x/x/x/x"]); Self { map } } fn get(&self, path: &str) -> MockLister { let inner = self.map.get(path).expect("must have value").to_vec(); MockLister { inner: inner.into_iter(), } } } impl Access for MockService { type Reader = (); type BlockingReader = (); type Writer = (); type BlockingWriter = (); type Lister = (); type BlockingLister = MockLister; type Deleter = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.full_capability_mut().list = true; am.into() } fn blocking_list(&self, path: &str, _: OpList) -> Result<(RpList, Self::BlockingLister)> { debug!("visit path: {path}"); Ok((RpList::default(), self.get(path))) } } struct MockLister { inner: IntoIter<&'static str>, } impl BlockingList for MockLister { fn next(&mut self) -> Result> { Ok(self.inner.next().map(|path| { if path.ends_with('/') { oio::Entry::new(path, Metadata::new(EntryMode::DIR)) } else { oio::Entry::new(path, Metadata::new(EntryMode::FILE)) } })) } } #[test] fn test_blocking_list() -> Result<()> { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let acc = MockService::new(); let mut lister = FlatLister::new(acc, "x/"); let mut entries = Vec::default(); while let Some(e) = lister.next()? { entries.push(e) } assert_eq!( entries[0], oio::Entry::new("x/x/x/x", Metadata::new(EntryMode::FILE)) ); Ok(()) } } opendal-0.52.0/src/raw/oio/list/hierarchy_list.rs000064400000000000000000000146571046102023000200470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashSet; use crate::raw::*; use crate::*; /// ToHierarchyLister will convert a flat list to hierarchy by filter /// not needed entries. /// /// # Notes /// /// ToHierarchyLister filter entries after fetch entries. So it's possible /// to return an empty vec. It doesn't mean the all pages have been /// returned. /// /// Please keep calling next until we returned `Ok(None)` pub struct HierarchyLister

{ lister: P, path: String, visited: HashSet, recursive: bool, } impl

HierarchyLister

{ /// Create a new hierarchy lister pub fn new(lister: P, path: &str, recursive: bool) -> HierarchyLister

{ let path = if path == "/" { "".to_string() } else { path.to_string() }; HierarchyLister { lister, path, visited: HashSet::default(), recursive, } } /// ## NOTES /// /// We take `&mut Entry` here because we need to perform modification on entry in the case like /// listing "a/" with existing key `a/b/c`. /// /// In this case, we got a key `a/b/c`, but we should return `a/b/` instead to keep the hierarchy. fn keep_entry(&mut self, e: &mut oio::Entry) -> bool { // If path is not started with prefix, drop it. // // Ideally, it should never happen. But we just tolerate // this state. if !e.path().starts_with(&self.path) { return false; } // Don't return already visited path. if self.visited.contains(e.path()) { return false; } let prefix_len = self.path.len(); let idx = if let Some(idx) = e.path()[prefix_len..].find('/') { idx + prefix_len + 1 } else { // If there is no `/` in path, it's a normal file, we // can return it directly. return true; }; // idx == path.len() means it's contain only one `/` at the // end of path. if idx == e.path().len() { if !self.visited.contains(e.path()) { self.visited.insert(e.path().to_string()); } return true; } // If idx < path.len() means that are more levels to come. // We should check the first dir path. let has = { let path = &e.path()[..idx]; self.visited.contains(path) }; if !has { let path = { let path = &e.path()[..idx]; path.to_string() }; e.set_path(&path); e.set_mode(EntryMode::DIR); self.visited.insert(path); return true; } false } } impl oio::List for HierarchyLister

{ async fn next(&mut self) -> Result> { loop { let mut entry = match self.lister.next().await? { Some(entry) => entry, None => return Ok(None), }; if self.recursive { return Ok(Some(entry)); } if self.keep_entry(&mut entry) { return Ok(Some(entry)); } } } } impl oio::BlockingList for HierarchyLister

{ fn next(&mut self) -> Result> { loop { let mut entry = match self.lister.next()? { Some(entry) => entry, None => return Ok(None), }; if self.recursive { return Ok(Some(entry)); } if self.keep_entry(&mut entry) { return Ok(Some(entry)); } } } } #[cfg(test)] mod tests { use std::collections::HashSet; use std::vec::IntoIter; use log::debug; use oio::BlockingList; use super::*; struct MockLister { inner: IntoIter<&'static str>, } impl MockLister { fn new(inner: Vec<&'static str>) -> Self { Self { inner: inner.into_iter(), } } } impl BlockingList for MockLister { fn next(&mut self) -> Result> { let entry = self.inner.next().map(|path| { if path.ends_with('/') { oio::Entry::new(path, Metadata::new(EntryMode::DIR)) } else { oio::Entry::new(path, Metadata::new(EntryMode::FILE)) } }); Ok(entry) } } #[test] fn test_blocking_list() -> Result<()> { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let lister = MockLister::new(vec![ "x/x/", "x/y/", "y/", "x/x/x", "y/y", "xy/", "z", "y/a", ]); let mut lister = HierarchyLister::new(lister, "", false); let mut entries = Vec::default(); let mut set = HashSet::new(); while let Some(e) = lister.next()? { debug!("got path {}", e.path()); assert!( set.insert(e.path().to_string()), "duplicated value: {}", e.path() ); entries.push(e) } assert_eq!( entries[0], oio::Entry::new("x/", Metadata::new(EntryMode::DIR)) ); assert_eq!( entries[1], oio::Entry::new("y/", Metadata::new(EntryMode::DIR)) ); assert_eq!( entries[2], oio::Entry::new("xy/", Metadata::new(EntryMode::DIR)) ); assert_eq!( entries[3], oio::Entry::new("z", Metadata::new(EntryMode::FILE)) ); Ok(()) } } opendal-0.52.0/src/raw/oio/list/mod.rs000064400000000000000000000022311046102023000155760ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod api; pub use api::BlockingList; pub use api::BlockingLister; pub use api::List; pub use api::Lister; mod page_list; pub use page_list::PageContext; pub use page_list::PageList; pub use page_list::PageLister; mod flat_list; pub use flat_list::FlatLister; mod hierarchy_list; pub use hierarchy_list::HierarchyLister; mod prefix_list; pub use prefix_list::PrefixLister; opendal-0.52.0/src/raw/oio/list/page_list.rs000064400000000000000000000070761046102023000170020ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; use std::collections::VecDeque; use std::future::Future; /// PageList is used to implement [`oio::List`] based on API supporting pagination. By implementing /// PageList, services don't need to care about the details of page list. /// /// # Architecture /// /// The architecture after adopting [`PageList`]: /// /// - Services impl `PageList` /// - `PageLister` impl `List` /// - Expose `PageLister` as `Accessor::Lister` pub trait PageList: Send + Sync + Unpin + 'static { /// next_page is used to fetch next page of entries from underlying storage. #[cfg(not(target_arch = "wasm32"))] fn next_page(&self, ctx: &mut PageContext) -> impl Future> + MaybeSend; #[cfg(target_arch = "wasm32")] fn next_page(&self, ctx: &mut PageContext) -> impl Future>; } /// PageContext is the context passing between `PageList`. /// /// [`PageLister`] will init the PageContext, and implementer of [`PageList`] should fill the `PageContext` /// based on their needs. /// /// - Set `done` to `true` if all page have been fetched. /// - Update `token` if there is more page to fetch. `token` is not exposed to users, it's internal used only. /// - Push back into the entries for each entry fetched from underlying storage. /// /// NOTE: `entries` is a `VecDeque` to avoid unnecessary memory allocation. Only `push_back` is allowed. pub struct PageContext { /// done is used to indicate whether the list operation is done. pub done: bool, /// token is used by underlying storage services to fetch next page. pub token: String, /// entries are used to store entries fetched from underlying storage. /// /// Please always reuse the same `VecDeque` to avoid unnecessary memory allocation. /// PageLister makes sure that entries is reset before calling `next_page`. Implementer /// can call `push_back` on `entries` directly. pub entries: VecDeque, } /// PageLister implements [`oio::List`] based on [`PageList`]. pub struct PageLister { inner: L, ctx: PageContext, } impl PageLister where L: PageList, { /// Create a new PageLister. pub fn new(l: L) -> Self { Self { inner: l, ctx: PageContext { done: false, token: "".to_string(), entries: VecDeque::new(), }, } } } impl oio::List for PageLister where L: PageList, { async fn next(&mut self) -> Result> { loop { if let Some(entry) = self.ctx.entries.pop_front() { return Ok(Some(entry)); } if self.ctx.done { return Ok(None); } self.inner.next_page(&mut self.ctx).await?; } } } opendal-0.52.0/src/raw/oio/list/prefix_list.rs000064400000000000000000000041661046102023000173600ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; /// PrefixLister is used to filter entries by prefix. /// /// For example, if we have a lister that returns entries: /// /// ```txt /// . /// ├── file_a /// └── file_b /// ``` /// /// We can use `PrefixLister` to filter entries with prefix `file_`. pub struct PrefixLister { lister: L, prefix: String, } /// # Safety /// /// We will only take `&mut Self` reference for FsLister. unsafe impl Sync for PrefixLister {} impl PrefixLister { /// Create a new flat lister pub fn new(lister: L, prefix: &str) -> PrefixLister { PrefixLister { lister, prefix: prefix.to_string(), } } } impl oio::List for PrefixLister where L: oio::List, { async fn next(&mut self) -> Result> { loop { match self.lister.next().await { Ok(Some(e)) if !e.path().starts_with(&self.prefix) => continue, v => return v, } } } } impl oio::BlockingList for PrefixLister where L: oio::BlockingList, { fn next(&mut self) -> Result> { loop { match self.lister.next() { Ok(Some(e)) if !e.path().starts_with(&self.prefix) => continue, v => return v, } } } } opendal-0.52.0/src/raw/oio/mod.rs000064400000000000000000000022141046102023000146240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! `oio` provides OpenDAL's raw traits and types that opendal returns as //! output. //! //! Those types should only be used internally and we don't want users to //! depend on them. mod delete; pub use delete::*; mod read; pub use read::*; mod write; pub use write::*; mod list; pub use list::*; mod entry; pub use entry::Entry; mod buf; pub use buf::*; opendal-0.52.0/src/raw/oio/read/api.rs000064400000000000000000000113431046102023000155340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::mem; use std::ops::DerefMut; use bytes::Bytes; use futures::Future; use crate::raw::*; use crate::*; /// Reader is a type erased [`Read`]. pub type Reader = Box; /// Read is the internal trait used by OpenDAL to read data from storage. /// /// Users should not use or import this trait unless they are implementing an `Accessor`. /// /// # Notes /// /// ## Object Safety /// /// `Read` uses `async in trait`, making it not object safe, preventing the use of `Box`. /// To address this, we've introduced [`ReadDyn`] and its compatible type `Box`. /// /// `ReadDyn` uses `Box::pin()` to transform the returned future into a [`BoxedFuture`], introducing /// an additional layer of indirection and an extra allocation. Ideally, `ReadDyn` should occur only /// once, at the outermost level of our API. pub trait Read: Unpin + Send + Sync { /// Read at the given offset with the given size. fn read(&mut self) -> impl Future> + MaybeSend; /// Read all data from the reader. fn read_all(&mut self) -> impl Future> + MaybeSend { async { let mut bufs = vec![]; loop { match self.read().await { Ok(buf) if buf.is_empty() => break, Ok(buf) => bufs.push(buf), Err(err) => return Err(err), } } Ok(bufs.into_iter().flatten().collect()) } } } impl Read for () { async fn read(&mut self) -> Result { Err(Error::new( ErrorKind::Unsupported, "output reader doesn't support read", )) } } impl Read for Bytes { async fn read(&mut self) -> Result { Ok(Buffer::from(self.split_off(0))) } } impl Read for Buffer { async fn read(&mut self) -> Result { Ok(mem::take(self)) } } /// ReadDyn is the dyn version of [`Read`] make it possible to use as /// `Box`. pub trait ReadDyn: Unpin + Send + Sync { /// The dyn version of [`Read::read`]. /// /// This function returns a boxed future to make it object safe. fn read_dyn(&mut self) -> BoxedFuture>; /// The dyn version of [`Read::read_all`] fn read_all_dyn(&mut self) -> BoxedFuture>; } impl ReadDyn for T { fn read_dyn(&mut self) -> BoxedFuture> { Box::pin(self.read()) } fn read_all_dyn(&mut self) -> BoxedFuture> { Box::pin(self.read_all()) } } /// # NOTE /// /// Take care about the `deref_mut()` here. This makes sure that we are calling functions /// upon `&mut T` instead of `&mut Box`. The later could result in infinite recursion. impl Read for Box { async fn read(&mut self) -> Result { self.deref_mut().read_dyn().await } async fn read_all(&mut self) -> Result { self.deref_mut().read_all_dyn().await } } /// BlockingReader is a arc dyn `BlockingRead`. pub type BlockingReader = Box; /// Read is the trait that OpenDAL returns to callers. pub trait BlockingRead: Send + Sync { /// Read data from the reader at the given offset with the given size. fn read(&mut self) -> Result; } impl BlockingRead for () { fn read(&mut self) -> Result { unimplemented!("read is required to be implemented for oio::BlockingRead") } } impl BlockingRead for Bytes { fn read(&mut self) -> Result { Ok(Buffer::from(self.split_off(0))) } } impl BlockingRead for Buffer { fn read(&mut self) -> Result { Ok(mem::take(self)) } } /// `Arc` won't implement `BlockingRead` automatically. /// To make BlockingReader work as expected, we must add this impl. impl BlockingRead for Box { fn read(&mut self) -> Result { (**self).read() } } opendal-0.52.0/src/raw/oio/read/mod.rs000064400000000000000000000016411046102023000155420ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod api; pub use api::BlockingRead; pub use api::BlockingReader; pub use api::Read; pub use api::ReadDyn; pub use api::Reader; opendal-0.52.0/src/raw/oio/write/api.rs000064400000000000000000000107011046102023000157500ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use std::ops::DerefMut; use crate::raw::*; use crate::*; /// Writer is a type erased [`Write`] pub type Writer = Box; /// Write is the trait that OpenDAL returns to callers. pub trait Write: Unpin + Send + Sync { /// Write given bytes into writer. /// /// # Behavior /// /// - `Ok(())` means all bytes has been written successfully. /// - `Err(err)` means error happens and no bytes has been written. fn write(&mut self, bs: Buffer) -> impl Future> + MaybeSend; /// Close the writer and make sure all data has been flushed. fn close(&mut self) -> impl Future> + MaybeSend; /// Abort the pending writer. fn abort(&mut self) -> impl Future> + MaybeSend; } impl Write for () { async fn write(&mut self, _: Buffer) -> Result<()> { unimplemented!("write is required to be implemented for oio::Write") } async fn close(&mut self) -> Result { Err(Error::new( ErrorKind::Unsupported, "output writer doesn't support close", )) } async fn abort(&mut self) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "output writer doesn't support abort", )) } } pub trait WriteDyn: Unpin + Send + Sync { fn write_dyn(&mut self, bs: Buffer) -> BoxedFuture>; fn close_dyn(&mut self) -> BoxedFuture>; fn abort_dyn(&mut self) -> BoxedFuture>; } impl WriteDyn for T { fn write_dyn(&mut self, bs: Buffer) -> BoxedFuture> { Box::pin(self.write(bs)) } fn close_dyn(&mut self) -> BoxedFuture> { Box::pin(self.close()) } fn abort_dyn(&mut self) -> BoxedFuture> { Box::pin(self.abort()) } } impl Write for Box { async fn write(&mut self, bs: Buffer) -> Result<()> { self.deref_mut().write_dyn(bs).await } async fn close(&mut self) -> Result { self.deref_mut().close_dyn().await } async fn abort(&mut self) -> Result<()> { self.deref_mut().abort_dyn().await } } /// BlockingWriter is a type erased [`BlockingWrite`] pub type BlockingWriter = Box; /// BlockingWrite is the trait that OpenDAL returns to callers. pub trait BlockingWrite: Send + Sync + 'static { /// Write whole content at once. /// /// # Behavior /// /// - `Ok(n)` means `n` bytes has been written successfully. /// - `Err(err)` means error happens and no bytes has been written. /// /// It's possible that `n < bs.len()`, caller should pass the remaining bytes /// repeatedly until all bytes has been written. fn write(&mut self, bs: Buffer) -> Result<()>; /// Close the writer and make sure all data has been flushed. fn close(&mut self) -> Result; } impl BlockingWrite for () { fn write(&mut self, bs: Buffer) -> Result<()> { let _ = bs; unimplemented!("write is required to be implemented for oio::BlockingWrite") } fn close(&mut self) -> Result { Err(Error::new( ErrorKind::Unsupported, "output writer doesn't support close", )) } } /// `Box` won't implement `BlockingWrite` automatically. /// /// To make BlockingWriter work as expected, we must add this impl. impl BlockingWrite for Box { fn write(&mut self, bs: Buffer) -> Result<()> { (**self).write(bs) } fn close(&mut self) -> Result { (**self).close() } } opendal-0.52.0/src/raw/oio/write/append_write.rs000064400000000000000000000064211046102023000176640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use crate::raw::*; use crate::*; /// AppendWrite is used to implement [`oio::Write`] based on append /// object. By implementing AppendWrite, services don't need to /// care about the details of buffering and uploading parts. /// /// The layout after adopting [`AppendWrite`]: /// /// - Services impl `AppendWrite` /// - `AppendWriter` impl `Write` /// - Expose `AppendWriter` as `Accessor::Writer` /// /// ## Requirements /// /// Services that implement `AppendWrite` must fulfill the following requirements: /// /// - Must be a http service that could accept `AsyncBody`. /// - Provide a way to get the current offset of the append object. pub trait AppendWrite: Send + Sync + Unpin + 'static { /// Get the current offset of the append object. /// /// Returns `0` if the object is not exist. fn offset(&self) -> impl Future> + MaybeSend; /// Append the data to the end of this object. fn append( &self, offset: u64, size: u64, body: Buffer, ) -> impl Future> + MaybeSend; } /// AppendWriter will implements [`oio::Write`] based on append object. /// /// ## TODO /// /// - Allow users to switch to un-buffered mode if users write 16MiB every time. pub struct AppendWriter { inner: W, offset: Option, meta: Metadata, } /// # Safety /// /// wasm32 is a special target that we only have one event-loop for this state. impl AppendWriter { /// Create a new AppendWriter. pub fn new(inner: W) -> Self { Self { inner, offset: None, meta: Metadata::default(), } } } impl oio::Write for AppendWriter where W: AppendWrite, { async fn write(&mut self, bs: Buffer) -> Result<()> { let offset = match self.offset { Some(offset) => offset, None => { let offset = self.inner.offset().await?; self.offset = Some(offset); offset } }; let size = bs.len(); self.meta = self.inner.append(offset, size as u64, bs).await?; // Update offset after succeed. self.offset = Some(offset + size as u64); Ok(()) } async fn close(&mut self) -> Result { self.meta .set_content_length(self.offset.unwrap_or_default()); Ok(self.meta.clone()) } async fn abort(&mut self) -> Result<()> { Ok(()) } } opendal-0.52.0/src/raw/oio/write/block_write.rs000064400000000000000000000275221046102023000175140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use futures::select; use futures::Future; use futures::FutureExt; use futures::TryFutureExt; use uuid::Uuid; use crate::raw::*; use crate::*; /// BlockWrite is used to implement [`oio::Write`] based on block /// uploads. By implementing BlockWrite, services don't need to /// care about the details of uploading blocks. /// /// # Architecture /// /// The architecture after adopting [`BlockWrite`]: /// /// - Services impl `BlockWrite` /// - `BlockWriter` impl `Write` /// - Expose `BlockWriter` as `Accessor::Writer` /// /// # Notes /// /// `BlockWrite` has an oneshot optimization when `write` has been called only once: /// /// ```no_build /// w.write(bs).await?; /// w.close().await?; /// ``` /// /// We will use `write_once` instead of starting a new block upload. /// /// # Requirements /// /// Services that implement `BlockWrite` must fulfill the following requirements: /// /// - Must be a http service that could accept `AsyncBody`. /// - Don't need initialization before writing. /// - Block ID is generated by caller `BlockWrite` instead of services. /// - Complete block by an ordered block id list. pub trait BlockWrite: Send + Sync + Unpin + 'static { /// write_once is used to write the data to underlying storage at once. /// /// BlockWriter will call this API when: /// /// - All the data has been written to the buffer and we can perform the upload at once. fn write_once( &self, size: u64, body: Buffer, ) -> impl Future> + MaybeSend; /// write_block will write a block of the data. /// /// BlockWriter will call this API and stores the result in /// order. /// /// - block_id is the id of the block. fn write_block( &self, block_id: Uuid, size: u64, body: Buffer, ) -> impl Future> + MaybeSend; /// complete_block will complete the block upload to build the final /// file. fn complete_block( &self, block_ids: Vec, ) -> impl Future> + MaybeSend; /// abort_block will cancel the block upload and purge all data. fn abort_block(&self, block_ids: Vec) -> impl Future> + MaybeSend; } struct WriteInput { w: Arc, executor: Executor, block_id: Uuid, bytes: Buffer, } /// BlockWriter will implement [`oio::Write`] based on block /// uploads. pub struct BlockWriter { w: Arc, executor: Executor, started: bool, block_ids: Vec, cache: Option, tasks: ConcurrentTasks, Uuid>, } impl BlockWriter { /// Create a new BlockWriter. pub fn new(inner: W, executor: Option, concurrent: usize) -> Self { let executor = executor.unwrap_or_default(); Self { w: Arc::new(inner), executor: executor.clone(), started: false, block_ids: Vec::new(), cache: None, tasks: ConcurrentTasks::new(executor, concurrent, |input| { Box::pin(async move { let fut = input .w .write_block( input.block_id, input.bytes.len() as u64, input.bytes.clone(), ) .map_ok(|_| input.block_id); match input.executor.timeout() { None => { let result = fut.await; (input, result) } Some(timeout) => { let result = select! { result = fut.fuse() => { result } _ = timeout.fuse() => { Err(Error::new( ErrorKind::Unexpected, "write block timeout") .with_context("block_id", input.block_id.to_string()) .set_temporary()) } }; (input, result) } } }) }), } } fn fill_cache(&mut self, bs: Buffer) -> usize { let size = bs.len(); assert!(self.cache.is_none()); self.cache = Some(bs); size } } impl oio::Write for BlockWriter where W: BlockWrite, { async fn write(&mut self, bs: Buffer) -> Result<()> { if !self.started && self.cache.is_none() { self.fill_cache(bs); return Ok(()); } // The block upload process has been started. self.started = true; let bytes = self.cache.clone().expect("pending write must exist"); self.tasks .execute(WriteInput { w: self.w.clone(), executor: self.executor.clone(), block_id: Uuid::new_v4(), bytes, }) .await?; self.cache = None; self.fill_cache(bs); Ok(()) } async fn close(&mut self) -> Result { if !self.started { let (size, body) = match self.cache.clone() { Some(cache) => (cache.len(), cache), None => (0, Buffer::new()), }; let meta = self.w.write_once(size as u64, body).await?; self.cache = None; return Ok(meta); } if let Some(cache) = self.cache.clone() { self.tasks .execute(WriteInput { w: self.w.clone(), executor: self.executor.clone(), block_id: Uuid::new_v4(), bytes: cache, }) .await?; self.cache = None; } loop { let Some(result) = self.tasks.next().await.transpose()? else { break; }; self.block_ids.push(result); } let block_ids = self.block_ids.clone(); self.w.complete_block(block_ids).await } async fn abort(&mut self) -> Result<()> { if !self.started { return Ok(()); } self.tasks.clear(); self.cache = None; self.w.abort_block(self.block_ids.clone()).await?; Ok(()) } } #[cfg(test)] mod tests { use std::collections::HashMap; use std::sync::Mutex; use std::time::Duration; use pretty_assertions::assert_eq; use rand::thread_rng; use rand::Rng; use rand::RngCore; use tokio::time::sleep; use super::*; use crate::raw::oio::Write; struct TestWrite { length: u64, bytes: HashMap, content: Option, } impl TestWrite { pub fn new() -> Arc> { let v = Self { length: 0, bytes: HashMap::new(), content: None, }; Arc::new(Mutex::new(v)) } } impl BlockWrite for Arc> { async fn write_once(&self, size: u64, body: Buffer) -> Result { sleep(Duration::from_nanos(50)).await; if thread_rng().gen_bool(1.0 / 10.0) { return Err( Error::new(ErrorKind::Unexpected, "I'm a crazy monkey!").set_temporary() ); } let mut this = self.lock().unwrap(); this.length = size; this.content = Some(body); Ok(Metadata::default()) } async fn write_block(&self, block_id: Uuid, size: u64, body: Buffer) -> Result<()> { // Add an async sleep here to enforce some pending. sleep(Duration::from_millis(50)).await; // We will have 10% percent rate for write part to fail. if thread_rng().gen_bool(1.0 / 10.0) { return Err( Error::new(ErrorKind::Unexpected, "I'm a crazy monkey!").set_temporary() ); } let mut this = self.lock().unwrap(); this.length += size; this.bytes.insert(block_id, body); Ok(()) } async fn complete_block(&self, block_ids: Vec) -> Result { let mut this = self.lock().unwrap(); let mut bs = Vec::new(); for id in block_ids { bs.push(this.bytes[&id].clone()); } this.content = Some(bs.into_iter().flatten().collect()); Ok(Metadata::default()) } async fn abort_block(&self, _: Vec) -> Result<()> { Ok(()) } } #[tokio::test] async fn test_block_writer_with_concurrent_errors() { let mut rng = thread_rng(); let mut w = BlockWriter::new(TestWrite::new(), Some(Executor::new()), 8); let mut total_size = 0u64; let mut expected_content = Vec::new(); for _ in 0..1000 { let size = rng.gen_range(1..1024); total_size += size as u64; let mut bs = vec![0; size]; rng.fill_bytes(&mut bs); expected_content.extend_from_slice(&bs); loop { match w.write(bs.clone().into()).await { Ok(_) => break, Err(_) => continue, } } } loop { match w.close().await { Ok(_) => break, Err(_) => continue, } } let inner = w.w.lock().unwrap(); assert_eq!(total_size, inner.length, "length must be the same"); assert!(inner.content.is_some()); assert_eq!( expected_content, inner.content.clone().unwrap().to_bytes(), "content must be the same" ); } #[tokio::test] async fn test_block_writer_with_retry_when_write_once_error() { let mut rng = thread_rng(); for _ in 1..100 { let mut w = BlockWriter::new(TestWrite::new(), Some(Executor::new()), 8); let size = rng.gen_range(1..1024); let mut bs = vec![0; size]; rng.fill_bytes(&mut bs); loop { match w.write(bs.clone().into()).await { Ok(_) => break, Err(_) => continue, } } loop { match w.close().await { Ok(_) => break, Err(_) => continue, } } let inner = w.w.lock().unwrap(); assert_eq!(size as u64, inner.length, "length must be the same"); assert!(inner.content.is_some()); assert_eq!( bs, inner.content.clone().unwrap().to_bytes(), "content must be the same" ); } } } opendal-0.52.0/src/raw/oio/write/mod.rs000064400000000000000000000026231046102023000157620ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod api; pub use api::BlockingWrite; pub use api::BlockingWriter; pub use api::Write; pub use api::Writer; mod multipart_write; pub use multipart_write::MultipartPart; pub use multipart_write::MultipartWrite; pub use multipart_write::MultipartWriter; mod append_write; pub use append_write::AppendWrite; pub use append_write::AppendWriter; mod one_shot_write; pub use one_shot_write::OneShotWrite; pub use one_shot_write::OneShotWriter; mod block_write; pub use block_write::BlockWrite; pub use block_write::BlockWriter; mod position_write; pub use position_write::PositionWrite; pub use position_write::PositionWriter; opendal-0.52.0/src/raw/oio/write/multipart_write.rs000064400000000000000000000376261046102023000204510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use futures::select; use futures::Future; use futures::FutureExt; use crate::raw::*; use crate::*; /// MultipartWrite is used to implement [`oio::Write`] based on multipart /// uploads. By implementing MultipartWrite, services don't need to /// care about the details of uploading parts. /// /// # Architecture /// /// The architecture after adopting [`MultipartWrite`]: /// /// - Services impl `MultipartWrite` /// - `MultipartWriter` impl `Write` /// - Expose `MultipartWriter` as `Accessor::Writer` /// /// # Notes /// /// `MultipartWrite` has an oneshot optimization when `write` has been called only once: /// /// ```no_build /// w.write(bs).await?; /// w.close().await?; /// ``` /// /// We will use `write_once` instead of starting a new multipart upload. /// /// # Requirements /// /// Services that implement `BlockWrite` must fulfill the following requirements: /// /// - Must be a http service that could accept `AsyncBody`. /// - Don't need initialization before writing. /// - Block ID is generated by caller `BlockWrite` instead of services. /// - Complete block by an ordered block id list. pub trait MultipartWrite: Send + Sync + Unpin + 'static { /// write_once is used to write the data to underlying storage at once. /// /// MultipartWriter will call this API when: /// /// - All the data has been written to the buffer and we can perform the upload at once. fn write_once( &self, size: u64, body: Buffer, ) -> impl Future> + MaybeSend; /// initiate_part will call start a multipart upload and return the upload id. /// /// MultipartWriter will call this when: /// /// - the total size of data is unknown. /// - the total size of data is known, but the size of current write /// is less than the total size. fn initiate_part(&self) -> impl Future> + MaybeSend; /// write_part will write a part of the data and returns the result /// [`MultipartPart`]. /// /// MultipartWriter will call this API and stores the result in /// order. /// /// - part_number is the index of the part, starting from 0. fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> impl Future> + MaybeSend; /// complete_part will complete the multipart upload to build the final /// file. fn complete_part( &self, upload_id: &str, parts: &[MultipartPart], ) -> impl Future> + MaybeSend; /// abort_part will cancel the multipart upload and purge all data. fn abort_part(&self, upload_id: &str) -> impl Future> + MaybeSend; } /// The result of [`MultipartWrite::write_part`]. /// /// services implement should convert MultipartPart to their own represents. /// /// - `part_number` is the index of the part, starting from 0. /// - `etag` is the `ETag` of the part. /// - `checksum` is the optional checksum of the part. #[derive(Clone)] pub struct MultipartPart { /// The number of the part, starting from 0. pub part_number: usize, /// The etag of the part. pub etag: String, /// The checksum of the part. pub checksum: Option, } struct WriteInput { w: Arc, executor: Executor, upload_id: Arc, part_number: usize, bytes: Buffer, } /// MultipartWriter will implement [`oio::Write`] based on multipart /// uploads. pub struct MultipartWriter { w: Arc, executor: Executor, upload_id: Option>, parts: Vec, cache: Option, next_part_number: usize, tasks: ConcurrentTasks, MultipartPart>, } /// # Safety /// /// wasm32 is a special target that we only have one event-loop for this state. impl MultipartWriter { /// Create a new MultipartWriter. pub fn new(inner: W, executor: Option, concurrent: usize) -> Self { let w = Arc::new(inner); let executor = executor.unwrap_or_default(); Self { w, executor: executor.clone(), upload_id: None, parts: Vec::new(), cache: None, next_part_number: 0, tasks: ConcurrentTasks::new(executor, concurrent, |input| { Box::pin({ async move { let fut = input.w.write_part( &input.upload_id, input.part_number, input.bytes.len() as u64, input.bytes.clone(), ); match input.executor.timeout() { None => { let result = fut.await; (input, result) } Some(timeout) => { let result = select! { result = fut.fuse() => { result } _ = timeout.fuse() => { Err(Error::new( ErrorKind::Unexpected, "write part timeout") .with_context("upload_id", input.upload_id.to_string()) .with_context("part_number", input.part_number.to_string()) .set_temporary()) } }; (input, result) } } } }) }), } } fn fill_cache(&mut self, bs: Buffer) -> usize { let size = bs.len(); assert!(self.cache.is_none()); self.cache = Some(bs); size } } impl oio::Write for MultipartWriter where W: MultipartWrite, { async fn write(&mut self, bs: Buffer) -> Result<()> { let upload_id = match self.upload_id.clone() { Some(v) => v, None => { // Fill cache with the first write. if self.cache.is_none() { self.fill_cache(bs); return Ok(()); } let upload_id = self.w.initiate_part().await?; let upload_id = Arc::new(upload_id); self.upload_id = Some(upload_id.clone()); upload_id } }; let bytes = self.cache.clone().expect("pending write must exist"); let part_number = self.next_part_number; self.tasks .execute(WriteInput { w: self.w.clone(), executor: self.executor.clone(), upload_id: upload_id.clone(), part_number, bytes, }) .await?; self.cache = None; self.next_part_number += 1; self.fill_cache(bs); Ok(()) } async fn close(&mut self) -> Result { let upload_id = match self.upload_id.clone() { Some(v) => v, None => { let (size, body) = match self.cache.clone() { Some(cache) => (cache.len(), cache), None => (0, Buffer::new()), }; // Call write_once if there is no upload_id. let meta = self.w.write_once(size as u64, body).await?; // make sure to clear the cache only after write_once succeeds; otherwise, retries may fail. self.cache = None; return Ok(meta); } }; if let Some(cache) = self.cache.clone() { let part_number = self.next_part_number; self.tasks .execute(WriteInput { w: self.w.clone(), executor: self.executor.clone(), upload_id: upload_id.clone(), part_number, bytes: cache, }) .await?; self.cache = None; self.next_part_number += 1; } loop { let Some(result) = self.tasks.next().await.transpose()? else { break; }; self.parts.push(result) } if self.parts.len() != self.next_part_number { return Err(Error::new( ErrorKind::Unexpected, "multipart part numbers mismatch, please report bug to opendal", ) .with_context("expected", self.next_part_number) .with_context("actual", self.parts.len()) .with_context("upload_id", upload_id)); } self.w.complete_part(&upload_id, &self.parts).await } async fn abort(&mut self) -> Result<()> { let Some(upload_id) = self.upload_id.clone() else { return Ok(()); }; self.tasks.clear(); self.cache = None; self.w.abort_part(&upload_id).await?; Ok(()) } } #[cfg(test)] mod tests { use std::time::Duration; use pretty_assertions::assert_eq; use rand::thread_rng; use rand::Rng; use rand::RngCore; use tokio::sync::Mutex; use tokio::time::sleep; use tokio::time::timeout; use super::*; use crate::raw::oio::Write; struct TestWrite { upload_id: String, part_numbers: Vec, length: u64, content: Option, } impl TestWrite { pub fn new() -> Arc> { let v = Self { upload_id: uuid::Uuid::new_v4().to_string(), part_numbers: Vec::new(), length: 0, content: None, }; Arc::new(Mutex::new(v)) } } impl MultipartWrite for Arc> { async fn write_once(&self, size: u64, body: Buffer) -> Result { sleep(Duration::from_nanos(50)).await; if thread_rng().gen_bool(1.0 / 10.0) { return Err( Error::new(ErrorKind::Unexpected, "I'm a crazy monkey!").set_temporary() ); } let mut this = self.lock().await; this.length = size; this.content = Some(body); Ok(Metadata::default().with_content_length(size)) } async fn initiate_part(&self) -> Result { let upload_id = self.lock().await.upload_id.clone(); Ok(upload_id) } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, _: Buffer, ) -> Result { { let test = self.lock().await; assert_eq!(upload_id, test.upload_id); } // Add an async sleep here to enforce some pending. sleep(Duration::from_nanos(50)).await; // We will have 10% percent rate for write part to fail. if thread_rng().gen_bool(1.0 / 10.0) { return Err( Error::new(ErrorKind::Unexpected, "I'm a crazy monkey!").set_temporary() ); } { let mut test = self.lock().await; test.part_numbers.push(part_number); test.length += size; } Ok(MultipartPart { part_number, etag: "etag".to_string(), checksum: None, }) } async fn complete_part( &self, upload_id: &str, parts: &[MultipartPart], ) -> Result { let test = self.lock().await; assert_eq!(upload_id, test.upload_id); assert_eq!(parts.len(), test.part_numbers.len()); Ok(Metadata::default().with_content_length(test.length)) } async fn abort_part(&self, upload_id: &str) -> Result<()> { let test = self.lock().await; assert_eq!(upload_id, test.upload_id); Ok(()) } } struct TimeoutExecutor { exec: Arc, } impl TimeoutExecutor { pub fn new() -> Self { Self { exec: Executor::new().into_inner(), } } } impl Execute for TimeoutExecutor { fn execute(&self, f: BoxedStaticFuture<()>) { self.exec.execute(f) } fn timeout(&self) -> Option> { let time = thread_rng().gen_range(0..100); Some(Box::pin(tokio::time::sleep(Duration::from_nanos(time)))) } } #[tokio::test] async fn test_multipart_upload_writer_with_concurrent_errors() { let mut rng = thread_rng(); let mut w = MultipartWriter::new( TestWrite::new(), Some(Executor::with(TimeoutExecutor::new())), 200, ); let mut total_size = 0u64; for _ in 0..1000 { let size = rng.gen_range(1..1024); total_size += size as u64; let mut bs = vec![0; size]; rng.fill_bytes(&mut bs); loop { match timeout(Duration::from_nanos(10), w.write(bs.clone().into())).await { Ok(Ok(_)) => break, Ok(Err(_)) => continue, Err(_) => { continue; } } } } loop { match timeout(Duration::from_nanos(10), w.close()).await { Ok(Ok(_)) => break, Ok(Err(_)) => continue, Err(_) => { continue; } } } let actual_parts: Vec<_> = w.parts.into_iter().map(|v| v.part_number).collect(); let expected_parts: Vec<_> = (0..1000).collect(); assert_eq!(actual_parts, expected_parts); let actual_size = w.w.lock().await.length; assert_eq!(actual_size, total_size); } #[tokio::test] async fn test_multipart_writer_with_retry_when_write_once_error() { let mut rng = thread_rng(); for _ in 0..100 { let mut w = MultipartWriter::new(TestWrite::new(), None, 200); let size = rng.gen_range(1..1024); let mut bs = vec![0; size]; rng.fill_bytes(&mut bs); loop { match w.write(bs.clone().into()).await { Ok(_) => break, Err(_) => continue, } } loop { match w.close().await { Ok(_) => break, Err(_) => continue, } } let inner = w.w.lock().await; assert_eq!(inner.length, size as u64); assert!(inner.content.is_some()); assert_eq!(inner.content.clone().unwrap().to_bytes(), bs); } } } opendal-0.52.0/src/raw/oio/write/one_shot_write.rs000064400000000000000000000047241046102023000202370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use crate::raw::*; use crate::*; /// OneShotWrite is used to implement [`oio::Write`] based on one shot operation. /// By implementing OneShotWrite, services don't need to care about the details. /// /// For example, S3 `PUT Object` and fs `write_all`. /// /// The layout after adopting [`OneShotWrite`]: pub trait OneShotWrite: Send + Sync + Unpin + 'static { /// write_once write all data at once. /// /// Implementations should make sure that the data is written correctly at once. fn write_once(&self, bs: Buffer) -> impl Future> + MaybeSend; } /// OneShotWrite is used to implement [`oio::Write`] based on one shot. pub struct OneShotWriter { inner: W, buffer: Option, } impl OneShotWriter { /// Create a new one shot writer. pub fn new(inner: W) -> Self { Self { inner, buffer: None, } } } impl oio::Write for OneShotWriter { async fn write(&mut self, bs: Buffer) -> Result<()> { match &self.buffer { Some(_) => Err(Error::new( ErrorKind::Unsupported, "OneShotWriter doesn't support multiple write", )), None => { self.buffer = Some(bs); Ok(()) } } } async fn close(&mut self) -> Result { match self.buffer.clone() { Some(bs) => self.inner.write_once(bs).await, None => self.inner.write_once(Buffer::new()).await, } } async fn abort(&mut self) -> Result<()> { self.buffer = None; Ok(()) } } opendal-0.52.0/src/raw/oio/write/position_write.rs000064400000000000000000000204001046102023000202520ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use futures::select; use futures::Future; use futures::FutureExt; use crate::raw::*; use crate::*; /// PositionWrite is used to implement [`oio::Write`] based on position write. /// /// # Services /// /// Services like fs support position write. /// /// # Architecture /// /// The architecture after adopting [`PositionWrite`]: /// /// - Services impl `PositionWrite` /// - `PositionWriter` impl `Write` /// - Expose `PositionWriter` as `Accessor::Writer` /// /// # Requirements /// /// Services that implement `PositionWrite` must fulfill the following requirements: /// /// - Writing data based on position: `offset`. pub trait PositionWrite: Send + Sync + Unpin + 'static { /// write_all_at is used to write the data to underlying storage at the specified offset. fn write_all_at( &self, offset: u64, buf: Buffer, ) -> impl Future> + MaybeSend; /// close is used to close the underlying file. fn close(&self) -> impl Future> + MaybeSend; /// abort is used to abort the underlying abort. fn abort(&self) -> impl Future> + MaybeSend; } struct WriteInput { w: Arc, executor: Executor, offset: u64, bytes: Buffer, } /// PositionWriter will implement [`oio::Write`] based on position write. pub struct PositionWriter { w: Arc, executor: Executor, next_offset: u64, cache: Option, tasks: ConcurrentTasks, ()>, } #[allow(dead_code)] impl PositionWriter { /// Create a new PositionWriter. pub fn new(inner: W, executor: Option, concurrent: usize) -> Self { let executor = executor.unwrap_or_default(); Self { w: Arc::new(inner), executor: executor.clone(), next_offset: 0, cache: None, tasks: ConcurrentTasks::new(executor, concurrent, |input| { Box::pin(async move { let fut = input.w.write_all_at(input.offset, input.bytes.clone()); match input.executor.timeout() { None => { let result = fut.await; (input, result) } Some(timeout) => { let result = select! { result = fut.fuse() => { result } _ = timeout.fuse() => { Err(Error::new( ErrorKind::Unexpected, "write position timeout") .with_context("offset", input.offset.to_string()) .set_temporary()) } }; (input, result) } } }) }), } } fn fill_cache(&mut self, bs: Buffer) -> usize { let size = bs.len(); assert!(self.cache.is_none()); self.cache = Some(bs); size } } impl oio::Write for PositionWriter { async fn write(&mut self, bs: Buffer) -> Result<()> { if self.cache.is_none() { let _ = self.fill_cache(bs); return Ok(()); } let bytes = self.cache.clone().expect("pending write must exist"); let length = bytes.len() as u64; let offset = self.next_offset; self.tasks .execute(WriteInput { w: self.w.clone(), executor: self.executor.clone(), offset, bytes, }) .await?; self.cache = None; self.next_offset += length; let _ = self.fill_cache(bs); Ok(()) } async fn close(&mut self) -> Result { // Make sure all tasks are finished. while self.tasks.next().await.transpose()?.is_some() {} if let Some(buffer) = self.cache.clone() { let offset = self.next_offset; self.w.write_all_at(offset, buffer).await?; self.cache = None; } self.w.close().await } async fn abort(&mut self) -> Result<()> { self.tasks.clear(); self.cache = None; self.w.abort().await?; Ok(()) } } #[cfg(test)] mod tests { use std::collections::HashSet; use std::sync::Mutex; use std::time::Duration; use pretty_assertions::assert_eq; use rand::thread_rng; use rand::Rng; use rand::RngCore; use tokio::time::sleep; use super::*; use crate::raw::oio::Write; struct TestWrite { length: u64, bytes: HashSet, } impl TestWrite { pub fn new() -> Arc> { let v = Self { bytes: HashSet::new(), length: 0, }; Arc::new(Mutex::new(v)) } } impl PositionWrite for Arc> { async fn write_all_at(&self, offset: u64, buf: Buffer) -> Result<()> { // Add an async sleep here to enforce some pending. sleep(Duration::from_millis(50)).await; // We will have 10% percent rate for write part to fail. if thread_rng().gen_bool(1.0 / 10.0) { return Err( Error::new(ErrorKind::Unexpected, "I'm a crazy monkey!").set_temporary() ); } let mut test = self.lock().unwrap(); let size = buf.len() as u64; test.length += size; let input = (offset..offset + size).collect::>(); assert!( test.bytes.is_disjoint(&input), "input should not have overlap" ); test.bytes.extend(input); Ok(()) } async fn close(&self) -> Result { Ok(Metadata::default()) } async fn abort(&self) -> Result<()> { Ok(()) } } #[tokio::test] async fn test_position_writer_with_concurrent_errors() { let mut rng = thread_rng(); let mut w = PositionWriter::new(TestWrite::new(), Some(Executor::new()), 200); let mut total_size = 0u64; for _ in 0..1000 { let size = rng.gen_range(1..1024); total_size += size as u64; let mut bs = vec![0; size]; rng.fill_bytes(&mut bs); loop { match w.write(bs.clone().into()).await { Ok(_) => break, Err(e) => { println!("write error: {:?}", e); continue; } } } } loop { match w.close().await { Ok(n) => { println!("close: {:?}", n); break; } Err(e) => { println!("close error: {:?}", e); continue; } } } let actual_bytes = w.w.lock().unwrap().bytes.clone(); let expected_bytes: HashSet<_> = (0..total_size).collect(); assert_eq!(actual_bytes, expected_bytes); let actual_size = w.w.lock().unwrap().length; assert_eq!(actual_size, total_size); } } opendal-0.52.0/src/raw/operation.rs000064400000000000000000000141521046102023000152630ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Display; use std::fmt::Formatter; /// Operation is the name for APIs in `Accessor`. #[derive(Debug, Copy, Clone, Hash, Eq, PartialEq, Default)] #[non_exhaustive] pub enum Operation { /// Operation for [`crate::raw::Access::info`] #[default] Info, /// Operation for [`crate::raw::Access::create_dir`] CreateDir, /// Operation for [`crate::raw::Access::read`] Read, /// Operation for [`crate::raw::oio::Read::read`] ReaderRead, /// Operation for [`crate::raw::Access::write`] Write, /// Operation for [`crate::raw::oio::Write::write`] WriterWrite, /// Operation for [`crate::raw::oio::Write::close`] WriterClose, /// Operation for [`crate::raw::oio::Write::abort`] WriterAbort, /// Operation for [`crate::raw::Access::copy`] Copy, /// Operation for [`crate::raw::Access::rename`] Rename, /// Operation for [`crate::raw::Access::stat`] Stat, /// Operation for [`crate::raw::Access::delete`] Delete, /// Operation for [`crate::raw::oio::Delete::delete`] DeleterDelete, /// Operation for [`crate::raw::oio::Delete::flush`] DeleterFlush, /// Operation for [`crate::raw::Access::list`] List, /// Operation for [`crate::raw::oio::List::next`] ListerNext, /// Operation for [`crate::raw::Access::presign`] Presign, /// Operation for [`crate::raw::Access::blocking_create_dir`] BlockingCreateDir, /// Operation for [`crate::raw::Access::blocking_read`] BlockingRead, /// Operation for [`crate::raw::oio::BlockingRead::read`] BlockingReaderRead, /// Operation for [`crate::raw::Access::blocking_write`] BlockingWrite, /// Operation for [`crate::raw::oio::BlockingWrite::write`] BlockingWriterWrite, /// Operation for [`crate::raw::oio::BlockingWrite::close`] BlockingWriterClose, /// Operation for [`crate::raw::Access::blocking_copy`] BlockingCopy, /// Operation for [`crate::raw::Access::blocking_rename`] BlockingRename, /// Operation for [`crate::raw::Access::blocking_stat`] BlockingStat, /// Operation for [`crate::raw::Access::blocking_delete`] BlockingDelete, /// Operation for [`crate::raw::oio::BlockingDelete::delete`] BlockingDeleterDelete, /// Operation for [`crate::raw::oio::BlockingDelete::flush`] BlockingDeleterFlush, /// Operation for [`crate::raw::Access::blocking_list`] BlockingList, /// Operation for [`crate::raw::oio::BlockingList::next`] BlockingListerNext, } impl Operation { /// Convert self into static str. pub fn into_static(self) -> &'static str { self.into() } /// Check if given operation is oneshot or not. /// /// For example, `Stat` is oneshot but `ReaderRead` could happen multiple times. /// /// This function can be used to decide take actions based on operations like logging. pub fn is_oneshot(&self) -> bool { !matches!( self, Operation::ReaderRead | Operation::WriterWrite | Operation::ListerNext | Operation::DeleterDelete | Operation::BlockingReaderRead | Operation::BlockingWriterWrite | Operation::BlockingListerNext | Operation::BlockingDeleterDelete ) } } impl Display for Operation { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { write!(f, "{}", self.into_static()) } } impl From for &'static str { fn from(v: Operation) -> &'static str { match v { Operation::Info => "metadata", Operation::CreateDir => "create_dir", Operation::Read => "read", Operation::ReaderRead => "Reader::read", Operation::Write => "write", Operation::WriterWrite => "Writer::write", Operation::WriterClose => "Writer::close", Operation::WriterAbort => "Writer::abort", Operation::Copy => "copy", Operation::Rename => "rename", Operation::Stat => "stat", Operation::Delete => "delete", Operation::List => "list", Operation::ListerNext => "List::next", Operation::Presign => "presign", Operation::BlockingCreateDir => "blocking_create_dir", Operation::BlockingRead => "blocking_read", Operation::BlockingReaderRead => "BlockingReader::read", Operation::BlockingWrite => "blocking_write", Operation::BlockingWriterWrite => "BlockingWriter::write", Operation::BlockingWriterClose => "BlockingWriter::close", Operation::BlockingCopy => "blocking_copy", Operation::BlockingRename => "blocking_rename", Operation::BlockingStat => "blocking_stat", Operation::BlockingDelete => "blocking_delete", Operation::BlockingList => "blocking_list", Operation::BlockingListerNext => "BlockingLister::next", Operation::DeleterDelete => "Deleter::delete", Operation::DeleterFlush => "Deleter::flush", Operation::BlockingDeleterDelete => "BlockingDeleter::delete", Operation::BlockingDeleterFlush => "BlockingDeleter::flush", } } } impl From for String { fn from(v: Operation) -> Self { v.into_static().to_string() } } opendal-0.52.0/src/raw/ops.rs000064400000000000000000000564231046102023000140730ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Ops provides the operation args struct like [`OpRead`] for user. //! //! By using ops, users can add more context for operation. use crate::raw::*; use crate::*; use chrono::{DateTime, Utc}; use std::collections::HashMap; use std::time::Duration; /// Args for `create` operation. /// /// The path must be normalized. #[derive(Debug, Clone, Default)] pub struct OpCreateDir {} impl OpCreateDir { /// Create a new `OpCreateDir`. pub fn new() -> Self { Self::default() } } /// Args for `delete` operation. /// /// The path must be normalized. #[derive(Debug, Clone, Default, Eq, Hash, PartialEq)] pub struct OpDelete { version: Option, } impl OpDelete { /// Create a new `OpDelete`. pub fn new() -> Self { Self::default() } } impl OpDelete { /// Change the version of this delete operation. pub fn with_version(mut self, version: &str) -> Self { self.version = Some(version.into()); self } /// Get the version of this delete operation. pub fn version(&self) -> Option<&str> { self.version.as_deref() } } /// Args for `delete` operation. /// /// The path must be normalized. #[derive(Debug, Clone, Default)] pub struct OpDeleter {} impl OpDeleter { /// Create a new `OpDelete`. pub fn new() -> Self { Self::default() } } /// Args for `list` operation. #[derive(Debug, Clone)] pub struct OpList { /// The limit passed to underlying service to specify the max results /// that could return per-request. /// /// Users could use this to control the memory usage of list operation. limit: Option, /// The start_after passes to underlying service to specify the specified key /// to start listing from. start_after: Option, /// The recursive is used to control whether the list operation is recursive. /// /// - If `false`, list operation will only list the entries under the given path. /// - If `true`, list operation will list all entries that starts with given path. /// /// Default to `false`. recursive: bool, /// The concurrent of stat operations inside list operation. /// Users could use this to control the number of concurrent stat operation when metadata is unknown. /// /// - If this is set to <= 1, the list operation will be sequential. /// - If this is set to > 1, the list operation will be concurrent, /// and the maximum number of concurrent operations will be determined by this value. concurrent: usize, /// The version is used to control whether the object versions should be returned. /// /// - If `false`, list operation will not return with object versions /// - If `true`, list operation will return with object versions if object versioning is supported /// by the underlying service /// /// Default to `false` versions: bool, /// The deleted is used to control whether the deleted objects should be returned. /// /// - If `false`, list operation will not return with deleted objects /// - If `true`, list operation will return with deleted objects if object versioning is supported /// by the underlying service /// /// Default to `false` deleted: bool, } impl Default for OpList { fn default() -> Self { OpList { limit: None, start_after: None, recursive: false, concurrent: 1, versions: false, deleted: false, } } } impl OpList { /// Create a new `OpList`. pub fn new() -> Self { Self::default() } /// Change the limit of this list operation. pub fn with_limit(mut self, limit: usize) -> Self { self.limit = Some(limit); self } /// Get the limit of list operation. pub fn limit(&self) -> Option { self.limit } /// Change the start_after of this list operation. pub fn with_start_after(mut self, start_after: &str) -> Self { self.start_after = Some(start_after.into()); self } /// Get the start_after of list operation. pub fn start_after(&self) -> Option<&str> { self.start_after.as_deref() } /// The recursive is used to control whether the list operation is recursive. /// /// - If `false`, list operation will only list the entries under the given path. /// - If `true`, list operation will list all entries that starts with given path. /// /// Default to `false`. pub fn with_recursive(mut self, recursive: bool) -> Self { self.recursive = recursive; self } /// Get the current recursive. pub fn recursive(&self) -> bool { self.recursive } /// Change the concurrent of this list operation. /// /// The default concurrent is 1. pub fn with_concurrent(mut self, concurrent: usize) -> Self { self.concurrent = concurrent; self } /// Get the concurrent of list operation. pub fn concurrent(&self) -> usize { self.concurrent } /// Change the version of this list operation #[deprecated(since = "0.51.1", note = "use with_versions instead")] pub fn with_version(mut self, version: bool) -> Self { self.versions = version; self } /// Change the version of this list operation pub fn with_versions(mut self, versions: bool) -> Self { self.versions = versions; self } /// Get the version of this list operation #[deprecated(since = "0.51.1", note = "use versions instead")] pub fn version(&self) -> bool { self.versions } /// Get the version of this list operation pub fn versions(&self) -> bool { self.versions } /// Change the deleted of this list operation pub fn with_deleted(mut self, deleted: bool) -> Self { self.deleted = deleted; self } /// Get the deleted of this list operation pub fn deleted(&self) -> bool { self.deleted } } /// Args for `presign` operation. /// /// The path must be normalized. #[derive(Debug, Clone)] pub struct OpPresign { expire: Duration, op: PresignOperation, } impl OpPresign { /// Create a new `OpPresign`. pub fn new(op: impl Into, expire: Duration) -> Self { Self { op: op.into(), expire, } } /// Get operation from op. pub fn operation(&self) -> &PresignOperation { &self.op } /// Get expire from op. pub fn expire(&self) -> Duration { self.expire } /// Consume OpPresign into (Duration, PresignOperation) pub fn into_parts(self) -> (Duration, PresignOperation) { (self.expire, self.op) } } /// Presign operation used for presign. #[derive(Debug, Clone)] #[non_exhaustive] pub enum PresignOperation { /// Presign a stat(head) operation. Stat(OpStat), /// Presign a read operation. Read(OpRead), /// Presign a write operation. Write(OpWrite), } impl From for PresignOperation { fn from(op: OpStat) -> Self { Self::Stat(op) } } impl From for PresignOperation { fn from(v: OpRead) -> Self { Self::Read(v) } } impl From for PresignOperation { fn from(v: OpWrite) -> Self { Self::Write(v) } } /// Args for `read` operation. #[derive(Debug, Clone, Default)] pub struct OpRead { range: BytesRange, if_match: Option, if_none_match: Option, if_modified_since: Option>, if_unmodified_since: Option>, override_content_type: Option, override_cache_control: Option, override_content_disposition: Option, version: Option, executor: Option, } impl OpRead { /// Create a default `OpRead` which will read whole content of path. pub fn new() -> Self { Self::default() } /// Set the range of the option pub fn with_range(mut self, range: BytesRange) -> Self { self.range = range; self } /// Get range from option pub fn range(&self) -> BytesRange { self.range } /// Returns a mutable range to allow updating. pub(crate) fn range_mut(&mut self) -> &mut BytesRange { &mut self.range } /// Sets the content-disposition header that should be sent back by the remote read operation. pub fn with_override_content_disposition(mut self, content_disposition: &str) -> Self { self.override_content_disposition = Some(content_disposition.into()); self } /// Returns the content-disposition header that should be sent back by the remote read /// operation. pub fn override_content_disposition(&self) -> Option<&str> { self.override_content_disposition.as_deref() } /// Sets the cache-control header that should be sent back by the remote read operation. pub fn with_override_cache_control(mut self, cache_control: &str) -> Self { self.override_cache_control = Some(cache_control.into()); self } /// Returns the cache-control header that should be sent back by the remote read operation. pub fn override_cache_control(&self) -> Option<&str> { self.override_cache_control.as_deref() } /// Sets the content-type header that should be sent back by the remote read operation. pub fn with_override_content_type(mut self, content_type: &str) -> Self { self.override_content_type = Some(content_type.into()); self } /// Returns the content-type header that should be sent back by the remote read operation. pub fn override_content_type(&self) -> Option<&str> { self.override_content_type.as_deref() } /// Set the If-Match of the option pub fn with_if_match(mut self, if_match: &str) -> Self { self.if_match = Some(if_match.to_string()); self } /// Get If-Match from option pub fn if_match(&self) -> Option<&str> { self.if_match.as_deref() } /// Set the If-None-Match of the option pub fn with_if_none_match(mut self, if_none_match: &str) -> Self { self.if_none_match = Some(if_none_match.to_string()); self } /// Get If-None-Match from option pub fn if_none_match(&self) -> Option<&str> { self.if_none_match.as_deref() } /// Set the If-Modified-Since of the option pub fn with_if_modified_since(mut self, v: DateTime) -> Self { self.if_modified_since = Some(v); self } /// Get If-Modified-Since from option pub fn if_modified_since(&self) -> Option> { self.if_modified_since } /// Set the If-Unmodified-Since of the option pub fn with_if_unmodified_since(mut self, v: DateTime) -> Self { self.if_unmodified_since = Some(v); self } /// Get If-Unmodified-Since from option pub fn if_unmodified_since(&self) -> Option> { self.if_unmodified_since } /// Set the version of the option pub fn with_version(mut self, version: &str) -> Self { self.version = Some(version.to_string()); self } /// Get version from option pub fn version(&self) -> Option<&str> { self.version.as_deref() } /// Set the executor of the option pub fn with_executor(mut self, executor: Executor) -> Self { self.executor = Some(executor); self } /// Merge given executor into option. /// /// If executor has already been set, this will do nothing. /// Otherwise, this will set the given executor. pub(crate) fn merge_executor(self, executor: Option) -> Self { if self.executor.is_some() { return self; } if let Some(exec) = executor { return self.with_executor(exec); } self } /// Get executor from option pub fn executor(&self) -> Option<&Executor> { self.executor.as_ref() } } /// Args for reader operation. #[derive(Debug, Clone)] pub struct OpReader { /// The concurrent requests that reader can send. concurrent: usize, /// The chunk size of each request. chunk: Option, /// The gap size of each request. gap: Option, } impl Default for OpReader { fn default() -> Self { Self { concurrent: 1, chunk: None, gap: None, } } } impl OpReader { /// Create a new `OpReader`. pub fn new() -> Self { Self::default() } /// Set the concurrent of the option pub fn with_concurrent(mut self, concurrent: usize) -> Self { self.concurrent = concurrent.max(1); self } /// Get concurrent from option pub fn concurrent(&self) -> usize { self.concurrent } /// Set the chunk of the option pub fn with_chunk(mut self, chunk: usize) -> Self { self.chunk = Some(chunk.max(1)); self } /// Get chunk from option pub fn chunk(&self) -> Option { self.chunk } /// Set the gap of the option pub fn with_gap(mut self, gap: usize) -> Self { self.gap = Some(gap.max(1)); self } /// Get gap from option pub fn gap(&self) -> Option { self.gap } } /// Args for `stat` operation. #[derive(Debug, Clone, Default)] pub struct OpStat { if_match: Option, if_none_match: Option, if_modified_since: Option>, if_unmodified_since: Option>, override_content_type: Option, override_cache_control: Option, override_content_disposition: Option, version: Option, } impl OpStat { /// Create a new `OpStat`. pub fn new() -> Self { Self::default() } /// Set the If-Match of the option pub fn with_if_match(mut self, if_match: &str) -> Self { self.if_match = Some(if_match.to_string()); self } /// Get If-Match from option pub fn if_match(&self) -> Option<&str> { self.if_match.as_deref() } /// Set the If-None-Match of the option pub fn with_if_none_match(mut self, if_none_match: &str) -> Self { self.if_none_match = Some(if_none_match.to_string()); self } /// Get If-None-Match from option pub fn if_none_match(&self) -> Option<&str> { self.if_none_match.as_deref() } /// Set the If-Modified-Since of the option pub fn with_if_modified_since(mut self, v: DateTime) -> Self { self.if_modified_since = Some(v); self } /// Get If-Modified-Since from option pub fn if_modified_since(&self) -> Option> { self.if_modified_since } /// Set the If-Unmodified-Since of the option pub fn with_if_unmodified_since(mut self, v: DateTime) -> Self { self.if_unmodified_since = Some(v); self } /// Get If-Unmodified-Since from option pub fn if_unmodified_since(&self) -> Option> { self.if_unmodified_since } /// Sets the content-disposition header that should be sent back by the remote read operation. pub fn with_override_content_disposition(mut self, content_disposition: &str) -> Self { self.override_content_disposition = Some(content_disposition.into()); self } /// Returns the content-disposition header that should be sent back by the remote read /// operation. pub fn override_content_disposition(&self) -> Option<&str> { self.override_content_disposition.as_deref() } /// Sets the cache-control header that should be sent back by the remote read operation. pub fn with_override_cache_control(mut self, cache_control: &str) -> Self { self.override_cache_control = Some(cache_control.into()); self } /// Returns the cache-control header that should be sent back by the remote read operation. pub fn override_cache_control(&self) -> Option<&str> { self.override_cache_control.as_deref() } /// Sets the content-type header that should be sent back by the remote read operation. pub fn with_override_content_type(mut self, content_type: &str) -> Self { self.override_content_type = Some(content_type.into()); self } /// Returns the content-type header that should be sent back by the remote read operation. pub fn override_content_type(&self) -> Option<&str> { self.override_content_type.as_deref() } /// Set the version of the option pub fn with_version(mut self, version: &str) -> Self { self.version = Some(version.to_string()); self } /// Get version from option pub fn version(&self) -> Option<&str> { self.version.as_deref() } } /// Args for `write` operation. #[derive(Debug, Clone, Default)] pub struct OpWrite { append: bool, concurrent: usize, content_type: Option, content_disposition: Option, content_encoding: Option, cache_control: Option, executor: Option, if_match: Option, if_none_match: Option, if_not_exists: bool, user_metadata: Option>, } impl OpWrite { /// Create a new `OpWrite`. /// /// If input path is not a file path, an error will be returned. pub fn new() -> Self { Self::default() } /// Get the append from op. /// /// The append is the flag to indicate that this write operation is an append operation. pub fn append(&self) -> bool { self.append } /// Set the append mode of op. /// /// If the append mode is set, the data will be appended to the end of the file. /// /// # Notes /// /// Service could return `Unsupported` if the underlying storage does not support append. pub fn with_append(mut self, append: bool) -> Self { self.append = append; self } /// Get the content type from option pub fn content_type(&self) -> Option<&str> { self.content_type.as_deref() } /// Set the content type of option pub fn with_content_type(mut self, content_type: &str) -> Self { self.content_type = Some(content_type.to_string()); self } /// Get the content disposition from option pub fn content_disposition(&self) -> Option<&str> { self.content_disposition.as_deref() } /// Set the content disposition of option pub fn with_content_disposition(mut self, content_disposition: &str) -> Self { self.content_disposition = Some(content_disposition.to_string()); self } /// Get the content encoding from option pub fn content_encoding(&self) -> Option<&str> { self.content_encoding.as_deref() } /// Set the content encoding of option pub fn with_content_encoding(mut self, content_encoding: &str) -> Self { self.content_encoding = Some(content_encoding.to_string()); self } /// Get the cache control from option pub fn cache_control(&self) -> Option<&str> { self.cache_control.as_deref() } /// Set the content type of option pub fn with_cache_control(mut self, cache_control: &str) -> Self { self.cache_control = Some(cache_control.to_string()); self } /// Get the concurrent. pub fn concurrent(&self) -> usize { self.concurrent } /// Set the maximum concurrent write task amount. pub fn with_concurrent(mut self, concurrent: usize) -> Self { self.concurrent = concurrent; self } /// Get the executor from option pub fn executor(&self) -> Option<&Executor> { self.executor.as_ref() } /// Set the executor of the option pub fn with_executor(mut self, executor: Executor) -> Self { self.executor = Some(executor); self } /// Set the If-Match of the option pub fn with_if_match(mut self, s: &str) -> Self { self.if_match = Some(s.to_string()); self } /// Get If-Match from option pub fn if_match(&self) -> Option<&str> { self.if_match.as_deref() } /// Set the If-None-Match of the option pub fn with_if_none_match(mut self, s: &str) -> Self { self.if_none_match = Some(s.to_string()); self } /// Get If-None-Match from option pub fn if_none_match(&self) -> Option<&str> { self.if_none_match.as_deref() } /// Set the If-Not-Exist of the option pub fn with_if_not_exists(mut self, b: bool) -> Self { self.if_not_exists = b; self } /// Get If-Not-Exist from option pub fn if_not_exists(&self) -> bool { self.if_not_exists } /// Merge given executor into option. /// /// If executor has already been set, this will do nothing. /// Otherwise, this will set the given executor. pub(crate) fn merge_executor(self, executor: Option) -> Self { if self.executor.is_some() { return self; } if let Some(exec) = executor { return self.with_executor(exec); } self } /// Set the user defined metadata of the op pub fn with_user_metadata(mut self, metadata: HashMap) -> Self { self.user_metadata = Some(metadata); self } /// Get the user defined metadata from the op pub fn user_metadata(&self) -> Option<&HashMap> { self.user_metadata.as_ref() } } /// Args for `writer` operation. #[derive(Debug, Clone, Default)] pub struct OpWriter { chunk: Option, } impl OpWriter { /// Create a new `OpWriter`. pub fn new() -> Self { Self::default() } /// Get the chunk from op. /// /// The chunk is used by service to decide the chunk size of the underlying writer. pub fn chunk(&self) -> Option { self.chunk } /// Set the chunk of op. /// /// If chunk is set, the data will be chunked by the underlying writer. /// /// ## NOTE /// /// Service could have their own minimum chunk size while perform write /// operations like multipart uploads. So the chunk size may be larger than /// the given buffer size. pub fn with_chunk(mut self, chunk: usize) -> Self { self.chunk = Some(chunk); self } } /// Args for `copy` operation. #[derive(Debug, Clone, Default)] pub struct OpCopy {} impl OpCopy { /// Create a new `OpCopy`. pub fn new() -> Self { Self::default() } } /// Args for `rename` operation. #[derive(Debug, Clone, Default)] pub struct OpRename {} impl OpRename { /// Create a new `OpMove`. pub fn new() -> Self { Self::default() } } opendal-0.52.0/src/raw/path.rs000064400000000000000000000266001046102023000142200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::*; /// build_abs_path will build an absolute path with root. /// /// # Rules /// /// - Input root MUST be the format like `/abc/def/` /// - Output will be the format like `path/to/root/path`. pub fn build_abs_path(root: &str, path: &str) -> String { debug_assert!(root.starts_with('/'), "root must start with /"); debug_assert!(root.ends_with('/'), "root must end with /"); let p = root[1..].to_string(); if path == "/" { p } else { debug_assert!(!path.starts_with('/'), "path must not start with /"); p + path } } /// build_rooted_abs_path will build an absolute path with root. /// /// # Rules /// /// - Input root MUST be the format like `/abc/def/` /// - Output will be the format like `/path/to/root/path`. pub fn build_rooted_abs_path(root: &str, path: &str) -> String { debug_assert!(root.starts_with('/'), "root must start with /"); debug_assert!(root.ends_with('/'), "root must end with /"); let p = root.to_string(); if path == "/" { p } else { debug_assert!(!path.starts_with('/'), "path must not start with /"); p + path } } /// build_rel_path will build a relative path towards root. /// /// # Rules /// /// - Input root MUST be the format like `/abc/def/` /// - Input path MUST start with root like `/abc/def/path/to/file` /// - Output will be the format like `path/to/file`. pub fn build_rel_path(root: &str, path: &str) -> String { debug_assert!(root != path, "get rel path with root is invalid"); if path.starts_with('/') { debug_assert!( path.starts_with(root), "path {path} doesn't start with root {root}" ); path[root.len()..].to_string() } else { debug_assert!( path.starts_with(&root[1..]), "path {path} doesn't start with root {root}" ); path[root.len() - 1..].to_string() } } /// Make sure all operation are constructed by normalized path: /// /// - Path endswith `/` means it's a dir path. /// - Otherwise, it's a file path. /// /// # Normalize Rules /// /// - All whitespace will be trimmed: ` abc/def ` => `abc/def` /// - All leading / will be trimmed: `///abc` => `abc` /// - Internal // will be replaced by /: `abc///def` => `abc/def` /// - Empty path will be `/`: `` => `/` pub fn normalize_path(path: &str) -> String { // - all whitespace has been trimmed. // - all leading `/` has been trimmed. let path = path.trim().trim_start_matches('/'); // Fast line for empty path. if path.is_empty() { return "/".to_string(); } let has_trailing = path.ends_with('/'); let mut p = path .split('/') .filter(|v| !v.is_empty()) .collect::>() .join("/"); // Append trailing back if input path is endswith `/`. if has_trailing { p.push('/'); } p } /// Make sure root is normalized to style like `/abc/def/`. /// /// # Normalize Rules /// /// - All whitespace will be trimmed: ` abc/def ` => `abc/def` /// - All leading / will be trimmed: `///abc` => `abc` /// - Internal // will be replaced by /: `abc///def` => `abc/def` /// - Empty path will be `/`: `` => `/` /// - Add leading `/` if not starts with: `abc/` => `/abc/` /// - Add trailing `/` if not ends with: `/abc` => `/abc/` /// /// Finally, we will get path like `/path/to/root/`. pub fn normalize_root(v: &str) -> String { let mut v = v .split('/') .filter(|v| !v.is_empty()) .collect::>() .join("/"); if !v.starts_with('/') { v.insert(0, '/'); } if !v.ends_with('/') { v.push('/') } v } /// Get basename from path. pub fn get_basename(path: &str) -> &str { // Handle root case if path == "/" { return "/"; } // Handle file case if !path.ends_with('/') { return path .split('/') .next_back() .expect("file path without name is invalid"); } // The idx of second `/` if path in reserve order. // - `abc/` => `None` // - `abc/def/` => `Some(3)` let idx = path[..path.len() - 1].rfind('/').map(|v| v + 1); match idx { Some(v) => { let (_, name) = path.split_at(v); name } None => path, } } /// Get parent from path. pub fn get_parent(path: &str) -> &str { if path == "/" { return "/"; } if !path.ends_with('/') { // The idx of first `/` if path in reserve order. // - `abc` => `None` // - `abc/def` => `Some(3)` let idx = path.rfind('/'); return match idx { Some(v) => { let (parent, _) = path.split_at(v + 1); parent } None => "/", }; } // The idx of second `/` if path in reserve order. // - `abc/` => `None` // - `abc/def/` => `Some(3)` let idx = path[..path.len() - 1].rfind('/').map(|v| v + 1); match idx { Some(v) => { let (parent, _) = path.split_at(v); parent } None => "/", } } /// Validate given path is match with given EntryMode. pub fn validate_path(path: &str, mode: EntryMode) -> bool { debug_assert!(!path.is_empty(), "input path should not be empty"); match mode { EntryMode::FILE => !path.ends_with('/'), EntryMode::DIR => path.ends_with('/'), EntryMode::Unknown => false, } } #[cfg(test)] mod tests { use super::*; #[test] fn test_normalize_path() { let cases = vec![ ("file path", "abc", "abc"), ("dir path", "abc/", "abc/"), ("empty path", "", "/"), ("root path", "/", "/"), ("root path with extra /", "///", "/"), ("abs file path", "/abc/def", "abc/def"), ("abs dir path", "/abc/def/", "abc/def/"), ("abs file path with extra /", "///abc/def", "abc/def"), ("abs dir path with extra /", "///abc/def/", "abc/def/"), ("file path contains ///", "abc///def", "abc/def"), ("dir path contains ///", "abc///def///", "abc/def/"), ("file with whitespace", "abc/def ", "abc/def"), ]; for (name, input, expect) in cases { assert_eq!(normalize_path(input), expect, "{name}") } } #[test] fn test_normalize_root() { let cases = vec![ ("dir path", "abc/", "/abc/"), ("empty path", "", "/"), ("root path", "/", "/"), ("root path with extra /", "///", "/"), ("abs dir path", "/abc/def/", "/abc/def/"), ("abs file path with extra /", "///abc/def", "/abc/def/"), ("abs dir path with extra /", "///abc/def/", "/abc/def/"), ("dir path contains ///", "abc///def///", "/abc/def/"), ]; for (name, input, expect) in cases { assert_eq!(normalize_root(input), expect, "{name}") } } #[test] fn test_get_basename() { let cases = vec![ ("file abs path", "foo/bar/baz.txt", "baz.txt"), ("file rel path", "bar/baz.txt", "baz.txt"), ("file walk", "foo/bar/baz", "baz"), ("dir rel path", "bar/baz/", "baz/"), ("dir root", "/", "/"), ("dir walk", "foo/bar/baz/", "baz/"), ]; for (name, input, expect) in cases { let actual = get_basename(input); assert_eq!(actual, expect, "{name}") } } #[test] fn test_get_parent() { let cases = vec![ ("file abs path", "foo/bar/baz.txt", "foo/bar/"), ("file rel path", "bar/baz.txt", "bar/"), ("file walk", "foo/bar/baz", "foo/bar/"), ("dir rel path", "bar/baz/", "bar/"), ("dir root", "/", "/"), ("dir walk", "foo/bar/baz/", "foo/bar/"), ]; for (name, input, expect) in cases { let actual = get_parent(input); assert_eq!(actual, expect, "{name}") } } #[test] fn test_build_abs_path() { let cases = vec![ ("input abs file", "/abc/", "/", "abc/"), ("input dir", "/abc/", "def/", "abc/def/"), ("input file", "/abc/", "def", "abc/def"), ("input abs file with root /", "/", "/", ""), ("input empty with root /", "/", "", ""), ("input dir with root /", "/", "def/", "def/"), ("input file with root /", "/", "def", "def"), ]; for (name, root, input, expect) in cases { let actual = build_abs_path(root, input); assert_eq!(actual, expect, "{name}") } } #[test] fn test_build_rooted_abs_path() { let cases = vec![ ("input abs file", "/abc/", "/", "/abc/"), ("input dir", "/abc/", "def/", "/abc/def/"), ("input file", "/abc/", "def", "/abc/def"), ("input abs file with root /", "/", "/", "/"), ("input dir with root /", "/", "def/", "/def/"), ("input file with root /", "/", "def", "/def"), ]; for (name, root, input, expect) in cases { let actual = build_rooted_abs_path(root, input); assert_eq!(actual, expect, "{name}") } } #[test] fn test_build_rel_path() { let cases = vec![ ("input abs file", "/abc/", "/abc/def", "def"), ("input dir", "/abc/", "/abc/def/", "def/"), ("input file", "/abc/", "abc/def", "def"), ("input dir with root /", "/", "def/", "def/"), ("input file with root /", "/", "def", "def"), ]; for (name, root, input, expect) in cases { let actual = build_rel_path(root, input); assert_eq!(actual, expect, "{name}") } } #[test] fn test_validate_path() { let cases = vec![ ("input file with mode file", "abc", EntryMode::FILE, true), ("input file with mode dir", "abc", EntryMode::DIR, false), ("input dir with mode file", "abc/", EntryMode::FILE, false), ("input dir with mode dir", "abc/", EntryMode::DIR, true), ("root with mode dir", "/", EntryMode::DIR, true), ( "input file with mode unknown", "abc", EntryMode::Unknown, false, ), ( "input dir with mode unknown", "abc/", EntryMode::Unknown, false, ), ]; for (name, path, mode, expect) in cases { let actual = validate_path(path, mode); assert_eq!(actual, expect, "{name}") } } } opendal-0.52.0/src/raw/path_cache.rs000064400000000000000000000172431046102023000153460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::VecDeque; use futures::Future; use moka::sync::Cache; use tokio::sync::Mutex; use tokio::sync::MutexGuard; use crate::raw::*; use crate::*; /// The trait required for path cacher. pub trait PathQuery { /// Fetch the id for the root of the service. fn root(&self) -> impl Future> + MaybeSend; /// Query the id by parent_id and name. fn query( &self, parent_id: &str, name: &str, ) -> impl Future>> + MaybeSend; /// Create a dir by parent_id and name. fn create_dir( &self, parent_id: &str, name: &str, ) -> impl Future> + MaybeSend; } /// PathCacher is a cache for path query. /// /// OpenDAL is designed for path based storage systems, such as S3, HDFS, etc. But there are many /// services that are not path based, such as OneDrive, Google Drive, etc. For these services, we /// look up files based on id. The lookup of id is very expensive, so we cache the path to id mapping /// in PathCacher. /// /// # Behavior /// /// The `path` in the cache is always an absolute one. For example, if the service root is `/root/`, /// then the path of file `a/b` in cache will be `/root/a/b`. pub struct PathCacher { query: Q, cache: Cache, /// This optional lock here is used to prevent concurrent insertions of the same path. /// /// Some services like gdrive allows the same name to exist in the same directory. We need to introduce /// a global lock to prevent concurrent insertions of the same path. lock: Option>, } impl PathCacher { /// Create a new path cacher. pub fn new(query: Q) -> Self { Self { query, cache: Cache::new(64 * 1024), lock: None, } } /// Enable the lock for the path cacher. pub fn with_lock(mut self) -> Self { self.lock = Some(Mutex::default()); self } async fn lock(&self) -> Option> { if let Some(l) = &self.lock { Some(l.lock().await) } else { None } } /// Insert a new cache entry. pub async fn insert(&self, path: &str, id: &str) { let _guard = self.lock().await; // This should never happen, but let's ignore the insert if happened. if self.cache.contains_key(path) { debug_assert!( self.cache.get(path) == Some(id.to_string()), "path {path} exists but it's value is inconsistent" ); return; } self.cache.insert(path.to_string(), id.to_string()); } /// Remove a cache entry. pub async fn remove(&self, path: &str) { let _guard = self.lock().await; self.cache.invalidate(path) } /// Get the id for the given path. pub async fn get(&self, path: &str) -> Result> { let _guard = self.lock().await; if let Some(id) = self.cache.get(path) { return Ok(Some(id)); } let mut paths = VecDeque::new(); let mut current_path = path; while current_path != "/" && !current_path.is_empty() { paths.push_front(current_path.to_string()); current_path = get_parent(current_path); if let Some(id) = self.cache.get(current_path) { return self.query_down(&id, paths).await; } } let root_id = self.query.root().await?; self.cache.insert("/".to_string(), root_id.clone()); self.query_down(&root_id, paths).await } /// `start_id` is the `file_id` to the start dir to query down. /// `paths` is in the order like `["/a/", "/a/b/", "/a/b/c/"]`. /// /// We should fetch the next `file_id` by sending `query`. async fn query_down(&self, start_id: &str, paths: VecDeque) -> Result> { let mut current_id = start_id.to_string(); for path in paths.into_iter() { let name = get_basename(&path); current_id = match self.query.query(¤t_id, name).await? { Some(id) => { self.cache.insert(path, id.clone()); id } None => return Ok(None), }; } Ok(Some(current_id)) } /// Ensure input dir exists. pub async fn ensure_dir(&self, path: &str) -> Result { let _guard = self.lock().await; let mut tmp = "".to_string(); // All parents that need to check. let mut parents = vec![]; for component in path.split('/') { if component.is_empty() { continue; } tmp.push_str(component); tmp.push('/'); parents.push(tmp.to_string()); } let mut parent_id = match self.cache.get("/") { Some(v) => v, None => self.query.root().await?, }; for parent in parents { parent_id = match self.cache.get(&parent) { Some(value) => value, None => { let value = match self.query.query(&parent_id, get_basename(&parent)).await? { Some(value) => value, None => { self.query .create_dir(&parent_id, get_basename(&parent)) .await? } }; self.cache.insert(parent, value.clone()); value } } } Ok(parent_id) } } #[cfg(test)] mod tests { use crate::raw::PathCacher; use crate::raw::PathQuery; use crate::*; struct TestQuery {} impl PathQuery for TestQuery { async fn root(&self) -> Result { Ok("root/".to_string()) } async fn query(&self, parent_id: &str, name: &str) -> Result> { if name.starts_with("not_exist") { return Ok(None); } Ok(Some(format!("{parent_id}{name}"))) } async fn create_dir(&self, parent_id: &str, name: &str) -> Result { Ok(format!("{parent_id}{name}")) } } #[tokio::test] async fn test_path_cacher_get() { let cases = vec![ ("root", "/", Some("root/")), ("normal path", "/a", Some("root/a")), ("not exist normal dir", "/not_exist/a", None), ("not exist normal file", "/a/b/not_exist", None), ("nest path", "/a/b/c/d", Some("root/a/b/c/d")), ]; for (name, input, expect) in cases { let cache = PathCacher::new(TestQuery {}); let actual = cache.get(input).await.unwrap(); assert_eq!(actual.as_deref(), expect, "{}", name) } } } opendal-0.52.0/src/raw/rps.rs000064400000000000000000000152701046102023000140710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Request; use crate::raw::*; use crate::*; /// Reply for `create_dir` operation #[derive(Debug, Clone, Default)] pub struct RpCreateDir {} /// Reply for `delete` operation #[derive(Debug, Clone, Default)] pub struct RpDelete {} /// Reply for `list` operation. #[derive(Debug, Clone, Default)] pub struct RpList {} /// Reply for `presign` operation. #[derive(Debug, Clone)] pub struct RpPresign { req: PresignedRequest, } impl RpPresign { /// Create a new reply for `presign`. pub fn new(req: PresignedRequest) -> Self { RpPresign { req } } /// Consume reply to build a presigned request. pub fn into_presigned_request(self) -> PresignedRequest { self.req } } /// PresignedRequest is a presigned request return by `presign`. #[derive(Debug, Clone)] pub struct PresignedRequest { method: http::Method, uri: http::Uri, headers: http::HeaderMap, } impl PresignedRequest { /// Create a new PresignedRequest pub fn new(method: http::Method, uri: http::Uri, headers: http::HeaderMap) -> Self { Self { method, uri, headers, } } /// Return request's method. pub fn method(&self) -> &http::Method { &self.method } /// Return request's uri. pub fn uri(&self) -> &http::Uri { &self.uri } /// Return request's header. pub fn header(&self) -> &http::HeaderMap { &self.headers } } impl From for Request { fn from(v: PresignedRequest) -> Self { let mut builder = Request::builder().method(v.method).uri(v.uri); let headers = builder.headers_mut().expect("header map must be valid"); headers.extend(v.headers); builder .body(T::default()) .expect("request must build succeed") } } /// Reply for `read` operation. #[derive(Debug, Clone, Default)] pub struct RpRead { /// Size is the size of the reader returned by this read operation. /// /// - `Some(size)` means the reader has at most size bytes. /// - `None` means the reader has unknown size. /// /// It's ok to leave size as empty, but it's recommended to set size if possible. We will use /// this size as hint to do some optimization like avoid an extra stat or read. size: Option, /// Range is the range of the reader returned by this read operation. /// /// - `Some(range)` means the reader's content range inside the whole file. /// - `None` means the reader's content range is unknown. /// /// It's ok to leave range as empty, but it's recommended to set range if possible. We will use /// this range as hint to do some optimization like avoid an extra stat or read. range: Option, } impl RpRead { /// Create a new reply for `read`. pub fn new() -> Self { RpRead::default() } /// Got the size of the reader returned by this read operation. /// /// - `Some(size)` means the reader has at most size bytes. /// - `None` means the reader has unknown size. pub fn size(&self) -> Option { self.size } /// Set the size of the reader returned by this read operation. pub fn with_size(mut self, size: Option) -> Self { self.size = size; self } /// Got the range of the reader returned by this read operation. /// /// - `Some(range)` means the reader has content range inside the whole file. /// - `None` means the reader has unknown size. pub fn range(&self) -> Option { self.range } /// Set the range of the reader returned by this read operation. pub fn with_range(mut self, range: Option) -> Self { self.range = range; self } } /// Reply for `stat` operation. #[derive(Debug, Clone)] pub struct RpStat { meta: Metadata, } impl RpStat { /// Create a new reply for `stat`. pub fn new(meta: Metadata) -> Self { RpStat { meta } } /// Operate on inner metadata. pub fn map_metadata(mut self, f: impl FnOnce(Metadata) -> Metadata) -> Self { self.meta = f(self.meta); self } /// Consume RpStat to get the inner metadata. pub fn into_metadata(self) -> Metadata { self.meta } } /// Reply for `write` operation. #[derive(Debug, Clone, Default)] pub struct RpWrite {} impl RpWrite { /// Create a new reply for `write`. pub fn new() -> Self { Self {} } } /// Reply for `copy` operation. #[derive(Debug, Clone, Default)] pub struct RpCopy {} impl RpCopy { /// Create a new reply for `copy`. pub fn new() -> Self { Self {} } } /// Reply for `rename` operation. #[derive(Debug, Clone, Default)] pub struct RpRename {} impl RpRename { /// Create a new reply for `rename`. pub fn new() -> Self { Self {} } } #[cfg(test)] mod tests { use anyhow::Result; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::HeaderMap; use http::Method; use http::Uri; use super::*; #[test] fn test_presigned_request_convert() -> Result<()> { let pr = PresignedRequest { method: Method::PATCH, uri: Uri::from_static("https://opendal.apache.org/path/to/file"), headers: { let mut headers = HeaderMap::new(); headers.insert(CONTENT_LENGTH, "123".parse()?); headers.insert(CONTENT_TYPE, "application/json".parse()?); headers }, }; let req: Request = pr.into(); assert_eq!(Method::PATCH, req.method()); assert_eq!( "https://opendal.apache.org/path/to/file", req.uri().to_string() ); assert_eq!("123", req.headers().get(CONTENT_LENGTH).unwrap()); assert_eq!("application/json", req.headers().get(CONTENT_TYPE).unwrap()); Ok(()) } } opendal-0.52.0/src/raw/serde_util.rs000064400000000000000000000321121046102023000154160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::hash_map::IntoIter; use std::collections::HashMap; use std::iter::empty; use serde::de::value::MapDeserializer; use serde::de::value::SeqDeserializer; use serde::de::Deserializer; use serde::de::IntoDeserializer; use serde::de::Visitor; use serde::de::{self}; use crate::*; /// Parse xml deserialize error into opendal::Error. pub fn new_xml_deserialize_error(e: quick_xml::DeError) -> Error { Error::new(ErrorKind::Unexpected, "deserialize xml").set_source(e) } /// Parse json serialize error into opendal::Error. pub fn new_json_serialize_error(e: serde_json::Error) -> Error { Error::new(ErrorKind::Unexpected, "serialize json").set_source(e) } /// Parse json deserialize error into opendal::Error. pub fn new_json_deserialize_error(e: serde_json::Error) -> Error { Error::new(ErrorKind::Unexpected, "deserialize json").set_source(e) } /// ConfigDeserializer is used to deserialize given configs from `HashMap`. /// /// This is only used by our services' config. pub struct ConfigDeserializer(MapDeserializer<'static, Pairs, de::value::Error>); impl ConfigDeserializer { /// Create a new config deserializer. pub fn new(map: HashMap) -> Self { let pairs = Pairs(map.into_iter()); Self(MapDeserializer::new(pairs)) } } impl<'de> Deserializer<'de> for ConfigDeserializer { type Error = de::value::Error; fn deserialize_any(self, visitor: V) -> Result where V: Visitor<'de>, { self.deserialize_map(visitor) } fn deserialize_map(self, visitor: V) -> Result where V: Visitor<'de>, { visitor.visit_map(self.0) } serde::forward_to_deserialize_any! { bool u8 u16 u32 u64 i8 i16 i32 i64 f32 f64 char str string unit seq bytes byte_buf unit_struct tuple_struct identifier tuple ignored_any option newtype_struct enum struct } } /// Pairs is used to implement Iterator to meet the requirement of [`MapDeserializer`]. struct Pairs(IntoIter); impl Iterator for Pairs { type Item = (String, Pair); fn next(&mut self) -> Option { self.0.next().map(|(k, v)| (k.to_lowercase(), Pair(k, v))) } } /// Pair is used to hold both key and value of a config for better error output. struct Pair(String, String); impl IntoDeserializer<'_, de::value::Error> for Pair { type Deserializer = Self; fn into_deserializer(self) -> Self::Deserializer { self } } impl<'de> Deserializer<'de> for Pair { type Error = de::value::Error; fn deserialize_any(self, visitor: V) -> Result where V: Visitor<'de>, { self.1.into_deserializer().deserialize_any(visitor) } fn deserialize_bool(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.to_lowercase().as_str() { "true" | "on" => true.into_deserializer().deserialize_bool(visitor), "false" | "off" => false.into_deserializer().deserialize_bool(visitor), _ => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, "invalid bool value" ))), } } fn deserialize_i8(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_i8(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_i16(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_i16(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_i32(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_i32(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_i64(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_i64(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_u8(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_u8(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_u16(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_u16(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_u32(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_u32(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_u64(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_u64(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_f32(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_f32(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_f64(self, visitor: V) -> Result where V: Visitor<'de>, { match self.1.parse::() { Ok(val) => val.into_deserializer().deserialize_f64(visitor), Err(e) => Err(de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e ))), } } fn deserialize_option(self, visitor: V) -> Result where V: Visitor<'de>, { if self.1.is_empty() { visitor.visit_none() } else { visitor.visit_some(self) } } fn deserialize_seq(self, visitor: V) -> Result where V: Visitor<'de>, { // Return empty instead of `[""]`. if self.1.is_empty() { SeqDeserializer::new(empty::()) .deserialize_seq(visitor) .map_err(|e| { de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e )) }) } else { let values = self .1 .split(',') .map(|v| Pair(self.0.clone(), v.trim().to_owned())); SeqDeserializer::new(values) .deserialize_seq(visitor) .map_err(|e| { de::Error::custom(format_args!( "parse config '{}' with value '{}' failed for {:?}", self.0, self.1, e )) }) } } serde::forward_to_deserialize_any! { char str string unit newtype_struct enum bytes byte_buf map unit_struct tuple_struct identifier tuple ignored_any struct } } #[cfg(test)] mod tests { use serde::Deserialize; use super::*; #[derive(Debug, Default, Deserialize, Eq, PartialEq)] #[serde(default)] #[non_exhaustive] pub struct TestConfig { bool_value: bool, bool_option_value_none: Option, bool_option_value_some: Option, bool_value_with_on: bool, bool_value_with_off: bool, string_value: String, string_option_value_none: Option, string_option_value_some: Option, u8_value: u8, u16_value: u16, u32_value: u32, u64_value: u64, i8_value: i8, i16_value: i16, i32_value: i32, i64_value: i64, vec_value: Vec, vec_value_two: Vec, vec_none: Option>, vec_empty: Vec, } #[test] fn test_config_deserializer() { let mut map = HashMap::new(); map.insert("bool_value", "true"); map.insert("bool_option_value_none", ""); map.insert("bool_option_value_some", "false"); map.insert("bool_value_with_on", "on"); map.insert("bool_value_with_off", "off"); map.insert("string_value", "hello"); map.insert("string_option_value_none", ""); map.insert("string_option_value_some", "hello"); map.insert("u8_value", "8"); map.insert("u16_value", "16"); map.insert("u32_value", "32"); map.insert("u64_value", "64"); map.insert("i8_value", "-8"); map.insert("i16_value", "16"); map.insert("i32_value", "-32"); map.insert("i64_value", "64"); map.insert("vec_value", "hello"); map.insert("vec_value_two", "hello,world"); map.insert("vec_none", ""); map.insert("vec_empty", ""); let map = map .into_iter() .map(|(k, v)| (k.to_string(), v.to_string())) .collect(); let output = TestConfig::deserialize(ConfigDeserializer::new(map)).unwrap(); assert_eq!( output, TestConfig { bool_value: true, bool_option_value_none: None, bool_option_value_some: Some(false), bool_value_with_on: true, bool_value_with_off: false, string_value: "hello".to_string(), string_option_value_none: None, string_option_value_some: Some("hello".to_string()), u8_value: 8, u16_value: 16, u32_value: 32, u64_value: 64, i8_value: -8, i16_value: 16, i32_value: -32, i64_value: 64, vec_value: vec!["hello".to_string()], vec_value_two: vec!["hello".to_string(), "world".to_string()], vec_none: None, vec_empty: vec![], } ); } #[test] fn test_part_config_deserializer() { let mut map = HashMap::new(); map.insert("bool_value", "true"); let map = map .into_iter() .map(|(k, v)| (k.to_string(), v.to_string())) .collect(); let output = TestConfig::deserialize(ConfigDeserializer::new(map)).unwrap(); assert_eq!( output, TestConfig { bool_value: true, ..TestConfig::default() } ); } } opendal-0.52.0/src/raw/std_io_util.rs000064400000000000000000000042461046102023000156040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io; use crate::*; /// Parse std io error into opendal::Error. /// /// # TODO /// /// Add `NotADirectory` and `IsADirectory` once they are stable. /// /// ref: pub fn new_std_io_error(err: std::io::Error) -> Error { use std::io::ErrorKind::*; let (kind, retryable) = match err.kind() { NotFound => (ErrorKind::NotFound, false), PermissionDenied => (ErrorKind::PermissionDenied, false), AlreadyExists => (ErrorKind::AlreadyExists, false), Unsupported => (ErrorKind::Unsupported, false), Interrupted | UnexpectedEof | TimedOut | WouldBlock => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, true), }; let mut err = Error::new(kind, err.kind().to_string()).set_source(err); if retryable { err = err.set_temporary(); } err } /// helper functions to format `Error` into `io::Error`. /// /// This function is added privately by design and only valid in current /// context (i.e. `raw` mod). We don't want to expose this function to /// users. #[inline] pub(crate) fn format_std_io_error(err: Error) -> io::Error { let kind = match err.kind() { ErrorKind::NotFound => io::ErrorKind::NotFound, ErrorKind::PermissionDenied => io::ErrorKind::PermissionDenied, _ => io::ErrorKind::Interrupted, }; io::Error::new(kind, err) } opendal-0.52.0/src/raw/tests/mod.rs000064400000000000000000000020251046102023000152000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Utilities for opendal testing. mod read; pub use read::ReadAction; pub use read::ReadChecker; mod write; pub use write::WriteAction; pub use write::WriteChecker; mod utils; pub use utils::init_test_service; pub use utils::TEST_RUNTIME; opendal-0.52.0/src/raw/tests/read.rs000064400000000000000000000107311046102023000153370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Bytes; use rand::thread_rng; use rand::RngCore; use sha2::Digest; use sha2::Sha256; use crate::*; /// ReadAction represents a read action. #[derive(Debug, Clone, Copy, Eq, PartialEq)] pub enum ReadAction { /// Read represents a read action with given input buf size. /// /// # NOTE /// /// The size is the input buf size, it's possible that the actual read size is smaller. Read(usize, usize), } /// ReadChecker is used to check the correctness of the read process. pub struct ReadChecker { /// Raw Data is the data we write to the storage. raw_data: Bytes, } impl ReadChecker { /// Create a new read checker by given size and range. /// /// It's by design that we use a random generator to generate the raw data. The content of data /// is not important, we only care about the correctness of the read process. pub fn new(size: usize) -> Self { let mut rng = thread_rng(); let mut data = vec![0; size]; rng.fill_bytes(&mut data); let raw_data = Bytes::from(data); Self { raw_data } } /// Return the raw data of this read checker. pub fn data(&self) -> Bytes { self.raw_data.clone() } /// check_read checks the correctness of the read process after a read action. /// /// - buf_size is the read action's buf size. /// - output is the output of this read action. fn check_read(&self, offset: usize, size: usize, output: &[u8]) { if size == 0 { assert_eq!( output.len(), 0, "check read failed: output must be empty if buf_size is 0" ); return; } if size > 0 && output.is_empty() { assert!( offset >= self.raw_data.len(), "check read failed: no data read means cur must outsides of ranged_data", ); return; } assert!( offset + output.len() <= self.raw_data.len(), "check read failed: cur + output length must be less than ranged_data length, offset: {}, output: {}, ranged_data: {}", offset, output.len(), self.raw_data.len(), ); let expected = &self.raw_data[offset..offset + output.len()]; // Check the read result assert_eq!( format!("{:x}", Sha256::digest(output)), format!("{:x}", Sha256::digest(expected)), "check read failed: output bs is different with expected bs", ); } /// Check will check the correctness of the read process via given actions. /// /// Check will panic if any check failed. pub async fn check(&mut self, r: Reader, actions: &[ReadAction]) { for action in actions { match *action { ReadAction::Read(offset, size) => { let bs = r .read(offset as u64..(offset + size) as u64) .await .expect("read must success"); self.check_read(offset, size, bs.to_bytes().as_ref()); } } } } /// Check will check the correctness of the read process via given actions. /// /// Check will panic if any check failed. pub fn blocking_check(&mut self, r: BlockingReader, actions: &[ReadAction]) { for action in actions { match *action { ReadAction::Read(offset, size) => { let bs = r .read(offset as u64..(offset + size) as u64) .expect("read must success"); self.check_read(offset, size, bs.to_bytes().as_ref()); } } } } } opendal-0.52.0/src/raw/tests/utils.rs000064400000000000000000000057201046102023000155660ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::env; use std::str::FromStr; use once_cell::sync::Lazy; use crate::*; /// TEST_RUNTIME is the runtime used for running tests. pub static TEST_RUNTIME: Lazy = Lazy::new(|| { tokio::runtime::Builder::new_multi_thread() .enable_all() .build() .unwrap() }); /// Init a service with given scheme. /// /// - Load scheme from `OPENDAL_TEST` /// - Construct a new Operator with given root. /// - Else, returns a `None` to represent no valid config for operator. pub fn init_test_service() -> Result> { let _ = dotenvy::dotenv(); let scheme = if let Ok(v) = env::var("OPENDAL_TEST") { v } else { return Ok(None); }; let scheme = Scheme::from_str(&scheme).unwrap(); let prefix = format!("opendal_{scheme}_"); let mut cfg = env::vars() .filter_map(|(k, v)| { k.to_lowercase() .strip_prefix(&prefix) .map(|k| (k.to_string(), v)) }) .collect::>(); // Use random root unless OPENDAL_DISABLE_RANDOM_ROOT is set to true. let disable_random_root = env::var("OPENDAL_DISABLE_RANDOM_ROOT").unwrap_or_default() == "true"; if !disable_random_root { let root = format!( "{}{}/", cfg.get("root").cloned().unwrap_or_else(|| "/".to_string()), uuid::Uuid::new_v4() ); cfg.insert("root".to_string(), root); } let op = Operator::via_iter(scheme, cfg).expect("must succeed"); #[cfg(feature = "layers-chaos")] let op = { op.layer(layers::ChaosLayer::new(0.1)) }; let mut op = op .layer(layers::LoggingLayer::default()) .layer(layers::TimeoutLayer::new()) .layer(layers::RetryLayer::new().with_max_times(4)); // Enable blocking layer if needed. if !op.info().full_capability().blocking { // Don't enable blocking layer for compfs if op.info().scheme() != Scheme::Compfs { let _guard = TEST_RUNTIME.enter(); op = op.layer(layers::BlockingLayer::create().expect("blocking layer must be created")); } } Ok(Some(op)) } opendal-0.52.0/src/raw/tests/write.rs000064400000000000000000000044671046102023000155670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Bytes; use bytes::BytesMut; use rand::thread_rng; use rand::RngCore; use sha2::Digest; use sha2::Sha256; /// WriteAction represents a read action. #[derive(Debug, Clone, Eq, PartialEq)] pub enum WriteAction { /// Write represents a write action with given input buf size. /// /// # NOTE /// /// The size is the input buf size, it's possible that the actual write size is smaller. Write(usize), } /// WriteAction is used to check the correctness of the write process. pub struct WriteChecker { chunks: Vec, data: Bytes, } impl WriteChecker { /// Create a new WriteChecker with given size. pub fn new(size: Vec) -> Self { let mut rng = thread_rng(); let mut chunks = Vec::with_capacity(size.len()); for i in size { let mut bs = vec![0u8; i]; rng.fill_bytes(&mut bs); chunks.push(Bytes::from(bs)); } let data = chunks.iter().fold(BytesMut::new(), |mut acc, x| { acc.extend_from_slice(x); acc }); WriteChecker { chunks, data: data.freeze(), } } /// Get the check's chunks. pub fn chunks(&self) -> &[Bytes] { &self.chunks } /// Check the correctness of the write process. pub fn check(&self, actual: &[u8]) { assert_eq!( format!("{:x}", Sha256::digest(actual)), format!("{:x}", Sha256::digest(&self.data)), "check failed: result is not expected" ) } } opendal-0.52.0/src/raw/tokio_util.rs000064400000000000000000000020071046102023000154410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::Error; use crate::ErrorKind; /// Parse tokio error into opendal::Error. pub fn new_task_join_error(e: tokio::task::JoinError) -> Error { Error::new(ErrorKind::Unexpected, "tokio task join failed").set_source(e) } opendal-0.52.0/src/raw/version.rs000064400000000000000000000016071046102023000147510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. /// VERSION is the compiled version of OpenDAL. pub const VERSION: &str = env!("CARGO_PKG_VERSION"); opendal-0.52.0/src/services/aliyun_drive/backend.rs000064400000000000000000000351571046102023000204060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use chrono::Utc; use http::header; use http::Request; use http::Response; use http::StatusCode; use log::debug; use tokio::sync::Mutex; use super::core::*; use super::delete::AliyunDriveDeleter; use super::error::parse_error; use super::lister::AliyunDriveLister; use super::lister::AliyunDriveParent; use super::writer::AliyunDriveWriter; use crate::raw::*; use crate::services::AliyunDriveConfig; use crate::*; impl Configurator for AliyunDriveConfig { type Builder = AliyunDriveBuilder; fn into_builder(self) -> Self::Builder { AliyunDriveBuilder { config: self, http_client: None, } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct AliyunDriveBuilder { config: AliyunDriveConfig, http_client: Option, } impl Debug for AliyunDriveBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("AliyunDriveBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl AliyunDriveBuilder { /// Set the root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set access_token of this backend. pub fn access_token(mut self, access_token: &str) -> Self { self.config.access_token = Some(access_token.to_string()); self } /// Set client_id of this backend. pub fn client_id(mut self, client_id: &str) -> Self { self.config.client_id = Some(client_id.to_string()); self } /// Set client_secret of this backend. pub fn client_secret(mut self, client_secret: &str) -> Self { self.config.client_secret = Some(client_secret.to_string()); self } /// Set refresh_token of this backend. pub fn refresh_token(mut self, refresh_token: &str) -> Self { self.config.refresh_token = Some(refresh_token.to_string()); self } /// Set drive_type of this backend. pub fn drive_type(mut self, drive_type: &str) -> Self { self.config.drive_type = drive_type.to_string(); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for AliyunDriveBuilder { const SCHEME: Scheme = Scheme::AliyunDrive; type Config = AliyunDriveConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::AliyunDrive) })? }; let sign = match self.config.access_token.clone() { Some(access_token) if !access_token.is_empty() => { AliyunDriveSign::Access(access_token) } _ => match ( self.config.client_id.clone(), self.config.client_secret.clone(), self.config.refresh_token.clone(), ) { (Some(client_id), Some(client_secret), Some(refresh_token)) if !client_id.is_empty() && !client_secret.is_empty() && !refresh_token.is_empty() => { AliyunDriveSign::Refresh(client_id, client_secret, refresh_token, None, 0) } _ => return Err(Error::new( ErrorKind::ConfigInvalid, "access_token and a set of client_id, client_secret, and refresh_token are both missing.") .with_operation("Builder::build") .with_context("service", Scheme::AliyunDrive)), }, }; let drive_type = match self.config.drive_type.as_str() { "" | "default" => DriveType::Default, "resource" => DriveType::Resource, "backup" => DriveType::Backup, _ => { return Err(Error::new( ErrorKind::ConfigInvalid, "drive_type is invalid.", )) } }; debug!("backend use drive_type {:?}", drive_type); Ok(AliyunDriveBackend { core: Arc::new(AliyunDriveCore { endpoint: "https://openapi.alipan.com".to_string(), root, drive_type, signer: Arc::new(Mutex::new(AliyunDriveSigner { drive_id: None, sign, })), client, dir_lock: Arc::new(Mutex::new(())), }), }) } } #[derive(Clone, Debug)] pub struct AliyunDriveBackend { core: Arc, } impl Access for AliyunDriveBackend { type Reader = HttpBody; type Writer = AliyunDriveWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::AliyunDrive) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, create_dir: true, read: true, write: true, write_can_multi: true, // The min multipart size of AliyunDrive is 100 KiB. write_multi_min_size: Some(100 * 1024), // The max multipart size of AliyunDrive is 5 GiB. write_multi_max_size: if cfg!(target_pointer_width = "64") { Some(5 * 1024 * 1024 * 1024) } else { Some(usize::MAX) }, delete: true, copy: true, rename: true, list: true, list_with_limit: true, shared: true, stat_has_content_length: true, stat_has_content_type: true, list_has_last_modified: true, list_has_content_length: true, list_has_content_type: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _args: OpCreateDir) -> Result { self.core.ensure_dir_exists(path).await?; Ok(RpCreateDir::default()) } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { if from == to { return Ok(RpRename::default()); } let res = self.core.get_by_path(from).await?; let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; // rename can overwrite. match self.core.get_by_path(to).await { Err(err) if err.kind() == ErrorKind::NotFound => {} Err(err) => return Err(err), Ok(res) => { let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; self.core.delete_path(&file.file_id).await?; } }; let parent_file_id = self.core.ensure_dir_exists(get_parent(to)).await?; self.core.move_path(&file.file_id, &parent_file_id).await?; let from_name = get_basename(from); let to_name = get_basename(to); if from_name != to_name { self.core.update_path(&file.file_id, to_name).await?; } Ok(RpRename::default()) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { if from == to { return Ok(RpCopy::default()); } let res = self.core.get_by_path(from).await?; let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; // copy can overwrite. match self.core.get_by_path(to).await { Err(err) if err.kind() == ErrorKind::NotFound => {} Err(err) => return Err(err), Ok(res) => { let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; self.core.delete_path(&file.file_id).await?; } }; // there is no direct copy in AliyunDrive. // so we need to copy the path first and then rename it. let parent_path = get_parent(to); let parent_file_id = self.core.ensure_dir_exists(parent_path).await?; // if from and to are going to be placed in the same folder, // copy_path will fail as we cannot change the name during this action. // it has to be auto renamed. let auto_rename = file.parent_file_id == parent_file_id; let res = self .core .copy_path(&file.file_id, &parent_file_id, auto_rename) .await?; let file: CopyResponse = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; let file_id = file.file_id; let from_name = get_basename(from); let to_name = get_basename(to); if from_name != to_name { self.core.update_path(&file_id, to_name).await?; } Ok(RpCopy::default()) } async fn stat(&self, path: &str, _args: OpStat) -> Result { let res = self.core.get_by_path(path).await?; let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; if file.path_type == "folder" { let meta = Metadata::new(EntryMode::DIR).with_last_modified( file.updated_at .parse::>() .map_err(|e| { Error::new(ErrorKind::Unexpected, "parse last modified time").set_source(e) })?, ); return Ok(RpStat::new(meta)); } let mut meta = Metadata::new(EntryMode::FILE).with_last_modified( file.updated_at .parse::>() .map_err(|e| { Error::new(ErrorKind::Unexpected, "parse last modified time").set_source(e) })?, ); if let Some(v) = file.size { meta = meta.with_content_length(v); } if let Some(v) = file.content_type { meta = meta.with_content_type(v); } Ok(RpStat::new(meta)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let res = self.core.get_by_path(path).await?; let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; let download_url = self.core.get_download_url(&file.file_id).await?; let req = Request::get(&download_url) .header(header::RANGE, args.range().to_header()) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.core.client.fetch(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(AliyunDriveDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let parent = match self.core.get_by_path(path).await { Err(err) if err.kind() == ErrorKind::NotFound => None, Err(err) => return Err(err), Ok(res) => { let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; Some(AliyunDriveParent { file_id: file.file_id, path: path.to_string(), updated_at: file.updated_at, }) } }; let l = AliyunDriveLister::new(self.core.clone(), parent, args.limit()); Ok((RpList::default(), oio::PageLister::new(l))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let parent_path = get_parent(path); let parent_file_id = self.core.ensure_dir_exists(parent_path).await?; // write can overwrite match self.core.get_by_path(path).await { Err(err) if err.kind() == ErrorKind::NotFound => {} Err(err) => return Err(err), Ok(res) => { let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; self.core.delete_path(&file.file_id).await?; } }; let writer = AliyunDriveWriter::new(self.core.clone(), &parent_file_id, get_basename(path), args); Ok((RpWrite::default(), writer)) } } opendal-0.52.0/src/services/aliyun_drive/config.rs000064400000000000000000000046261046102023000202610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Aliyun Drive services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct AliyunDriveConfig { /// The Root of this backend. /// /// All operations will happen under this root. /// /// Default to `/` if not set. pub root: Option, /// The access_token of this backend. /// /// Solution for client-only purpose. #4733 /// /// Required if no client_id, client_secret and refresh_token are provided. pub access_token: Option, /// The client_id of this backend. /// /// Required if no access_token is provided. pub client_id: Option, /// The client_secret of this backend. /// /// Required if no access_token is provided. pub client_secret: Option, /// The refresh_token of this backend. /// /// Required if no access_token is provided. pub refresh_token: Option, /// The drive_type of this backend. /// /// All operations will happen under this type of drive. /// /// Available values are `default`, `backup` and `resource`. /// /// Fallback to default if not set or no other drives can be found. pub drive_type: String, } impl Debug for AliyunDriveConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("AliyunDriveConfig"); d.field("root", &self.root) .field("drive_type", &self.drive_type); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/aliyun_drive/core.rs000064400000000000000000000452631046102023000177460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::sync::Arc; use bytes::Buf; use chrono::Utc; use http::header::HeaderValue; use http::header::{self}; use http::Method; use http::Request; use serde::Deserialize; use serde::Serialize; use tokio::sync::Mutex; use super::error::parse_error; use crate::raw::*; use crate::*; /// Available Aliyun Drive Type. #[derive(Debug, Deserialize, Default, Clone)] pub enum DriveType { /// Use the default type of Aliyun Drive. #[default] Default, /// Use the backup type of Aliyun Drive. /// /// Fallback to the default type if no backup drive is found. Backup, /// Use the resource type of Aliyun Drive. /// /// Fallback to the default type if no resource drive is found. Resource, } /// Available Aliyun Drive Signer Set pub enum AliyunDriveSign { Refresh(String, String, String, Option, i64), Access(String), } pub struct AliyunDriveSigner { pub drive_id: Option, pub sign: AliyunDriveSign, } pub struct AliyunDriveCore { pub endpoint: String, pub root: String, pub drive_type: DriveType, pub signer: Arc>, pub client: HttpClient, pub dir_lock: Arc>, } impl Debug for AliyunDriveCore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("AliyunDriveCore") .field("root", &self.root) .field("drive_type", &self.drive_type) .finish_non_exhaustive() } } impl AliyunDriveCore { async fn send(&self, mut req: Request, token: Option<&str>) -> Result { // AliyunDrive raise NullPointerException if you haven't set a user-agent. req.headers_mut().insert( header::USER_AGENT, HeaderValue::from_str(&format!("opendal/{}", VERSION)) .expect("user agent must be valid header value"), ); if req.method() == Method::POST { req.headers_mut().insert( header::CONTENT_TYPE, HeaderValue::from_static("application/json;charset=UTF-8"), ); } if let Some(token) = token { req.headers_mut().insert( header::AUTHORIZATION, HeaderValue::from_str(&format_authorization_by_bearer(token)?) .expect("access token must be valid header value"), ); } let res = self.client.send(req).await?; if !res.status().is_success() { return Err(parse_error(res)); } Ok(res.into_body()) } async fn get_access_token( &self, client_id: &str, client_secret: &str, refresh_token: &str, ) -> Result { let body = serde_json::to_vec(&AccessTokenRequest { refresh_token, grant_type: "refresh_token", client_id, client_secret, }) .map_err(new_json_serialize_error)?; let req = Request::post(format!("{}/oauth/access_token", self.endpoint)) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, None).await } async fn get_drive_id(&self, token: Option<&str>) -> Result { let req = Request::post(format!("{}/adrive/v1.0/user/getDriveInfo", self.endpoint)) .body(Buffer::new()) .map_err(new_request_build_error)?; self.send(req, token).await } pub async fn get_token_and_drive(&self) -> Result<(Option, String)> { let mut signer = self.signer.lock().await; let token = match &mut signer.sign { AliyunDriveSign::Access(access_token) => Some(access_token.clone()), AliyunDriveSign::Refresh( client_id, client_secret, refresh_token, access_token, expire_at, ) => { if *expire_at < Utc::now().timestamp() || access_token.is_none() { let res = self .get_access_token(client_id, client_secret, refresh_token) .await?; let output: RefreshTokenResponse = serde_json::from_reader(res.reader()) .map_err(new_json_deserialize_error)?; *access_token = Some(output.access_token); *expire_at = output.expires_in + Utc::now().timestamp(); *refresh_token = output.refresh_token; } access_token.clone() } }; let Some(drive_id) = &signer.drive_id else { let res = self.get_drive_id(token.as_deref()).await?; let output: DriveInfoResponse = serde_json::from_reader(res.reader()).map_err(new_json_deserialize_error)?; let drive_id = match self.drive_type { DriveType::Default => output.default_drive_id, DriveType::Backup => output.backup_drive_id.unwrap_or(output.default_drive_id), DriveType::Resource => output.resource_drive_id.unwrap_or(output.default_drive_id), }; signer.drive_id = Some(drive_id.clone()); return Ok((token, drive_id)); }; Ok((token, drive_id.clone())) } pub fn build_path(&self, path: &str, rooted: bool) -> String { let file_path = if rooted { build_rooted_abs_path(&self.root, path) } else { build_abs_path(&self.root, path) }; let file_path = file_path.strip_suffix('/').unwrap_or(&file_path); if file_path.is_empty() { return "/".to_string(); } file_path.to_string() } pub async fn get_by_path(&self, path: &str) -> Result { let file_path = self.build_path(path, true); let req = Request::post(format!( "{}/adrive/v1.0/openFile/get_by_path", self.endpoint )); let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&GetByPathRequest { drive_id: &drive_id, file_path: &file_path, }) .map_err(new_json_serialize_error)?; let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await } pub async fn ensure_dir_exists(&self, path: &str) -> Result { let file_path = self.build_path(path, false); if file_path == "/" { return Ok("root".to_string()); } let file_path = file_path.strip_suffix('/').unwrap_or(&file_path); let paths = file_path.split('/').collect::>(); let mut parent: Option = None; for path in paths { let _guard = self.dir_lock.lock().await; let res = self .create( parent.as_deref(), path, CreateType::Folder, CheckNameMode::Refuse, ) .await?; let output: CreateResponse = serde_json::from_reader(res.reader()).map_err(new_json_deserialize_error)?; parent = Some(output.file_id); } Ok(parent.expect("ensure_dir_exists must succeed")) } pub async fn create_with_rapid_upload( &self, parent_file_id: Option<&str>, name: &str, typ: CreateType, check_name_mode: CheckNameMode, size: Option, rapid_upload: Option, ) -> Result { let mut content_hash = None; let mut proof_code = None; let mut pre_hash = None; if let Some(rapid_upload) = rapid_upload { content_hash = rapid_upload.content_hash; proof_code = rapid_upload.proof_code; pre_hash = rapid_upload.pre_hash; } let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&CreateRequest { drive_id: &drive_id, parent_file_id: parent_file_id.unwrap_or("root"), name, typ, check_name_mode, size, pre_hash: pre_hash.as_deref(), content_hash: content_hash.as_deref(), content_hash_name: content_hash.is_some().then_some("sha1"), proof_code: proof_code.as_deref(), proof_version: proof_code.is_some().then_some("v1"), }) .map_err(new_json_serialize_error)?; let req = Request::post(format!("{}/adrive/v1.0/openFile/create", self.endpoint)) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await } pub async fn create( &self, parent_file_id: Option<&str>, name: &str, typ: CreateType, check_name_mode: CheckNameMode, ) -> Result { self.create_with_rapid_upload(parent_file_id, name, typ, check_name_mode, None, None) .await } pub async fn get_download_url(&self, file_id: &str) -> Result { let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&FileRequest { drive_id: &drive_id, file_id, }) .map_err(new_json_serialize_error)?; let req = Request::post(format!( "{}/adrive/v1.0/openFile/getDownloadUrl", self.endpoint )) .body(Buffer::from(body)) .map_err(new_request_build_error)?; let res = self.send(req, token.as_deref()).await?; let output: GetDownloadUrlResponse = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; Ok(output.url) } pub async fn move_path(&self, file_id: &str, to_parent_file_id: &str) -> Result<()> { let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&MovePathRequest { drive_id: &drive_id, file_id, to_parent_file_id, check_name_mode: CheckNameMode::AutoRename, }) .map_err(new_json_serialize_error)?; let req = Request::post(format!("{}/adrive/v1.0/openFile/move", self.endpoint)) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await?; Ok(()) } pub async fn update_path(&self, file_id: &str, name: &str) -> Result<()> { let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&UpdatePathRequest { drive_id: &drive_id, file_id, name, check_name_mode: CheckNameMode::Refuse, }) .map_err(new_json_serialize_error)?; let req = Request::post(format!("{}/adrive/v1.0/openFile/update", self.endpoint)) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await?; Ok(()) } pub async fn copy_path( &self, file_id: &str, to_parent_file_id: &str, auto_rename: bool, ) -> Result { let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&CopyPathRequest { drive_id: &drive_id, file_id, to_parent_file_id, auto_rename, }) .map_err(new_json_serialize_error)?; let req = Request::post(format!("{}/adrive/v1.0/openFile/copy", self.endpoint)) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await } pub async fn delete_path(&self, file_id: &str) -> Result<()> { let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&FileRequest { drive_id: &drive_id, file_id, }) .map_err(new_json_serialize_error)?; let req = Request::post(format!("{}/adrive/v1.0/openFile/delete", self.endpoint)) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await?; Ok(()) } pub async fn list( &self, parent_file_id: &str, limit: Option, marker: Option, ) -> Result { let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&ListRequest { drive_id: &drive_id, parent_file_id, limit, marker: marker.as_deref(), }) .map_err(new_json_serialize_error)?; let req = Request::post(format!("{}/adrive/v1.0/openFile/list", self.endpoint)) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await } pub async fn upload(&self, upload_url: &str, body: Buffer) -> Result { let req = Request::put(upload_url) .body(body) .map_err(new_request_build_error)?; self.send(req, None).await } pub async fn complete(&self, file_id: &str, upload_id: &str) -> Result { let (token, drive_id) = self.get_token_and_drive().await?; let body = serde_json::to_vec(&CompleteRequest { drive_id: &drive_id, file_id, upload_id, }) .map_err(new_json_serialize_error)?; let req = Request::post(format!("{}/adrive/v1.0/openFile/complete", self.endpoint)) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await } pub async fn get_upload_url( &self, file_id: &str, upload_id: &str, part_number: Option, ) -> Result { let (token, drive_id) = self.get_token_and_drive().await?; let part_info_list = part_number.map(|part_number| { vec![PartInfoItem { part_number: Some(part_number), }] }); let body = serde_json::to_vec(&GetUploadRequest { drive_id: &drive_id, file_id, upload_id, part_info_list, }) .map_err(new_json_serialize_error)?; let req = Request::post(format!( "{}/adrive/v1.0/openFile/getUploadUrl", self.endpoint )) .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req, token.as_deref()).await } } pub struct RapidUpload { pub pre_hash: Option, pub content_hash: Option, pub proof_code: Option, } #[derive(Debug, Deserialize)] pub struct RefreshTokenResponse { pub access_token: String, pub expires_in: i64, pub refresh_token: String, } #[derive(Debug, Deserialize)] pub struct DriveInfoResponse { pub default_drive_id: String, pub resource_drive_id: Option, pub backup_drive_id: Option, } #[derive(Debug, Serialize)] #[serde(rename_all = "snake_case")] pub enum CreateType { File, Folder, } #[derive(Debug, Serialize)] #[serde(rename_all = "snake_case")] pub enum CheckNameMode { Refuse, AutoRename, } #[derive(Deserialize)] pub struct UploadUrlResponse { pub part_info_list: Option>, } #[derive(Deserialize)] pub struct CreateResponse { pub file_id: String, pub upload_id: Option, pub exist: Option, } #[derive(Serialize, Deserialize)] pub struct PartInfo { pub etag: Option, pub part_number: usize, pub part_size: Option, pub upload_url: String, pub content_type: Option, } #[derive(Deserialize)] pub struct AliyunDriveFileList { pub items: Vec, pub next_marker: Option, } #[derive(Deserialize)] pub struct CopyResponse { pub file_id: String, } #[derive(Deserialize)] pub struct AliyunDriveFile { pub file_id: String, pub parent_file_id: String, pub name: String, pub size: Option, pub content_type: Option, #[serde(rename = "type")] pub path_type: String, pub updated_at: String, } #[derive(Deserialize)] pub struct GetDownloadUrlResponse { pub url: String, } #[derive(Serialize)] pub struct AccessTokenRequest<'a> { refresh_token: &'a str, grant_type: &'a str, client_id: &'a str, client_secret: &'a str, } #[derive(Serialize)] pub struct GetByPathRequest<'a> { drive_id: &'a str, file_path: &'a str, } #[derive(Serialize)] pub struct CreateRequest<'a> { drive_id: &'a str, parent_file_id: &'a str, name: &'a str, #[serde(rename = "type")] typ: CreateType, check_name_mode: CheckNameMode, size: Option, pre_hash: Option<&'a str>, content_hash: Option<&'a str>, content_hash_name: Option<&'a str>, proof_code: Option<&'a str>, proof_version: Option<&'a str>, } #[derive(Serialize)] pub struct FileRequest<'a> { drive_id: &'a str, file_id: &'a str, } #[derive(Serialize)] pub struct MovePathRequest<'a> { drive_id: &'a str, file_id: &'a str, to_parent_file_id: &'a str, check_name_mode: CheckNameMode, } #[derive(Serialize)] pub struct UpdatePathRequest<'a> { drive_id: &'a str, file_id: &'a str, name: &'a str, check_name_mode: CheckNameMode, } #[derive(Serialize)] pub struct CopyPathRequest<'a> { drive_id: &'a str, file_id: &'a str, to_parent_file_id: &'a str, auto_rename: bool, } #[derive(Serialize)] pub struct ListRequest<'a> { drive_id: &'a str, parent_file_id: &'a str, limit: Option, marker: Option<&'a str>, } #[derive(Serialize)] pub struct CompleteRequest<'a> { drive_id: &'a str, file_id: &'a str, upload_id: &'a str, } #[derive(Serialize)] pub struct GetUploadRequest<'a> { drive_id: &'a str, file_id: &'a str, upload_id: &'a str, part_info_list: Option>, } #[derive(Serialize)] pub struct PartInfoItem { part_number: Option, } opendal-0.52.0/src/services/aliyun_drive/delete.rs000064400000000000000000000032611046102023000202500ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::{AliyunDriveCore, AliyunDriveFile}; use crate::raw::*; use crate::*; use bytes::Buf; use std::sync::Arc; pub struct AliyunDriveDeleter { core: Arc, } impl AliyunDriveDeleter { pub fn new(core: Arc) -> Self { AliyunDriveDeleter { core } } } impl oio::OneShotDelete for AliyunDriveDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let res = match self.core.get_by_path(&path).await { Ok(output) => Some(output), Err(err) if err.kind() == ErrorKind::NotFound => None, Err(err) => return Err(err), }; if let Some(res) = res { let file: AliyunDriveFile = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; self.core.delete_path(&file.file_id).await?; } Ok(()) } } opendal-0.52.0/src/services/aliyun_drive/docs.md000064400000000000000000000027561046102023000177220ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work dir for backend. - `access_token`: Set the access_token for backend. - `client_id`: Set the client_id for backend. - `client_secret`: Set the client_secret for backend. - `refresh_token`: Set the refresh_token for backend. - `drive_type`: Set the drive_type for backend. Refer to [`AliyunDriveBuilder`]`s public API docs for more information. ## Example ### Basic Setup ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::AliyunDrive; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create aliyun-drive backend builder. let mut builder = AliyunDrive::default() // Set the root for aliyun-drive, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/path/to/dir") // Set the client_id. This is required. .client_id("client_id") // Set the client_secret. This is required. .client_secret("client_secret") // Set the refresh_token. This is required. .refresh_token("refresh_token") // Set the drive_type. This is required. // // Fallback to the default type if no other types found. .drive_type("resource"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/aliyun_drive/error.rs000064400000000000000000000042101046102023000201320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use serde::Deserialize; use crate::*; #[derive(Default, Debug, Deserialize)] struct AliyunDriveError { code: String, message: String, } pub(super) fn parse_error(res: Response) -> Error { let (parts, body) = res.into_parts(); let bs = body.to_bytes(); let (code, message) = serde_json::from_reader::<_, AliyunDriveError>(bs.clone().reader()) .map(|err| (Some(err.code), err.message)) .unwrap_or((None, String::from_utf8_lossy(&bs).into_owned())); let (kind, retryable) = match parts.status.as_u16() { 403 => (ErrorKind::PermissionDenied, false), 400 => match code { Some(code) if code == "NotFound.File" => (ErrorKind::NotFound, false), Some(code) if code == "AlreadyExist.File" => (ErrorKind::AlreadyExists, false), Some(code) if code == "PreHashMatched" => (ErrorKind::IsSameFile, false), _ => (ErrorKind::Unexpected, false), }, 409 => (ErrorKind::AlreadyExists, false), 429 => match code { Some(code) if code == "TooManyRequests" => (ErrorKind::RateLimited, true), _ => (ErrorKind::Unexpected, false), }, _ => (ErrorKind::Unexpected, false), }; let mut err = Error::new(kind, message); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/aliyun_drive/lister.rs000064400000000000000000000101121046102023000203010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use chrono::Utc; use self::oio::Entry; use super::core::AliyunDriveCore; use super::core::AliyunDriveFileList; use crate::raw::*; use crate::EntryMode; use crate::Error; use crate::ErrorKind; use crate::Metadata; use crate::Result; pub struct AliyunDriveLister { core: Arc, parent: Option, limit: Option, } pub struct AliyunDriveParent { pub file_id: String, pub path: String, pub updated_at: String, } impl AliyunDriveLister { pub fn new( core: Arc, parent: Option, limit: Option, ) -> Self { AliyunDriveLister { core, parent, limit, } } } impl oio::PageList for AliyunDriveLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let Some(parent) = &self.parent else { ctx.done = true; return Ok(()); }; let offset = if ctx.token.is_empty() { // Push self into the list result. ctx.entries.push_back(Entry::new( &parent.path, Metadata::new(EntryMode::DIR).with_last_modified( parent .updated_at .parse::>() .map_err(|e| { Error::new(ErrorKind::Unexpected, "parse last modified time") .set_source(e) })?, ), )); None } else { Some(ctx.token.clone()) }; let res = self.core.list(&parent.file_id, self.limit, offset).await; let res = match res { Err(err) if err.kind() == ErrorKind::NotFound => { ctx.done = true; None } Err(err) => return Err(err), Ok(res) => Some(res), }; let Some(res) = res else { return Ok(()); }; let result: AliyunDriveFileList = serde_json::from_reader(res.reader()).map_err(new_json_serialize_error)?; for item in result.items { let (path, mut md) = if item.path_type == "folder" { let path = format!("{}{}/", &parent.path.trim_start_matches('/'), &item.name); (path, Metadata::new(EntryMode::DIR)) } else { let path = format!("{}{}", &parent.path.trim_start_matches('/'), &item.name); (path, Metadata::new(EntryMode::FILE)) }; md = md.with_last_modified(item.updated_at.parse::>().map_err( |e| Error::new(ErrorKind::Unexpected, "parse last modified time").set_source(e), )?); if let Some(v) = item.size { md = md.with_content_length(v); } if let Some(v) = item.content_type { md = md.with_content_type(v); } ctx.entries.push_back(Entry::new(&path, md)); } let next_marker = result.next_marker.unwrap_or_default(); if next_marker.is_empty() { ctx.done = true; } else { ctx.token = next_marker; } Ok(()) } } opendal-0.52.0/src/services/aliyun_drive/mod.rs000064400000000000000000000023641046102023000175700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-aliyun-drive")] mod core; #[cfg(feature = "services-aliyun-drive")] mod backend; #[cfg(feature = "services-aliyun-drive")] mod delete; #[cfg(feature = "services-aliyun-drive")] mod error; #[cfg(feature = "services-aliyun-drive")] mod lister; #[cfg(feature = "services-aliyun-drive")] mod writer; #[cfg(feature = "services-aliyun-drive")] pub use backend::AliyunDriveBuilder as AliyunDrive; mod config; pub use config::AliyunDriveConfig; opendal-0.52.0/src/services/aliyun_drive/writer.rs000064400000000000000000000101111046102023000203120ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use super::core::AliyunDriveCore; use super::core::CheckNameMode; use super::core::CreateResponse; use super::core::CreateType; use super::core::UploadUrlResponse; use crate::raw::*; use crate::*; pub struct AliyunDriveWriter { core: Arc, _op: OpWrite, parent_file_id: String, name: String, file_id: Option, upload_id: Option, part_number: usize, } impl AliyunDriveWriter { pub fn new(core: Arc, parent_file_id: &str, name: &str, op: OpWrite) -> Self { AliyunDriveWriter { core, _op: op, parent_file_id: parent_file_id.to_string(), name: name.to_string(), file_id: None, upload_id: None, part_number: 1, // must start from 1 } } } impl oio::Write for AliyunDriveWriter { async fn write(&mut self, bs: Buffer) -> Result<()> { let (upload_id, file_id) = match (self.upload_id.as_ref(), self.file_id.as_ref()) { (Some(upload_id), Some(file_id)) => (upload_id, file_id), _ => { let res = self .core .create( Some(&self.parent_file_id), &self.name, CreateType::File, CheckNameMode::Refuse, ) .await?; let output: CreateResponse = serde_json::from_reader(res.reader()).map_err(new_json_deserialize_error)?; if output.exist.is_some_and(|x| x) { return Err(Error::new(ErrorKind::AlreadyExists, "file exists")); } self.upload_id = output.upload_id; self.file_id = Some(output.file_id); ( self.upload_id.as_ref().expect("cannot find upload_id"), self.file_id.as_ref().expect("cannot find file_id"), ) } }; let res = self .core .get_upload_url(file_id, upload_id, Some(self.part_number)) .await?; let output: UploadUrlResponse = serde_json::from_reader(res.reader()).map_err(new_json_deserialize_error)?; let Some(upload_url) = output .part_info_list .as_ref() .and_then(|list| list.first()) .map(|part_info| &part_info.upload_url) else { return Err(Error::new(ErrorKind::Unexpected, "cannot find upload_url")); }; if let Err(err) = self.core.upload(upload_url, bs).await { if err.kind() != ErrorKind::AlreadyExists { return Err(err); } }; self.part_number += 1; Ok(()) } async fn close(&mut self) -> Result { let (Some(upload_id), Some(file_id)) = (self.upload_id.as_ref(), self.file_id.as_ref()) else { return Ok(Metadata::default()); }; self.core.complete(file_id, upload_id).await?; Ok(Metadata::default()) } async fn abort(&mut self) -> Result<()> { let Some(file_id) = self.file_id.as_ref() else { return Ok(()); }; self.core.delete_path(file_id).await } } opendal-0.52.0/src/services/alluxio/backend.rs000064400000000000000000000173111046102023000173610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use log::debug; use super::core::AlluxioCore; use super::delete::AlluxioDeleter; use super::error::parse_error; use super::lister::AlluxioLister; use super::writer::AlluxioWriter; use super::writer::AlluxioWriters; use crate::raw::*; use crate::services::AlluxioConfig; use crate::*; impl Configurator for AlluxioConfig { type Builder = AlluxioBuilder; fn into_builder(self) -> Self::Builder { AlluxioBuilder { config: self, http_client: None, } } } /// [Alluxio](https://www.alluxio.io/) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct AlluxioBuilder { config: AlluxioConfig, http_client: Option, } impl Debug for AlluxioBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("AlluxioBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl AlluxioBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// endpoint of this backend. /// /// Endpoint must be full uri, mostly like `http://127.0.0.1:39999`. pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:39999/` self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()) } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for AlluxioBuilder { const SCHEME: Scheme = Scheme::Alluxio; type Config = AlluxioConfig; /// Builds the backend and returns the result of AlluxioBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); let endpoint = match &self.config.endpoint { Some(endpoint) => Ok(endpoint.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Alluxio)), }?; debug!("backend use endpoint {}", &endpoint); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Alluxio) })? }; Ok(AlluxioBackend { core: Arc::new(AlluxioCore { root, endpoint, client, }), }) } } #[derive(Debug, Clone)] pub struct AlluxioBackend { core: Arc, } impl Access for AlluxioBackend { type Reader = HttpBody; type Writer = AlluxioWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Alluxio) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, // FIXME: // // alluxio's read support is not implemented correctly // We need to refactor by use [page_read](https://github.com/Alluxio/alluxio-py/blob/main/alluxio/const.py#L18) read: false, write: true, write_can_multi: true, create_dir: true, delete: true, list: true, shared: true, stat_has_content_length: true, stat_has_last_modified: true, list_has_content_length: true, list_has_last_modified: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { self.core.create_dir(path).await?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _: OpStat) -> Result { let file_info = self.core.get_status(path).await?; Ok(RpStat::new(file_info.try_into()?)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let stream_id = self.core.open_file(path).await?; let resp = self.core.read(stream_id, args.range()).await?; if !resp.status().is_success() { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; return Err(parse_error(Response::from_parts(part, buf))); } Ok((RpRead::new(), resp.into_body())) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let w = AlluxioWriter::new(self.core.clone(), args.clone(), path.to_string()); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(AlluxioDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, _args: OpList) -> Result<(RpList, Self::Lister)> { let l = AlluxioLister::new(self.core.clone(), path); Ok((RpList::default(), oio::PageLister::new(l))) } async fn rename(&self, from: &str, to: &str, _: OpRename) -> Result { self.core.rename(from, to).await?; Ok(RpRename::default()) } } #[cfg(test)] mod test { use std::collections::HashMap; use super::*; #[test] fn test_builder_from_map() { let mut map = HashMap::new(); map.insert("root".to_string(), "/".to_string()); map.insert("endpoint".to_string(), "http://127.0.0.1:39999".to_string()); let builder = AlluxioConfig::from_iter(map).unwrap(); assert_eq!(builder.root, Some("/".to_string())); assert_eq!(builder.endpoint, Some("http://127.0.0.1:39999".to_string())); } #[test] fn test_builder_build() { let builder = AlluxioBuilder::default() .root("/root") .endpoint("http://127.0.0.1:39999") .build(); assert!(builder.is_ok()); } } opendal-0.52.0/src/services/alluxio/config.rs000064400000000000000000000031671046102023000172430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for alluxio services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct AlluxioConfig { /// root of this backend. /// /// All operations will happen under this root. /// /// default to `/` if not set. pub root: Option, /// endpoint of this backend. /// /// Endpoint must be full uri, mostly like `http://127.0.0.1:39999`. pub endpoint: Option, } impl Debug for AlluxioConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("AlluxioConfig"); d.field("root", &self.root) .field("endpoint", &self.endpoint); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/alluxio/core.rs000064400000000000000000000242661046102023000167310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use bytes::Buf; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use serde::Serialize; use super::error::parse_error; use crate::raw::*; use crate::*; #[derive(Debug, Serialize)] struct CreateFileRequest { #[serde(skip_serializing_if = "Option::is_none")] recursive: Option, } #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] struct CreateDirRequest { #[serde(skip_serializing_if = "Option::is_none")] recursive: Option, #[serde(skip_serializing_if = "Option::is_none")] allow_exists: Option, } /// Metadata of alluxio object #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub(super) struct FileInfo { /// The path of the object pub path: String, /// The last modification time of the object pub last_modification_time_ms: i64, /// Whether the object is a folder pub folder: bool, /// The length of the object in bytes pub length: u64, } impl TryFrom for Metadata { type Error = Error; fn try_from(file_info: FileInfo) -> Result { let mut metadata = if file_info.folder { Metadata::new(EntryMode::DIR) } else { Metadata::new(EntryMode::FILE) }; metadata .set_content_length(file_info.length) .set_last_modified(parse_datetime_from_from_timestamp_millis( file_info.last_modification_time_ms, )?); Ok(metadata) } } /// Alluxio core #[derive(Clone)] pub struct AlluxioCore { /// root of this backend. pub root: String, /// endpoint of alluxio pub endpoint: String, /// prefix of alluxio pub client: HttpClient, } impl Debug for AlluxioCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("endpoint", &self.endpoint) .finish_non_exhaustive() } } impl AlluxioCore { pub async fn create_dir(&self, path: &str) -> Result<()> { let path = build_rooted_abs_path(&self.root, path); let r = CreateDirRequest { recursive: Some(true), allow_exists: Some(true), }; let body = serde_json::to_vec(&r).map_err(new_json_serialize_error)?; let body = bytes::Bytes::from(body); let mut req = Request::post(format!( "{}/api/v1/paths/{}/create-directory", self.endpoint, percent_encode_path(&path) )); req = req.header("Content-Type", "application/json"); let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } pub async fn create_file(&self, path: &str) -> Result { let path = build_rooted_abs_path(&self.root, path); let r = CreateFileRequest { recursive: Some(true), }; let body = serde_json::to_vec(&r).map_err(new_json_serialize_error)?; let body = bytes::Bytes::from(body); let mut req = Request::post(format!( "{}/api/v1/paths/{}/create-file", self.endpoint, percent_encode_path(&path) )); req = req.header("Content-Type", "application/json"); let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let steam_id: u64 = serde_json::from_reader(body.reader()).map_err(new_json_serialize_error)?; Ok(steam_id) } _ => Err(parse_error(resp)), } } pub(super) async fn open_file(&self, path: &str) -> Result { let path = build_rooted_abs_path(&self.root, path); let req = Request::post(format!( "{}/api/v1/paths/{}/open-file", self.endpoint, percent_encode_path(&path) )); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let steam_id: u64 = serde_json::from_reader(body.reader()).map_err(new_json_serialize_error)?; Ok(steam_id) } _ => Err(parse_error(resp)), } } pub(super) async fn delete(&self, path: &str) -> Result<()> { let path = build_rooted_abs_path(&self.root, path); let req = Request::post(format!( "{}/api/v1/paths/{}/delete", self.endpoint, percent_encode_path(&path) )); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => { let err = parse_error(resp); if err.kind() == ErrorKind::NotFound { return Ok(()); } Err(err) } } } pub(super) async fn rename(&self, path: &str, dst: &str) -> Result<()> { let path = build_rooted_abs_path(&self.root, path); let dst = build_rooted_abs_path(&self.root, dst); let req = Request::post(format!( "{}/api/v1/paths/{}/rename?dst={}", self.endpoint, percent_encode_path(&path), percent_encode_path(&dst) )); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } pub(super) async fn get_status(&self, path: &str) -> Result { let path = build_rooted_abs_path(&self.root, path); let req = Request::post(format!( "{}/api/v1/paths/{}/get-status", self.endpoint, percent_encode_path(&path) )); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let file_info: FileInfo = serde_json::from_reader(body.reader()).map_err(new_json_serialize_error)?; Ok(file_info) } _ => Err(parse_error(resp)), } } pub(super) async fn list_status(&self, path: &str) -> Result> { let path = build_rooted_abs_path(&self.root, path); let req = Request::post(format!( "{}/api/v1/paths/{}/list-status", self.endpoint, percent_encode_path(&path) )); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let file_infos: Vec = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; Ok(file_infos) } _ => Err(parse_error(resp)), } } /// TODO: we should implement range support correctly. /// /// Please refer to [alluxio-py](https://github.com/Alluxio/alluxio-py/blob/main/alluxio/const.py#L18) pub async fn read(&self, stream_id: u64, _: BytesRange) -> Result> { let req = Request::post(format!( "{}/api/v1/streams/{}/read", self.endpoint, stream_id, )); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub(super) async fn write(&self, stream_id: u64, body: Buffer) -> Result { let req = Request::post(format!( "{}/api/v1/streams/{}/write", self.endpoint, stream_id )); let req = req.body(body).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let size: usize = serde_json::from_reader(body.reader()).map_err(new_json_serialize_error)?; Ok(size) } _ => Err(parse_error(resp)), } } pub(super) async fn close(&self, stream_id: u64) -> Result<()> { let req = Request::post(format!( "{}/api/v1/streams/{}/close", self.endpoint, stream_id )); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/alluxio/delete.rs000064400000000000000000000022751046102023000172370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use crate::raw::*; use crate::*; use std::sync::Arc; pub struct AlluxioDeleter { core: Arc, } impl AlluxioDeleter { pub fn new(core: Arc) -> Self { AlluxioDeleter { core } } } impl oio::OneShotDelete for AlluxioDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { self.core.delete(&path).await } } opendal-0.52.0/src/services/alluxio/docs.md000064400000000000000000000014551046102023000167000ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [x] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `endpoint`: Customizable endpoint setting You can refer to [`AlluxioBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Alluxio; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Alluxio::default() // set the storage bucket for OpenDAL .root("/") // set the endpoint for OpenDAL .endpoint("http://127.0.0.1:39999"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/alluxio/error.rs000064400000000000000000000061001046102023000171150ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use serde::Deserialize; use crate::raw::*; use crate::*; /// the error response of alluxio #[derive(Default, Debug, Deserialize)] #[serde(rename_all = "camelCase")] #[allow(dead_code)] struct AlluxioError { status_code: String, message: String, } pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let mut kind = match parts.status.as_u16() { 500 => ErrorKind::Unexpected, _ => ErrorKind::Unexpected, }; let (message, alluxio_err) = serde_json::from_reader::<_, AlluxioError>(bs.clone().reader()) .map(|alluxio_err| (format!("{alluxio_err:?}"), Some(alluxio_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); if let Some(alluxio_err) = alluxio_err { kind = match alluxio_err.status_code.as_str() { "ALREADY_EXISTS" => ErrorKind::AlreadyExists, "NOT_FOUND" => ErrorKind::NotFound, _ => ErrorKind::Unexpected, } } let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); err } #[cfg(test)] mod tests { use http::StatusCode; use super::*; /// Error response example is from https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html #[test] fn test_parse_error() { let err_res = vec![ ( r#"{"statusCode":"ALREADY_EXISTS","message":"The resource you requested already exist"}"#, ErrorKind::AlreadyExists, ), ( r#"{"statusCode":"NOT_FOUND","message":"The resource you requested does not exist"}"#, ErrorKind::NotFound, ), ( r#"{"statusCode":"INTERNAL_SERVER_ERROR","message":"Internal server error"}"#, ErrorKind::Unexpected, ), ]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder() .status(StatusCode::INTERNAL_SERVER_ERROR) .body(body) .unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/alluxio/lister.rs000064400000000000000000000042621046102023000172750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use super::core::AlluxioCore; use crate::raw::oio::Entry; use crate::raw::*; use crate::ErrorKind; use crate::Result; pub struct AlluxioLister { core: Arc, path: String, } impl AlluxioLister { pub(super) fn new(core: Arc, path: &str) -> Self { AlluxioLister { core, path: path.to_string(), } } } impl oio::PageList for AlluxioLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let result = self.core.list_status(&self.path).await; match result { Ok(file_infos) => { ctx.done = true; for file_info in file_infos { let path: String = file_info.path.clone(); let path = if file_info.folder { format!("{}/", path) } else { path }; ctx.entries.push_back(Entry::new( &build_rel_path(&self.core.root, &path), file_info.try_into()?, )); } Ok(()) } Err(e) => { if e.kind() == ErrorKind::NotFound { ctx.done = true; return Ok(()); } Err(e) } } } } opendal-0.52.0/src/services/alluxio/mod.rs000064400000000000000000000023051046102023000165460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-alluxio")] mod core; #[cfg(feature = "services-alluxio")] mod delete; #[cfg(feature = "services-alluxio")] mod error; #[cfg(feature = "services-alluxio")] mod lister; #[cfg(feature = "services-alluxio")] mod writer; #[cfg(feature = "services-alluxio")] mod backend; #[cfg(feature = "services-alluxio")] pub use backend::AlluxioBuilder as Alluxio; mod config; pub use config::AlluxioConfig; opendal-0.52.0/src/services/alluxio/writer.rs000064400000000000000000000041361046102023000173070ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use super::core::AlluxioCore; use crate::raw::*; use crate::*; pub type AlluxioWriters = AlluxioWriter; pub struct AlluxioWriter { core: Arc, _op: OpWrite, path: String, stream_id: Option, } impl AlluxioWriter { pub fn new(core: Arc, _op: OpWrite, path: String) -> Self { AlluxioWriter { core, _op, path, stream_id: None, } } } impl oio::Write for AlluxioWriter { async fn write(&mut self, bs: Buffer) -> Result<()> { let stream_id = match self.stream_id { Some(stream_id) => stream_id, None => { let stream_id = self.core.create_file(&self.path).await?; self.stream_id = Some(stream_id); stream_id } }; self.core.write(stream_id, bs).await?; Ok(()) } async fn close(&mut self) -> Result { let Some(stream_id) = self.stream_id else { return Ok(Metadata::default()); }; self.core.close(stream_id).await?; Ok(Metadata::default()) } async fn abort(&mut self) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "AlluxioWriter doesn't support abort", )) } } opendal-0.52.0/src/services/atomicserver/backend.rs000064400000000000000000000330331046102023000204060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use atomic_lib::agents::Agent; use atomic_lib::client::get_authentication_headers; use atomic_lib::commit::sign_message; use bytes::Buf; use http::header::CONTENT_DISPOSITION; use http::header::CONTENT_TYPE; use http::Request; use serde::Deserialize; use serde::Serialize; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::AtomicserverConfig; use crate::*; impl Configurator for AtomicserverConfig { type Builder = AtomicserverBuilder; fn into_builder(self) -> Self::Builder { AtomicserverBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct AtomicserverBuilder { config: AtomicserverConfig, } impl Debug for AtomicserverBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("AtomicserverBuilder") .field("config", &self.config) .finish() } } impl AtomicserverBuilder { /// Set the root for Atomicserver. pub fn root(mut self, path: &str) -> Self { self.config.root = Some(path.into()); self } /// Set the server address for Atomicserver. pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = Some(endpoint.into()); self } /// Set the private key for agent used for Atomicserver. pub fn private_key(mut self, private_key: &str) -> Self { self.config.private_key = Some(private_key.into()); self } /// Set the public key for agent used for Atomicserver. /// For example, if the subject URL for the agent being used /// is ${endpoint}/agents/lTB+W3C/2YfDu9IAVleEy34uCmb56iXXuzWCKBVwdRI= /// Then the required public key is `lTB+W3C/2YfDu9IAVleEy34uCmb56iXXuzWCKBVwdRI=` pub fn public_key(mut self, public_key: &str) -> Self { self.config.public_key = Some(public_key.into()); self } /// Set the parent resource id (url) that Atomicserver uses to store resources under. pub fn parent_resource_id(mut self, parent_resource_id: &str) -> Self { self.config.parent_resource_id = Some(parent_resource_id.into()); self } } impl Builder for AtomicserverBuilder { const SCHEME: Scheme = Scheme::Atomicserver; type Config = AtomicserverConfig; fn build(self) -> Result { let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); let endpoint = self.config.endpoint.clone().unwrap(); let parent_resource_id = self.config.parent_resource_id.clone().unwrap(); let agent = Agent { private_key: self.config.private_key.clone(), public_key: self.config.public_key.clone().unwrap(), subject: format!( "{}/agents/{}", endpoint, self.config.public_key.clone().unwrap() ), created_at: 1, name: Some("agent".to_string()), }; Ok(AtomicserverBackend::new(Adapter { parent_resource_id, endpoint, agent, client: HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Atomicserver) })?, }) .with_normalized_root(root)) } } /// Backend for Atomicserver services. pub type AtomicserverBackend = kv::Backend; const FILENAME_PROPERTY: &str = "https://atomicdata.dev/properties/filename"; #[derive(Debug, Serialize)] struct CommitStruct { #[serde(rename = "https://atomicdata.dev/properties/createdAt")] created_at: i64, #[serde(rename = "https://atomicdata.dev/properties/destroy")] destroy: bool, #[serde(rename = "https://atomicdata.dev/properties/isA")] is_a: Vec, #[serde(rename = "https://atomicdata.dev/properties/signer")] signer: String, #[serde(rename = "https://atomicdata.dev/properties/subject")] subject: String, } #[derive(Debug, Serialize)] struct CommitStructSigned { #[serde(rename = "https://atomicdata.dev/properties/createdAt")] created_at: i64, #[serde(rename = "https://atomicdata.dev/properties/destroy")] destroy: bool, #[serde(rename = "https://atomicdata.dev/properties/isA")] is_a: Vec, #[serde(rename = "https://atomicdata.dev/properties/signature")] signature: String, #[serde(rename = "https://atomicdata.dev/properties/signer")] signer: String, #[serde(rename = "https://atomicdata.dev/properties/subject")] subject: String, } #[derive(Debug, Deserialize)] struct FileStruct { #[serde(rename = "@id")] id: String, #[serde(rename = "https://atomicdata.dev/properties/downloadURL")] download_url: String, } #[derive(Debug, Deserialize)] struct QueryResultStruct { #[serde( rename = "https://atomicdata.dev/properties/endpoint/results", default = "empty_vec" )] results: Vec, } fn empty_vec() -> Vec { Vec::new() } #[derive(Clone)] pub struct Adapter { parent_resource_id: String, endpoint: String, agent: Agent, client: HttpClient, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.finish() } } impl Adapter { fn sign(&self, url: &str, mut req: http::request::Builder) -> http::request::Builder { let auth_headers = get_authentication_headers(url, &self.agent) .map_err(|err| { Error::new( ErrorKind::Unexpected, "Failed to get authentication headers", ) .with_context("service", Scheme::Atomicserver) .set_source(err) }) .unwrap(); for (k, v) in &auth_headers { req = req.header(k, v); } req } } impl Adapter { pub fn atomic_get_object_request(&self, path: &str) -> Result> { let path = normalize_path(path); let path = path.as_str(); let filename_property_escaped = FILENAME_PROPERTY.replace(':', "\\:").replace('.', "\\."); let url = format!( "{}/search?filters={}:%22{}%22", self.endpoint, percent_encode_path(&filename_property_escaped), percent_encode_path(path) ); let mut req = Request::get(&url); req = self.sign(&url, req); req = req.header(http::header::ACCEPT, "application/ad+json"); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } fn atomic_post_object_request(&self, path: &str, value: Buffer) -> Result> { let path = normalize_path(path); let path = path.as_str(); let url = format!( "{}/upload?parent={}", self.endpoint, percent_encode_path(&self.parent_resource_id) ); let mut req = Request::post(&url); req = self.sign(&url, req); let datapart = FormDataPart::new("assets") .header( CONTENT_DISPOSITION, format!("form-data; name=\"assets\"; filename=\"{}\"", path) .parse() .unwrap(), ) .header(CONTENT_TYPE, "text/plain".parse().unwrap()) .content(value.to_vec()); let multipart = Multipart::new().part(datapart); let req = multipart.apply(req)?; Ok(req) } pub fn atomic_delete_object_request(&self, subject: &str) -> Result> { let url = format!("{}/commit", self.endpoint); let timestamp = std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) .expect("You're a time traveler") .as_millis() as i64; let commit_to_sign = CommitStruct { created_at: timestamp, destroy: true, is_a: ["https://atomicdata.dev/classes/Commit".to_string()].to_vec(), signer: self.agent.subject.to_string(), subject: subject.to_string().clone(), }; let commit_sign_string = serde_json::to_string(&commit_to_sign).map_err(new_json_serialize_error)?; let signature = sign_message( &commit_sign_string, self.agent.private_key.as_ref().unwrap(), &self.agent.public_key, ) .unwrap(); let commit = CommitStructSigned { created_at: timestamp, destroy: true, is_a: ["https://atomicdata.dev/classes/Commit".to_string()].to_vec(), signature, signer: self.agent.subject.to_string(), subject: subject.to_string().clone(), }; let req = Request::post(&url); let body_string = serde_json::to_string(&commit).map_err(new_json_serialize_error)?; let body_bytes = body_string.as_bytes().to_owned(); let req = req .body(Buffer::from(body_bytes)) .map_err(new_request_build_error)?; Ok(req) } pub async fn download_from_url(&self, download_url: &String) -> Result { let req = Request::get(download_url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; Ok(resp.into_body()) } } impl Adapter { async fn wait_for_resource(&self, path: &str, expect_exist: bool) -> Result<()> { // This is used to wait until insert/delete is actually effective // This wait function is needed because atomicserver commits are not processed in real-time // See https://docs.atomicdata.dev/commits/intro.html#motivation for _i in 0..1000 { let req = self.atomic_get_object_request(path)?; let resp = self.client.send(req).await?; let bytes = resp.into_body(); let query_result: QueryResultStruct = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; if !expect_exist && query_result.results.is_empty() { break; } if expect_exist && !query_result.results.is_empty() { break; } std::thread::sleep(std::time::Duration::from_millis(30)); } Ok(()) } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Atomicserver, "atomicserver", Capability { read: true, write: true, delete: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let req = self.atomic_get_object_request(path)?; let resp = self.client.send(req).await?; let bytes = resp.into_body(); let query_result: QueryResultStruct = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; if query_result.results.is_empty() { return Err(Error::new( ErrorKind::NotFound, "atomicserver: key not found", )); } let bytes_file = self .download_from_url(&query_result.results[0].download_url) .await?; Ok(Some(bytes_file)) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let req = self.atomic_get_object_request(path)?; let res = self.client.send(req).await?; let bytes = res.into_body(); let query_result: QueryResultStruct = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; for result in query_result.results { let req = self.atomic_delete_object_request(&result.id)?; let _res = self.client.send(req).await?; } let _ = self.wait_for_resource(path, false).await; let req = self.atomic_post_object_request(path, value)?; let _res = self.client.send(req).await?; let _ = self.wait_for_resource(path, true).await; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let req = self.atomic_get_object_request(path)?; let res = self.client.send(req).await?; let bytes = res.into_body(); let query_result: QueryResultStruct = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; for result in query_result.results { let req = self.atomic_delete_object_request(&result.id)?; let _res = self.client.send(req).await?; } let _ = self.wait_for_resource(path, false).await; Ok(()) } } opendal-0.52.0/src/services/atomicserver/config.rs000064400000000000000000000034471046102023000202720ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Atomicserver services support #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct AtomicserverConfig { /// work dir of this backend pub root: Option, /// endpoint of this backend pub endpoint: Option, /// private_key of this backend pub private_key: Option, /// public_key of this backend pub public_key: Option, /// parent_resource_id of this backend pub parent_resource_id: Option, } impl Debug for AtomicserverConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("AtomicserverConfig") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("public_key", &self.public_key) .field("parent_resource_id", &self.parent_resource_id) .finish_non_exhaustive() } } opendal-0.52.0/src/services/atomicserver/docs.md000064400000000000000000000024571046102023000177310ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `endpoint`: Set the server address for `Atomicserver` - `private_key`: Set the private key for agent used for `Atomicserver` - `public_key`: Set the public key for agent used for `Atomicserver` - `parent_resource_id`: Set the parent resource id (url) that `Atomicserver` uses to store resources under You can refer to [`AtomicserverBuilder`]'s docs for more information. ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Atomicserver; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Atomicserver::default() // Set the server address for Atomicserver .endpoint("http://localhost:9883") // Set the public/private key for agent for Atomicserver .private_key("") .public_key("") // Set the parent resource id for Atomicserver. In this case // We are using the root resource (Drive) .parent_resource_id("http://localhost:9883"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/atomicserver/mod.rs000064400000000000000000000017521046102023000176010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-atomicserver")] mod backend; #[cfg(feature = "services-atomicserver")] pub use backend::AtomicserverBuilder as Atomicserver; mod config; pub use config::AtomicserverConfig; opendal-0.52.0/src/services/azblob/backend.rs000064400000000000000000000623671046102023000171700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use base64::prelude::BASE64_STANDARD; use base64::Engine; use http::Response; use http::StatusCode; use log::debug; use reqsign::AzureStorageConfig; use reqsign::AzureStorageLoader; use reqsign::AzureStorageSigner; use sha2::Digest; use sha2::Sha256; use super::core::constants::X_MS_META_PREFIX; use super::core::AzblobCore; use super::delete::AzblobDeleter; use super::error::parse_error; use super::lister::AzblobLister; use super::writer::AzblobWriter; use super::writer::AzblobWriters; use crate::raw::*; use crate::services::AzblobConfig; use crate::*; /// Known endpoint suffix Azure Storage Blob services resource URI syntax. /// Azure public cloud: https://accountname.blob.core.windows.net /// Azure US Government: https://accountname.blob.core.usgovcloudapi.net /// Azure China: https://accountname.blob.core.chinacloudapi.cn const KNOWN_AZBLOB_ENDPOINT_SUFFIX: &[&str] = &[ "blob.core.windows.net", "blob.core.usgovcloudapi.net", "blob.core.chinacloudapi.cn", ]; const AZBLOB_BATCH_LIMIT: usize = 256; impl Configurator for AzblobConfig { type Builder = AzblobBuilder; fn into_builder(self) -> Self::Builder { AzblobBuilder { config: self, http_client: None, } } } #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct AzblobBuilder { config: AzblobConfig, http_client: Option, } impl Debug for AzblobBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("AzblobBuilder"); ds.field("config", &self.config); ds.finish() } } impl AzblobBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set container name of this backend. pub fn container(mut self, container: &str) -> Self { self.config.container = container.to_string(); self } /// Set endpoint of this backend /// /// Endpoint must be full uri, e.g. /// /// - Azblob: `https://accountname.blob.core.windows.net` /// - Azurite: `http://127.0.0.1:10000/devstoreaccount1` pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()); } self } /// Set account_name of this backend. /// /// - If account_name is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn account_name(mut self, account_name: &str) -> Self { if !account_name.is_empty() { self.config.account_name = Some(account_name.to_string()); } self } /// Set account_key of this backend. /// /// - If account_key is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn account_key(mut self, account_key: &str) -> Self { if !account_key.is_empty() { self.config.account_key = Some(account_key.to_string()); } self } /// Set encryption_key of this backend. /// /// # Args /// /// `v`: Base64-encoded key that matches algorithm specified in `encryption_algorithm`. /// /// # Note /// /// This function is the low-level setting for SSE related features. /// /// SSE related options should be set carefully to make them works. /// Please use `server_side_encryption_with_*` helpers if even possible. pub fn encryption_key(mut self, v: &str) -> Self { if !v.is_empty() { self.config.encryption_key = Some(v.to_string()); } self } /// Set encryption_key_sha256 of this backend. /// /// # Args /// /// `v`: Base64-encoded SHA256 digest of the key specified in encryption_key. /// /// # Note /// /// This function is the low-level setting for SSE related features. /// /// SSE related options should be set carefully to make them works. /// Please use `server_side_encryption_with_*` helpers if even possible. pub fn encryption_key_sha256(mut self, v: &str) -> Self { if !v.is_empty() { self.config.encryption_key_sha256 = Some(v.to_string()); } self } /// Set encryption_algorithm of this backend. /// /// # Args /// /// `v`: server-side encryption algorithm. (Available values: `AES256`) /// /// # Note /// /// This function is the low-level setting for SSE related features. /// /// SSE related options should be set carefully to make them works. /// Please use `server_side_encryption_with_*` helpers if even possible. pub fn encryption_algorithm(mut self, v: &str) -> Self { if !v.is_empty() { self.config.encryption_algorithm = Some(v.to_string()); } self } /// Enable server side encryption with customer key. /// /// As known as: CPK /// /// # Args /// /// `key`: Base64-encoded SHA256 digest of the key specified in encryption_key. /// /// # Note /// /// Function that helps the user to set the server-side customer-provided encryption key, the key's SHA256, and the algorithm. /// See [Server-side encryption with customer-provided keys (CPK)](https://learn.microsoft.com/en-us/azure/storage/blobs/encryption-customer-provided-keys) /// for more info. pub fn server_side_encryption_with_customer_key(mut self, key: &[u8]) -> Self { // Only AES256 is supported for now self.config.encryption_algorithm = Some("AES256".to_string()); self.config.encryption_key = Some(BASE64_STANDARD.encode(key)); self.config.encryption_key_sha256 = Some(BASE64_STANDARD.encode(Sha256::digest(key).as_slice())); self } /// Set sas_token of this backend. /// /// - If sas_token is set, we will take user's input first. /// - If not, we will try to load it from environment. /// /// See [Grant limited access to Azure Storage resources using shared access signatures (SAS)](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) /// for more info. pub fn sas_token(mut self, sas_token: &str) -> Self { if !sas_token.is_empty() { self.config.sas_token = Some(sas_token.to_string()); } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } /// Set maximum batch operations of this backend. pub fn batch_max_operations(mut self, batch_max_operations: usize) -> Self { self.config.batch_max_operations = Some(batch_max_operations); self } /// from_connection_string will make a builder from connection string /// /// connection string looks like: /// /// ```txt /// DefaultEndpointsProtocol=http;AccountName=devstoreaccount1; /// AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==; /// BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1; /// QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1; /// TableEndpoint=http://127.0.0.1:10002/devstoreaccount1; /// ``` /// /// Or /// /// ```txt /// DefaultEndpointsProtocol=https; /// AccountName=storagesample; /// AccountKey=; /// EndpointSuffix=core.chinacloudapi.cn; /// ``` /// /// For reference: [Configure Azure Storage connection strings](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string) /// /// # Note /// /// connection string only configures the endpoint, account name and account key. /// User still needs to configure bucket names. pub fn from_connection_string(conn: &str) -> Result { let conn = conn.trim().replace('\n', ""); let mut conn_map: HashMap<_, _> = HashMap::default(); for v in conn.split(';') { let entry: Vec<_> = v.splitn(2, '=').collect(); if entry.len() != 2 { // Ignore invalid entries. continue; } conn_map.insert(entry[0], entry[1]); } let mut builder = AzblobBuilder::default(); if let Some(sas_token) = conn_map.get("SharedAccessSignature") { builder = builder.sas_token(sas_token); } else { let account_name = conn_map.get("AccountName").ok_or_else(|| { Error::new( ErrorKind::ConfigInvalid, "connection string must have AccountName", ) .with_operation("Builder::from_connection_string") })?; builder = builder.account_name(account_name); let account_key = conn_map.get("AccountKey").ok_or_else(|| { Error::new( ErrorKind::ConfigInvalid, "connection string must have AccountKey", ) .with_operation("Builder::from_connection_string") })?; builder = builder.account_key(account_key); } if let Some(v) = conn_map.get("BlobEndpoint") { builder = builder.endpoint(v); } else if let Some(v) = conn_map.get("EndpointSuffix") { let protocol = conn_map.get("DefaultEndpointsProtocol").unwrap_or(&"https"); let account_name = builder .config .account_name .as_ref() .ok_or_else(|| { Error::new( ErrorKind::ConfigInvalid, "connection string must have AccountName", ) .with_operation("Builder::from_connection_string") })? .clone(); builder = builder.endpoint(&format!("{protocol}://{account_name}.blob.{v}")); } Ok(builder) } } impl Builder for AzblobBuilder { const SCHEME: Scheme = Scheme::Azblob; type Config = AzblobConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); // Handle endpoint, region and container name. let container = match self.config.container.is_empty() { false => Ok(&self.config.container), true => Err(Error::new(ErrorKind::ConfigInvalid, "container is empty") .with_operation("Builder::build") .with_context("service", Scheme::Azblob)), }?; debug!("backend use container {}", &container); let endpoint = match &self.config.endpoint { Some(endpoint) => Ok(endpoint.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Azblob)), }?; debug!("backend use endpoint {}", &container); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Azblob) })? }; let mut config_loader = AzureStorageConfig::default().from_env(); if let Some(v) = self .config .account_name .clone() .or_else(|| infer_storage_name_from_endpoint(endpoint.as_str())) { config_loader.account_name = Some(v); } if let Some(v) = self.config.account_key.clone() { config_loader.account_key = Some(v); } if let Some(v) = self.config.sas_token.clone() { config_loader.sas_token = Some(v); } let encryption_key = match &self.config.encryption_key { None => None, Some(v) => Some(build_header_value(v).map_err(|err| { err.with_context("key", "server_side_encryption_customer_key") })?), }; let encryption_key_sha256 = match &self.config.encryption_key_sha256 { None => None, Some(v) => Some(build_header_value(v).map_err(|err| { err.with_context("key", "server_side_encryption_customer_key_sha256") })?), }; let encryption_algorithm = match &self.config.encryption_algorithm { None => None, Some(v) => { if v == "AES256" { Some(build_header_value(v).map_err(|err| { err.with_context("key", "server_side_encryption_customer_algorithm") })?) } else { return Err(Error::new( ErrorKind::ConfigInvalid, "encryption_algorithm value must be AES256", )); } } }; let cred_loader = AzureStorageLoader::new(config_loader); let signer = AzureStorageSigner::new(); Ok(AzblobBackend { core: Arc::new(AzblobCore { root, endpoint, encryption_key, encryption_key_sha256, encryption_algorithm, container: self.config.container.clone(), client, loader: cred_loader, signer, }), has_sas_token: self.config.sas_token.is_some(), }) } } fn infer_storage_name_from_endpoint(endpoint: &str) -> Option { let endpoint: &str = endpoint .strip_prefix("http://") .or_else(|| endpoint.strip_prefix("https://")) .unwrap_or(endpoint); let mut parts = endpoint.splitn(2, '.'); let storage_name = parts.next(); let endpoint_suffix = parts .next() .unwrap_or_default() .trim_end_matches('/') .to_lowercase(); if KNOWN_AZBLOB_ENDPOINT_SUFFIX .iter() .any(|s| *s == endpoint_suffix.as_str()) { storage_name.map(|s| s.to_string()) } else { None } } /// Backend for azblob services. #[derive(Debug, Clone)] pub struct AzblobBackend { core: Arc, has_sas_token: bool, } impl Access for AzblobBackend { type Reader = HttpBody; type Writer = AzblobWriters; type Lister = oio::PageLister; type Deleter = oio::BatchDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Azblob) .set_root(&self.core.root) .set_name(&self.core.container) .set_native_capability(Capability { stat: true, stat_with_if_match: true, stat_with_if_none_match: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, read: true, read_with_if_match: true, read_with_if_none_match: true, read_with_override_content_disposition: true, read_with_if_modified_since: true, read_with_if_unmodified_since: true, write: true, write_can_append: true, write_can_empty: true, write_can_multi: true, write_with_cache_control: true, write_with_content_type: true, write_with_if_not_exists: true, write_with_if_none_match: true, write_with_user_metadata: true, delete: true, delete_max_size: Some(AZBLOB_BATCH_LIMIT), copy: true, list: true, list_with_recursive: true, list_has_etag: true, list_has_content_length: true, list_has_content_md5: true, list_has_content_type: true, list_has_last_modified: true, presign: self.has_sas_token, presign_stat: self.has_sas_token, presign_read: self.has_sas_token, presign_write: self.has_sas_token, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, args: OpStat) -> Result { let resp = self.core.azblob_get_blob_properties(path, &args).await?; let status = resp.status(); match status { StatusCode::OK => { let headers = resp.headers(); let mut meta = parse_into_metadata(path, headers)?; let user_meta = parse_prefixed_headers(headers, X_MS_META_PREFIX); if !user_meta.is_empty() { meta.with_user_metadata(user_meta); } Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.azblob_get_blob(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let w = AzblobWriter::new(self.core.clone(), args.clone(), path.to_string()); let w = if args.append() { AzblobWriters::Two(oio::AppendWriter::new(w)) } else { AzblobWriters::One(oio::BlockWriter::new( w, args.executor().cloned(), args.concurrent(), )) }; Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::BatchDeleter::new(AzblobDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = AzblobLister::new( self.core.clone(), path.to_string(), args.recursive(), args.limit(), ); Ok((RpList::default(), oio::PageLister::new(l))) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.azblob_copy_blob(from, to).await?; let status = resp.status(); match status { StatusCode::ACCEPTED => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn presign(&self, path: &str, args: OpPresign) -> Result { let mut req = match args.operation() { PresignOperation::Stat(v) => self.core.azblob_head_blob_request(path, v)?, PresignOperation::Read(v) => { self.core .azblob_get_blob_request(path, BytesRange::default(), v)? } PresignOperation::Write(_) => { self.core .azblob_put_blob_request(path, None, &OpWrite::default(), Buffer::new())? } }; self.core.sign_query(&mut req).await?; let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } } #[cfg(test)] mod tests { use super::infer_storage_name_from_endpoint; use super::AzblobBuilder; #[test] fn test_infer_storage_name_from_endpoint() { let endpoint = "https://account.blob.core.windows.net"; let storage_name = infer_storage_name_from_endpoint(endpoint); assert_eq!(storage_name, Some("account".to_string())); } #[test] fn test_infer_storage_name_from_endpoint_with_trailing_slash() { let endpoint = "https://account.blob.core.windows.net/"; let storage_name = infer_storage_name_from_endpoint(endpoint); assert_eq!(storage_name, Some("account".to_string())); } #[test] fn test_builder_from_connection_string() { let builder = AzblobBuilder::from_connection_string( r#" DefaultEndpointsProtocol=http;AccountName=devstoreaccount1; AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==; BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1; QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1; TableEndpoint=http://127.0.0.1:10002/devstoreaccount1; "#, ) .expect("from connection string must succeed"); assert_eq!( builder.config.endpoint.unwrap(), "http://127.0.0.1:10000/devstoreaccount1" ); assert_eq!(builder.config.account_name.unwrap(), "devstoreaccount1"); assert_eq!(builder.config.account_key.unwrap(), "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="); let builder = AzblobBuilder::from_connection_string( r#" DefaultEndpointsProtocol=https; AccountName=storagesample; AccountKey=account-key; EndpointSuffix=core.chinacloudapi.cn; "#, ) .expect("from connection string must succeed"); assert_eq!( builder.config.endpoint.unwrap(), "https://storagesample.blob.core.chinacloudapi.cn" ); assert_eq!(builder.config.account_name.unwrap(), "storagesample"); assert_eq!(builder.config.account_key.unwrap(), "account-key") } #[test] fn test_sas_from_connection_string() { // Note, not a correct HMAC let builder = AzblobBuilder::from_connection_string( r#" BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1; QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1; TableEndpoint=http://127.0.0.1:10002/devstoreaccount1; SharedAccessSignature=sv=2021-01-01&ss=b&srt=c&sp=rwdlaciytfx&se=2022-01-01T11:00:14Z&st=2022-01-02T03:00:14Z&spr=https&sig=KEllk4N8f7rJfLjQCmikL2fRVt%2B%2Bl73UBkbgH%2FK3VGE%3D "#, ) .expect("from connection string must succeed"); assert_eq!( builder.config.endpoint.unwrap(), "http://127.0.0.1:10000/devstoreaccount1" ); assert_eq!(builder.config.sas_token.unwrap(), "sv=2021-01-01&ss=b&srt=c&sp=rwdlaciytfx&se=2022-01-01T11:00:14Z&st=2022-01-02T03:00:14Z&spr=https&sig=KEllk4N8f7rJfLjQCmikL2fRVt%2B%2Bl73UBkbgH%2FK3VGE%3D"); assert_eq!(builder.config.account_name, None); assert_eq!(builder.config.account_key, None); } #[test] pub fn test_sas_preferred() { let builder = AzblobBuilder::from_connection_string( r#" BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1; AccountName=storagesample; AccountKey=account-key; SharedAccessSignature=sv=2021-01-01&ss=b&srt=c&sp=rwdlaciytfx&se=2022-01-01T11:00:14Z&st=2022-01-02T03:00:14Z&spr=https&sig=KEllk4N8f7rJfLjQCmikL2fRVt%2B%2Bl73UBkbgH%2FK3VGE%3D "#, ) .expect("from connection string must succeed"); // SAS should be preferred over shared key assert_eq!(builder.config.sas_token.unwrap(), "sv=2021-01-01&ss=b&srt=c&sp=rwdlaciytfx&se=2022-01-01T11:00:14Z&st=2022-01-02T03:00:14Z&spr=https&sig=KEllk4N8f7rJfLjQCmikL2fRVt%2B%2Bl73UBkbgH%2FK3VGE%3D"); assert_eq!(builder.config.account_name, None); assert_eq!(builder.config.account_key, None); } } opendal-0.52.0/src/services/azblob/config.rs000064400000000000000000000053701046102023000170350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Azure Storage Blob services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] pub struct AzblobConfig { /// The root of Azblob service backend. /// /// All operations will happen under this root. pub root: Option, /// The container name of Azblob service backend. pub container: String, /// The endpoint of Azblob service backend. /// /// Endpoint must be full uri, e.g. /// /// - Azblob: `https://accountname.blob.core.windows.net` /// - Azurite: `http://127.0.0.1:10000/devstoreaccount1` pub endpoint: Option, /// The account name of Azblob service backend. pub account_name: Option, /// The account key of Azblob service backend. pub account_key: Option, /// The encryption key of Azblob service backend. pub encryption_key: Option, /// The encryption key sha256 of Azblob service backend. pub encryption_key_sha256: Option, /// The encryption algorithm of Azblob service backend. pub encryption_algorithm: Option, /// The sas token of Azblob service backend. pub sas_token: Option, /// The maximum batch operations of Azblob service backend. pub batch_max_operations: Option, } impl Debug for AzblobConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("AzblobConfig"); ds.field("root", &self.root); ds.field("container", &self.container); ds.field("endpoint", &self.endpoint); if self.account_name.is_some() { ds.field("account_name", &""); } if self.account_key.is_some() { ds.field("account_key", &""); } if self.sas_token.is_some() { ds.field("sas_token", &""); } ds.finish() } } opendal-0.52.0/src/services/azblob/core.rs000064400000000000000000000763551046102023000165330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt; use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::Write; use std::time::Duration; use base64::prelude::BASE64_STANDARD; use base64::Engine; use bytes::Bytes; use constants::X_MS_META_PREFIX; use http::header::HeaderName; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::header::IF_MATCH; use http::header::IF_MODIFIED_SINCE; use http::header::IF_NONE_MATCH; use http::header::IF_UNMODIFIED_SINCE; use http::HeaderValue; use http::Request; use http::Response; use reqsign::AzureStorageCredential; use reqsign::AzureStorageLoader; use reqsign::AzureStorageSigner; use serde::Deserialize; use serde::Serialize; use uuid::Uuid; use crate::raw::*; use crate::*; pub mod constants { pub const X_MS_VERSION: &str = "x-ms-version"; pub const X_MS_BLOB_TYPE: &str = "x-ms-blob-type"; pub const X_MS_COPY_SOURCE: &str = "x-ms-copy-source"; pub const X_MS_BLOB_CACHE_CONTROL: &str = "x-ms-blob-cache-control"; pub const X_MS_BLOB_CONDITION_APPENDPOS: &str = "x-ms-blob-condition-appendpos"; pub const X_MS_META_PREFIX: &str = "x-ms-meta-"; // Server-side encryption with customer-provided headers pub const X_MS_ENCRYPTION_KEY: &str = "x-ms-encryption-key"; pub const X_MS_ENCRYPTION_KEY_SHA256: &str = "x-ms-encryption-key-sha256"; pub const X_MS_ENCRYPTION_ALGORITHM: &str = "x-ms-encryption-algorithm"; } pub struct AzblobCore { pub container: String, pub root: String, pub endpoint: String, pub encryption_key: Option, pub encryption_key_sha256: Option, pub encryption_algorithm: Option, pub client: HttpClient, pub loader: AzureStorageLoader, pub signer: AzureStorageSigner, } impl Debug for AzblobCore { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { f.debug_struct("AzblobCore") .field("container", &self.container) .field("root", &self.root) .field("endpoint", &self.endpoint) .finish_non_exhaustive() } } impl AzblobCore { async fn load_credential(&self) -> Result { let cred = self .loader .load() .await .map_err(new_request_credential_error)?; if let Some(cred) = cred { Ok(cred) } else { Err(Error::new( ErrorKind::ConfigInvalid, "no valid credential found", )) } } pub async fn sign_query(&self, req: &mut Request) -> Result<()> { let cred = self.load_credential().await?; self.signer .sign_query(req, Duration::from_secs(3600), &cred) .map_err(new_request_sign_error) } pub async fn sign(&self, req: &mut Request) -> Result<()> { let cred = self.load_credential().await?; // Insert x-ms-version header for normal requests. req.headers_mut().insert( HeaderName::from_static(constants::X_MS_VERSION), // 2022-11-02 is the version supported by Azurite V3 and // used by Azure Portal, We use this version to make // sure most our developer happy. // // In the future, we could allow users to configure this value. HeaderValue::from_static("2022-11-02"), ); self.signer.sign(req, &cred).map_err(new_request_sign_error) } async fn batch_sign(&self, req: &mut Request) -> Result<()> { let cred = self.load_credential().await?; self.signer.sign(req, &cred).map_err(new_request_sign_error) } #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } pub fn insert_sse_headers(&self, mut req: http::request::Builder) -> http::request::Builder { if let Some(v) = &self.encryption_key { let mut v = v.clone(); v.set_sensitive(true); req = req.header(HeaderName::from_static(constants::X_MS_ENCRYPTION_KEY), v) } if let Some(v) = &self.encryption_key_sha256 { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_MS_ENCRYPTION_KEY_SHA256), v, ) } if let Some(v) = &self.encryption_algorithm { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_MS_ENCRYPTION_ALGORITHM), v, ) } req } } impl AzblobCore { pub fn azblob_get_blob_request( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/{}/{}", self.endpoint, self.container, percent_encode_path(&p) ); let mut query_args = Vec::new(); if let Some(override_content_disposition) = args.override_content_disposition() { query_args.push(format!( "rscd={}", percent_encode_path(override_content_disposition) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let mut req = Request::get(&url); // Set SSE headers. req = self.insert_sse_headers(req); if !range.is_full() { req = req.header(http::header::RANGE, range.to_header()); } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_modified_since) = args.if_modified_since() { req = req.header( IF_MODIFIED_SINCE, format_datetime_into_http_date(if_modified_since), ); } if let Some(if_unmodified_since) = args.if_unmodified_since() { req = req.header( IF_UNMODIFIED_SINCE, format_datetime_into_http_date(if_unmodified_since), ); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub async fn azblob_get_blob( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let mut req = self.azblob_get_blob_request(path, range, args)?; self.sign(&mut req).await?; self.client.fetch(req).await } pub fn azblob_put_blob_request( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", self.endpoint, self.container, percent_encode_path(&p) ); let mut req = Request::put(&url); req = req.header( HeaderName::from_static(constants::X_MS_BLOB_TYPE), "BlockBlob", ); if let Some(size) = size { req = req.header(CONTENT_LENGTH, size) } if let Some(ty) = args.content_type() { req = req.header(CONTENT_TYPE, ty) } // Specify the wildcard character (*) to perform the operation only if // the resource does not exist, and fail the operation if it does exist. if args.if_not_exists() { req = req.header(IF_NONE_MATCH, "*"); } if let Some(v) = args.if_none_match() { req = req.header(IF_NONE_MATCH, v); } if let Some(cache_control) = args.cache_control() { req = req.header(constants::X_MS_BLOB_CACHE_CONTROL, cache_control); } // Set SSE headers. req = self.insert_sse_headers(req); if let Some(user_metadata) = args.user_metadata() { for (key, value) in user_metadata { req = req.header(format!("{X_MS_META_PREFIX}{key}"), value) } } // Set body let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } /// For appendable object, it could be created by `put` an empty blob /// with `x-ms-blob-type` header set to `AppendBlob`. /// And it's just initialized with empty content. /// /// If want to append content to it, we should use the following method `azblob_append_blob_request`. /// /// # Notes /// /// Appendable blob's custom header could only be set when it's created. /// /// The following custom header could be set: /// - `content-type` /// - `x-ms-blob-cache-control` /// /// # Reference /// /// https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob pub fn azblob_init_appendable_blob_request( &self, path: &str, args: &OpWrite, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", self.endpoint, self.container, percent_encode_path(&p) ); let mut req = Request::put(&url); // Set SSE headers. req = self.insert_sse_headers(req); // The content-length header must be set to zero // when creating an appendable blob. req = req.header(CONTENT_LENGTH, 0); req = req.header( HeaderName::from_static(constants::X_MS_BLOB_TYPE), "AppendBlob", ); if let Some(ty) = args.content_type() { req = req.header(CONTENT_TYPE, ty) } if let Some(cache_control) = args.cache_control() { req = req.header(constants::X_MS_BLOB_CACHE_CONTROL, cache_control); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } /// Append content to an appendable blob. /// The content will be appended to the end of the blob. /// /// # Notes /// /// - The maximum size of the content could be appended is 4MB. /// - `Append Block` succeeds only if the blob already exists. /// /// # Reference /// /// https://learn.microsoft.com/en-us/rest/api/storageservices/append-block pub fn azblob_append_blob_request( &self, path: &str, position: u64, size: u64, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}?comp=appendblock", self.endpoint, self.container, percent_encode_path(&p) ); let mut req = Request::put(&url); // Set SSE headers. req = self.insert_sse_headers(req); req = req.header(CONTENT_LENGTH, size); req = req.header(constants::X_MS_BLOB_CONDITION_APPENDPOS, position); let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub fn azblob_put_block_request( &self, path: &str, block_id: Uuid, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { // To be written as part of a blob, a block must have been successfully written to the server in an earlier Put Block operation. // refer to https://learn.microsoft.com/en-us/rest/api/storageservices/put-block?tabs=microsoft-entra-id let p = build_abs_path(&self.root, path); let encoded_block_id: String = percent_encode_path(&BASE64_STANDARD.encode(block_id.as_bytes())); let url = format!( "{}/{}/{}?comp=block&blockid={}", self.endpoint, self.container, percent_encode_path(&p), encoded_block_id, ); let mut req = Request::put(&url); // Set SSE headers. req = self.insert_sse_headers(req); if let Some(cache_control) = args.cache_control() { req = req.header(constants::X_MS_BLOB_CACHE_CONTROL, cache_control); } if let Some(size) = size { req = req.header(CONTENT_LENGTH, size) } if let Some(ty) = args.content_type() { req = req.header(CONTENT_TYPE, ty) } // Set body let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn azblob_put_block( &self, path: &str, block_id: Uuid, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let mut req = self.azblob_put_block_request(path, block_id, size, args, body)?; self.sign(&mut req).await?; self.send(req).await } pub fn azblob_complete_put_block_list_request( &self, path: &str, block_ids: Vec, args: &OpWrite, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}?comp=blocklist", self.endpoint, self.container, percent_encode_path(&p), ); let req = Request::put(&url); // Set SSE headers. let mut req = self.insert_sse_headers(req); if let Some(cache_control) = args.cache_control() { req = req.header(constants::X_MS_BLOB_CACHE_CONTROL, cache_control); } let content = quick_xml::se::to_string(&PutBlockListRequest { latest: block_ids .into_iter() .map(|block_id| { let encoded_block_id: String = BASE64_STANDARD.encode(block_id.as_bytes()); encoded_block_id }) .collect(), }) .map_err(new_xml_deserialize_error)?; req = req.header(CONTENT_LENGTH, content.len()); let req = req .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; Ok(req) } pub async fn azblob_complete_put_block_list( &self, path: &str, block_ids: Vec, args: &OpWrite, ) -> Result> { let mut req = self.azblob_complete_put_block_list_request(path, block_ids, args)?; self.sign(&mut req).await?; self.send(req).await } pub fn azblob_head_blob_request(&self, path: &str, args: &OpStat) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", self.endpoint, self.container, percent_encode_path(&p) ); let mut req = Request::head(&url); // Set SSE headers. req = self.insert_sse_headers(req); if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub async fn azblob_get_blob_properties( &self, path: &str, args: &OpStat, ) -> Result> { let mut req = self.azblob_head_blob_request(path, args)?; self.sign(&mut req).await?; self.send(req).await } pub fn azblob_delete_blob_request(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", self.endpoint, self.container, percent_encode_path(&p) ); let req = Request::delete(&url); req.header(CONTENT_LENGTH, 0) .body(Buffer::new()) .map_err(new_request_build_error) } pub async fn azblob_delete_blob(&self, path: &str) -> Result> { let mut req = self.azblob_delete_blob_request(path)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azblob_copy_blob(&self, from: &str, to: &str) -> Result> { let source = build_abs_path(&self.root, from); let target = build_abs_path(&self.root, to); let source = format!( "{}/{}/{}", self.endpoint, self.container, percent_encode_path(&source) ); let target = format!( "{}/{}/{}", self.endpoint, self.container, percent_encode_path(&target) ); let mut req = Request::put(&target) .header(constants::X_MS_COPY_SOURCE, source) .header(CONTENT_LENGTH, 0) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azblob_list_blobs( &self, path: &str, next_marker: &str, delimiter: &str, limit: Option, ) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/{}?restype=container&comp=list", self.endpoint, self.container ); if !p.is_empty() { write!(url, "&prefix={}", percent_encode_path(&p)) .expect("write into string must succeed"); } if let Some(limit) = limit { write!(url, "&maxresults={limit}").expect("write into string must succeed"); } if !delimiter.is_empty() { write!(url, "&delimiter={delimiter}").expect("write into string must succeed"); } if !next_marker.is_empty() { write!(url, "&marker={next_marker}").expect("write into string must succeed"); } let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azblob_batch_delete(&self, paths: &[String]) -> Result> { let url = format!( "{}/{}?restype=container&comp=batch", self.endpoint, self.container ); let mut multipart = Multipart::new(); for (idx, path) in paths.iter().enumerate() { let mut req = self.azblob_delete_blob_request(path)?; self.batch_sign(&mut req).await?; multipart = multipart.part( MixedPart::from_request(req).part_header("content-id".parse().unwrap(), idx.into()), ); } let req = Request::post(url); let mut req = multipart.apply(req)?; self.sign(&mut req).await?; self.send(req).await } } /// Request of PutBlockListRequest #[derive(Default, Debug, Serialize, Deserialize)] #[serde(default, rename = "BlockList", rename_all = "PascalCase")] pub struct PutBlockListRequest { pub latest: Vec, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListBlobsOutput { pub blobs: Blobs, #[serde(rename = "NextMarker")] pub next_marker: Option, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct Blobs { pub blob: Vec, pub blob_prefix: Vec, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct BlobPrefix { pub name: String, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct Blob { pub properties: Properties, pub name: String, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct Properties { #[serde(rename = "Content-Length")] pub content_length: u64, #[serde(rename = "Last-Modified")] pub last_modified: String, #[serde(rename = "Content-MD5")] pub content_md5: String, #[serde(rename = "Content-Type")] pub content_type: String, pub etag: String, } #[cfg(test)] mod tests { use bytes::Buf; use bytes::Bytes; use quick_xml::de; use super::*; #[test] fn test_parse_xml() { let bs = bytes::Bytes::from( r#" dir1/ / dir1/2f018bb5-466f-4af1-84fa-2b167374ee06 Sun, 20 Mar 2022 11:29:03 GMT Sun, 20 Mar 2022 11:29:03 GMT 0x8DA0A64D66790C3 3485277 application/octet-stream llJ/+jOlx5GdA1sL7SdKuw== BlockBlob Hot true unlocked available true dir1/5b9432b2-79c0-48d8-90c2-7d3e153826ed Tue, 29 Mar 2022 01:54:07 GMT Tue, 29 Mar 2022 01:54:07 GMT 0x8DA112702D88FE4 2471869 application/octet-stream xmgUltSnopLSJOukgCHFtg== BlockBlob Hot true unlocked available true dir1/b2d96f8b-d467-40d1-bb11-4632dddbf5b5 Sun, 20 Mar 2022 11:31:57 GMT Sun, 20 Mar 2022 11:31:57 GMT 0x8DA0A653DC82981 1259677 application/octet-stream AxTiFXHwrXKaZC5b7ZRybw== BlockBlob Hot true unlocked available true dir1/dir2/ dir1/dir21/ "#, ); let out: ListBlobsOutput = de::from_reader(bs.reader()).expect("must success"); println!("{out:?}"); assert_eq!( out.blobs .blob .iter() .map(|v| v.name.clone()) .collect::>(), [ "dir1/2f018bb5-466f-4af1-84fa-2b167374ee06", "dir1/5b9432b2-79c0-48d8-90c2-7d3e153826ed", "dir1/b2d96f8b-d467-40d1-bb11-4632dddbf5b5" ] ); assert_eq!( out.blobs .blob .iter() .map(|v| v.properties.content_length) .collect::>(), [3485277, 2471869, 1259677] ); assert_eq!( out.blobs .blob .iter() .map(|v| v.properties.content_md5.clone()) .collect::>(), [ "llJ/+jOlx5GdA1sL7SdKuw==".to_string(), "xmgUltSnopLSJOukgCHFtg==".to_string(), "AxTiFXHwrXKaZC5b7ZRybw==".to_string() ] ); assert_eq!( out.blobs .blob .iter() .map(|v| v.properties.last_modified.clone()) .collect::>(), [ "Sun, 20 Mar 2022 11:29:03 GMT".to_string(), "Tue, 29 Mar 2022 01:54:07 GMT".to_string(), "Sun, 20 Mar 2022 11:31:57 GMT".to_string() ] ); assert_eq!( out.blobs .blob .iter() .map(|v| v.properties.etag.clone()) .collect::>(), [ "0x8DA0A64D66790C3".to_string(), "0x8DA112702D88FE4".to_string(), "0x8DA0A653DC82981".to_string() ] ); assert_eq!( out.blobs .blob_prefix .iter() .map(|v| v.name.clone()) .collect::>(), ["dir1/dir2/", "dir1/dir21/"] ); } /// This case is copied from real environment for testing /// quick-xml overlapped-lists features. By default, quick-xml /// can't deserialize content with overlapped-lists. /// /// For example, this case list blobs in this way: /// /// ```xml /// /// xxx /// yyy /// zzz /// /// ``` /// /// If `overlapped-lists` feature not enabled, we will get error `duplicate field Blob`. #[test] fn test_parse_overlapped_lists() { let bs = "9f7075e1-84d0-45ca-8196-ab9b71a8ef97/x//9f7075e1-84d0-45ca-8196-ab9b71a8ef97/x/Thu, 01 Sep 2022 07:26:49 GMTThu, 01 Sep 2022 07:26:49 GMT0x8DA8BEB55D0EA350application/octet-stream1B2M2Y8AsgTpgAmY7PhCfg==BlockBlobHottrueunlockedavailabletrue9f7075e1-84d0-45ca-8196-ab9b71a8ef97/x/x/9f7075e1-84d0-45ca-8196-ab9b71a8ef97/x/yThu, 01 Sep 2022 07:26:50 GMTThu, 01 Sep 2022 07:26:50 GMT0x8DA8BEB55D99C080application/octet-stream1B2M2Y8AsgTpgAmY7PhCfg==BlockBlobHottrueunlockedavailabletrue"; de::from_reader(Bytes::from(bs).reader()).expect("must success") } /// This example is from https://learn.microsoft.com/en-us/rest/api/storageservices/put-block-list?tabs=microsoft-entra-id #[test] fn test_serialize_put_block_list_request() { let req = PutBlockListRequest { latest: vec!["1".to_string(), "2".to_string(), "3".to_string()], }; let actual = quick_xml::se::to_string(&req).expect("must succeed"); pretty_assertions::assert_eq!( actual, r#" 1 2 3 "# // Cleanup space and new line .replace([' ', '\n'], "") // Escape `"` by hand to address .replace('"', """) ); let bs = " 1 2 3 "; let out: PutBlockListRequest = de::from_reader(Bytes::from(bs).reader()).expect("must success"); assert_eq!( out.latest, vec!["1".to_string(), "2".to_string(), "3".to_string()] ); } } opendal-0.52.0/src/services/azblob/delete.rs000064400000000000000000000067671046102023000170450ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::oio::BatchDeleteResult; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct AzblobDeleter { core: Arc, } impl AzblobDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::BatchDelete for AzblobDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.azblob_delete_blob(&path).await?; let status = resp.status(); match status { StatusCode::ACCEPTED | StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } async fn delete_batch(&self, batch: Vec<(String, OpDelete)>) -> Result { // TODO: Add remove version support. let paths = batch.into_iter().map(|(p, _)| p).collect::>(); // construct and complete batch request let resp = self.core.azblob_batch_delete(&paths).await?; // check response status if resp.status() != StatusCode::ACCEPTED { return Err(parse_error(resp)); } // get boundary from response header let boundary = parse_multipart_boundary(resp.headers())? .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "invalid response: no boundary provided in header", ) })? .to_string(); let bs = resp.into_body().to_bytes(); let multipart: Multipart = Multipart::new().with_boundary(&boundary).parse(bs)?; let parts = multipart.into_parts(); if paths.len() != parts.len() { return Err(Error::new( ErrorKind::Unexpected, "invalid batch response, paths and response parts don't match", )); } let mut batched_result = BatchDeleteResult::default(); for (i, part) in parts.into_iter().enumerate() { let resp = part.into_response(); let path = paths[i].clone(); // deleting not existing objects is ok if resp.status() == StatusCode::ACCEPTED || resp.status() == StatusCode::NOT_FOUND { batched_result.succeeded.push((path, OpDelete::default())); } else { batched_result .failed .push((path, OpDelete::default(), parse_error(resp))); } } // If no object is deleted, return directly. if batched_result.succeeded.is_empty() { let err = batched_result.failed.remove(0).2; return Err(err); } Ok(batched_result) } } opendal-0.52.0/src/services/azblob/docs.md000064400000000000000000000044401046102023000164710ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] append - [x] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [x] presign - [ ] blocking ## Configuration - `root`: Set the work dir for backend. - `container`: Set the container name for backend. - `endpoint`: Set the endpoint for backend. - `account_name`: Set the account_name for backend. - `account_key`: Set the account_key for backend. Refer to public API docs for more information. ## Examples This example works on [Azurite](https://github.com/Azure/Azurite) for local developments. ### Start local blob service ```shell docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite az storage container create --name test --connection-string "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;" ``` ### Init OpenDAL Operator ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Azblob; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create azblob backend builder. let mut builder = Azblob::default() // Set the root for azblob, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/path/to/dir") // Set the container name, this is required. .container("test") // Set the endpoint, this is required. // // For examples: // - "http://127.0.0.1:10000/devstoreaccount1" // - "https://accountname.blob.core.windows.net" .endpoint("http://127.0.0.1:10000/devstoreaccount1") // Set the account_name and account_key. // // OpenDAL will try load credential from the env. // If credential not set and no valid credential in env, OpenDAL will // send request without signing like anonymous user. .account_name("devstoreaccount1") .account_key("Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="); // `Accessor` provides the low level APIs, we will use `Operator` normally. let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/azblob/error.rs000064400000000000000000000120021046102023000167070ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use bytes::Buf; use http::Response; use http::StatusCode; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// AzblobError is the error returned by azure blob service. #[derive(Default, Deserialize)] #[serde(default, rename_all = "PascalCase")] struct AzblobError { code: String, message: String, query_parameter_name: String, query_parameter_value: String, reason: String, } impl Debug for AzblobError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("AzblobError"); de.field("code", &self.code); // replace `\n` to ` ` for better reading. de.field("message", &self.message.replace('\n', " ")); if !self.query_parameter_name.is_empty() { de.field("query_parameter_name", &self.query_parameter_name); } if !self.query_parameter_value.is_empty() { de.field("query_parameter_value", &self.query_parameter_value); } if !self.reason.is_empty() { de.field("reason", &self.reason); } de.finish() } } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::NOT_MODIFIED | StatusCode::CONFLICT => { (ErrorKind::ConditionNotMatch, false) } StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let bs_content = bs.chunk(); let mut message = match de::from_reader::<_, AzblobError>(bs_content.reader()) { Ok(azblob_err) => format!("{azblob_err:?}"), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; // If there is no body here, fill with error code. if message.is_empty() { if let Some(v) = parts.headers.get("x-ms-error-code") { if let Ok(code) = v.to_str() { message = format!( "{:?}", AzblobError { code: code.to_string(), ..Default::default() } ) } } } let mut err = Error::new(kind, &message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod tests { use super::*; #[test] fn test_parse_error() { let bs = bytes::Bytes::from( r#" string-value string-value "#, ); let out: AzblobError = de::from_reader(bs.reader()).expect("must success"); println!("{out:?}"); assert_eq!(out.code, "string-value"); assert_eq!(out.message, "string-value"); } #[test] fn test_parse_error_with_reason() { let bs = bytes::Bytes::from( r#" InvalidQueryParameterValue Value for one of the query parameters specified in the request URI is invalid. popreceipt 33537277-6a52-4a2b-b4eb-0f905051827b invalid receipt format "#, ); let out: AzblobError = de::from_reader(bs.reader()).expect("must success"); println!("{out:?}"); assert_eq!(out.code, "InvalidQueryParameterValue"); assert_eq!( out.message, "Value for one of the query parameters specified in the request URI is invalid." ); assert_eq!(out.query_parameter_name, "popreceipt"); assert_eq!( out.query_parameter_value, "33537277-6a52-4a2b-b4eb-0f905051827b" ); assert_eq!(out.reason, "invalid receipt format"); } } opendal-0.52.0/src/services/azblob/lister.rs000064400000000000000000000064761046102023000171020ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use quick_xml::de; use super::core::AzblobCore; use super::core::ListBlobsOutput; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct AzblobLister { core: Arc, path: String, delimiter: &'static str, limit: Option, } impl AzblobLister { pub fn new(core: Arc, path: String, recursive: bool, limit: Option) -> Self { let delimiter = if recursive { "" } else { "/" }; Self { core, path, delimiter, limit, } } } impl oio::PageList for AzblobLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .azblob_list_blobs(&self.path, &ctx.token, self.delimiter, self.limit) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let output: ListBlobsOutput = de::from_reader(bs.reader()).map_err(new_xml_deserialize_error)?; // Try our best to check whether this list is done. // // - Check `next_marker` if let Some(next_marker) = output.next_marker.as_ref() { ctx.done = next_marker.is_empty(); }; ctx.token = output.next_marker.clone().unwrap_or_default(); let prefixes = output.blobs.blob_prefix; for prefix in prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix.name), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de) } for object in output.blobs.blob { let mut path = build_rel_path(&self.core.root, &object.name); if path.is_empty() { path = "/".to_string(); } let meta = Metadata::new(EntryMode::from_path(&path)) // Keep fit with ETag header. .with_etag(format!("\"{}\"", object.properties.etag.as_str())) .with_content_length(object.properties.content_length) .with_content_md5(object.properties.content_md5) .with_content_type(object.properties.content_type) .with_last_modified(parse_datetime_from_rfc2822( object.properties.last_modified.as_str(), )?); let de = oio::Entry::with(path, meta); ctx.entries.push_back(de); } Ok(()) } } opendal-0.52.0/src/services/azblob/mod.rs000064400000000000000000000023201046102023000163370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-azblob")] pub(crate) mod core; #[cfg(feature = "services-azblob")] mod delete; #[cfg(feature = "services-azblob")] mod error; #[cfg(feature = "services-azblob")] mod lister; #[cfg(feature = "services-azblob")] pub(crate) mod writer; #[cfg(feature = "services-azblob")] mod backend; #[cfg(feature = "services-azblob")] pub use backend::AzblobBuilder as Azblob; mod config; pub use config::AzblobConfig; opendal-0.52.0/src/services/azblob/writer.rs000064400000000000000000000120061046102023000170760ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use uuid::Uuid; use super::core::AzblobCore; use super::error::parse_error; use crate::raw::*; use crate::*; const X_MS_BLOB_TYPE: &str = "x-ms-blob-type"; pub type AzblobWriters = TwoWays, oio::AppendWriter>; pub struct AzblobWriter { core: Arc, op: OpWrite, path: String, } impl AzblobWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { AzblobWriter { core, op, path } } } impl oio::AppendWrite for AzblobWriter { async fn offset(&self) -> Result { let resp = self .core .azblob_get_blob_properties(&self.path, &OpStat::default()) .await?; let status = resp.status(); match status { StatusCode::OK => { let headers = resp.headers(); let blob_type = headers.get(X_MS_BLOB_TYPE).and_then(|v| v.to_str().ok()); if blob_type != Some("AppendBlob") { return Err(Error::new( ErrorKind::ConditionNotMatch, "the blob is not an appendable blob.", )); } Ok(parse_content_length(headers)?.unwrap_or_default()) } StatusCode::NOT_FOUND => { let mut req = self .core .azblob_init_appendable_blob_request(&self.path, &self.op)?; self.core.sign(&mut req).await?; let resp = self.core.client.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED => { // do nothing } _ => { return Err(parse_error(resp)); } } Ok(0) } _ => Err(parse_error(resp)), } } async fn append(&self, offset: u64, size: u64, body: Buffer) -> Result { let mut req = self .core .azblob_append_blob_request(&self.path, offset, size, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } impl oio::BlockWrite for AzblobWriter { async fn write_once(&self, size: u64, body: Buffer) -> Result { let mut req: http::Request = self.core .azblob_put_blob_request(&self.path, Some(size), &self.op, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn write_block(&self, block_id: Uuid, size: u64, body: Buffer) -> Result<()> { let resp = self .core .azblob_put_block(&self.path, block_id, Some(size), &self.op, body) .await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } async fn complete_block(&self, block_ids: Vec) -> Result { let resp = self .core .azblob_complete_put_block_list(&self.path, block_ids, &self.op) .await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn abort_block(&self, _block_ids: Vec) -> Result<()> { // refer to https://learn.microsoft.com/en-us/rest/api/storageservices/put-block-list?tabs=microsoft-entra-id // Any uncommitted blocks are garbage collected if there are no successful calls to Put Block or Put Block List on the blob within a week. // If Put Blob is called on the blob, any uncommitted blocks are garbage collected. Ok(()) } } opendal-0.52.0/src/services/azdls/backend.rs000064400000000000000000000323501046102023000170210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use log::debug; use reqsign::AzureStorageConfig; use reqsign::AzureStorageLoader; use reqsign::AzureStorageSigner; use super::core::AzdlsCore; use super::delete::AzdlsDeleter; use super::error::parse_error; use super::lister::AzdlsLister; use super::writer::AzdlsWriter; use super::writer::AzdlsWriters; use crate::raw::*; use crate::services::AzdlsConfig; use crate::*; /// Known endpoint suffix Azure Data Lake Storage Gen2 URI syntax. /// Azure public cloud: https://accountname.dfs.core.windows.net /// Azure US Government: https://accountname.dfs.core.usgovcloudapi.net /// Azure China: https://accountname.dfs.core.chinacloudapi.cn const KNOWN_AZDLS_ENDPOINT_SUFFIX: &[&str] = &[ "dfs.core.windows.net", "dfs.core.usgovcloudapi.net", "dfs.core.chinacloudapi.cn", ]; impl Configurator for AzdlsConfig { type Builder = AzdlsBuilder; fn into_builder(self) -> Self::Builder { AzdlsBuilder { config: self, http_client: None, } } } /// Azure Data Lake Storage Gen2 Support. #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct AzdlsBuilder { config: AzdlsConfig, http_client: Option, } impl Debug for AzdlsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("AzdlsBuilder"); ds.field("config", &self.config); ds.finish() } } impl AzdlsBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set filesystem name of this backend. pub fn filesystem(mut self, filesystem: &str) -> Self { self.config.filesystem = filesystem.to_string(); self } /// Set endpoint of this backend. /// /// Endpoint must be full uri, e.g. /// /// - Azblob: `https://accountname.blob.core.windows.net` /// - Azurite: `http://127.0.0.1:10000/devstoreaccount1` pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()); } self } /// Set account_name of this backend. /// /// - If account_name is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn account_name(mut self, account_name: &str) -> Self { if !account_name.is_empty() { self.config.account_name = Some(account_name.to_string()); } self } /// Set account_key of this backend. /// /// - If account_key is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn account_key(mut self, account_key: &str) -> Self { if !account_key.is_empty() { self.config.account_key = Some(account_key.to_string()); } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for AzdlsBuilder { const SCHEME: Scheme = Scheme::Azdls; type Config = AzdlsConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); // Handle endpoint, region and container name. let filesystem = match self.config.filesystem.is_empty() { false => Ok(&self.config.filesystem), true => Err(Error::new(ErrorKind::ConfigInvalid, "filesystem is empty") .with_operation("Builder::build") .with_context("service", Scheme::Azdls)), }?; debug!("backend use filesystem {}", &filesystem); let endpoint = match &self.config.endpoint { Some(endpoint) => Ok(endpoint.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Azdls)), }?; debug!("backend use endpoint {}", &endpoint); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Azdls) })? }; let config_loader = AzureStorageConfig { account_name: self .config .account_name .clone() .or_else(|| infer_storage_name_from_endpoint(endpoint.as_str())), account_key: self.config.account_key.clone(), sas_token: None, ..Default::default() }; let cred_loader = AzureStorageLoader::new(config_loader); let signer = AzureStorageSigner::new(); Ok(AzdlsBackend { core: Arc::new(AzdlsCore { filesystem: self.config.filesystem.clone(), root, endpoint, client, loader: cred_loader, signer, }), }) } } /// Backend for azblob services. #[derive(Debug, Clone)] pub struct AzdlsBackend { core: Arc, } impl Access for AzdlsBackend { type Reader = HttpBody; type Writer = AzdlsWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Azdls) .set_root(&self.core.root) .set_name(&self.core.filesystem) .set_native_capability(Capability { stat: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, read: true, write: true, write_can_append: true, write_with_if_none_match: true, write_with_if_not_exists: true, create_dir: true, delete: true, rename: true, list: true, list_has_etag: true, list_has_content_length: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let mut req = self.core.azdls_create_request( path, "directory", &OpWrite::default(), Buffer::new(), )?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(RpCreateDir::default()), _ => Err(parse_error(resp)), } } async fn stat(&self, path: &str, _: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let resp = self.core.azdls_get_properties(path).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } let mut meta = parse_into_metadata(path, resp.headers())?; let resource = resp .headers() .get("x-ms-resource-type") .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "azdls should return x-ms-resource-type header, but it's missing", ) })? .to_str() .map_err(|err| { Error::new( ErrorKind::Unexpected, "azdls should return x-ms-resource-type header, but it's not a valid string", ) .set_source(err) })?; meta = match resource { "file" => meta.with_mode(EntryMode::FILE), "directory" => meta.with_mode(EntryMode::DIR), v => { return Err(Error::new( ErrorKind::Unexpected, "azdls returns not supported x-ms-resource-type", ) .with_context("resource", v)) } }; Ok(RpStat::new(meta)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.azdls_read(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let w = AzdlsWriter::new(self.core.clone(), args.clone(), path.to_string()); let w = if args.append() { AzdlsWriters::Two(oio::AppendWriter::new(w)) } else { AzdlsWriters::One(oio::OneShotWriter::new(w)) }; Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(AzdlsDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = AzdlsLister::new(self.core.clone(), path.to_string(), args.limit()); Ok((RpList::default(), oio::PageLister::new(l))) } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { if let Some(resp) = self.core.azdls_ensure_parent_path(to).await? { let status = resp.status(); match status { StatusCode::CREATED | StatusCode::CONFLICT => {} _ => return Err(parse_error(resp)), } } let resp = self.core.azdls_rename(from, to).await?; let status = resp.status(); match status { StatusCode::CREATED => Ok(RpRename::default()), _ => Err(parse_error(resp)), } } } fn infer_storage_name_from_endpoint(endpoint: &str) -> Option { let endpoint: &str = endpoint .strip_prefix("http://") .or_else(|| endpoint.strip_prefix("https://")) .unwrap_or(endpoint); let mut parts = endpoint.splitn(2, '.'); let storage_name = parts.next(); let endpoint_suffix = parts .next() .unwrap_or_default() .trim_end_matches('/') .to_lowercase(); if KNOWN_AZDLS_ENDPOINT_SUFFIX .iter() .any(|s| *s == endpoint_suffix.as_str()) { storage_name.map(|s| s.to_string()) } else { None } } #[cfg(test)] mod tests { use super::infer_storage_name_from_endpoint; #[test] fn test_infer_storage_name_from_endpoint() { let endpoint = "https://account.dfs.core.windows.net"; let storage_name = infer_storage_name_from_endpoint(endpoint); assert_eq!(storage_name, Some("account".to_string())); } #[test] fn test_infer_storage_name_from_endpoint_with_trailing_slash() { let endpoint = "https://account.dfs.core.windows.net/"; let storage_name = infer_storage_name_from_endpoint(endpoint); assert_eq!(storage_name, Some("account".to_string())); } } opendal-0.52.0/src/services/azdls/config.rs000064400000000000000000000035451046102023000167030ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Azure Data Lake Storage Gen2 Support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] pub struct AzdlsConfig { /// Root of this backend. pub root: Option, /// Filesystem name of this backend. pub filesystem: String, /// Endpoint of this backend. pub endpoint: Option, /// Account name of this backend. pub account_name: Option, /// Account key of this backend. pub account_key: Option, } impl Debug for AzdlsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("AzdlsConfig"); ds.field("root", &self.root); ds.field("filesystem", &self.filesystem); ds.field("endpoint", &self.endpoint); if self.account_name.is_some() { ds.field("account_name", &""); } if self.account_key.is_some() { ds.field("account_key", &""); } ds.finish() } } opendal-0.52.0/src/services/azdls/core.rs000064400000000000000000000230341046102023000163610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt; use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::Write; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::header::{CONTENT_DISPOSITION, IF_NONE_MATCH}; use http::HeaderName; use http::HeaderValue; use http::Request; use http::Response; use reqsign::AzureStorageCredential; use reqsign::AzureStorageLoader; use reqsign::AzureStorageSigner; use crate::raw::*; use crate::*; const X_MS_RENAME_SOURCE: &str = "x-ms-rename-source"; const X_MS_VERSION: &str = "x-ms-version"; pub struct AzdlsCore { pub filesystem: String, pub root: String, pub endpoint: String, pub client: HttpClient, pub loader: AzureStorageLoader, pub signer: AzureStorageSigner, } impl Debug for AzdlsCore { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { f.debug_struct("AzdlsCore") .field("filesystem", &self.filesystem) .field("root", &self.root) .field("endpoint", &self.endpoint) .finish_non_exhaustive() } } impl AzdlsCore { async fn load_credential(&self) -> Result { let cred = self .loader .load() .await .map_err(new_request_credential_error)?; if let Some(cred) = cred { Ok(cred) } else { Err(Error::new( ErrorKind::ConfigInvalid, "no valid credential found", )) } } pub async fn sign(&self, req: &mut Request) -> Result<()> { let cred = self.load_credential().await?; // Insert x-ms-version header for normal requests. req.headers_mut().insert( HeaderName::from_static(X_MS_VERSION), // 2022-11-02 is the version supported by Azurite V3 and // used by Azure Portal, We use this version to make // sure most our developer happy. // // In the future, we could allow users to configure this value. HeaderValue::from_static("2022-11-02"), ); self.signer.sign(req, &cred).map_err(new_request_sign_error) } #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } } impl AzdlsCore { pub async fn azdls_read(&self, path: &str, range: BytesRange) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", self.endpoint, self.filesystem, percent_encode_path(&p) ); let mut req = Request::get(&url); if !range.is_full() { req = req.header(http::header::RANGE, range.to_header()); } let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.fetch(req).await } /// resource should be one of `file` or `directory` /// /// ref: https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/create pub fn azdls_create_request( &self, path: &str, resource: &str, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/{}/{}?resource={resource}", self.endpoint, self.filesystem, percent_encode_path(&p) ); let mut req = Request::put(&url); // Content length must be 0 for create request. req = req.header(CONTENT_LENGTH, 0); if let Some(ty) = args.content_type() { req = req.header(CONTENT_TYPE, ty) } if let Some(pos) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, pos) } if args.if_not_exists() { req = req.header(IF_NONE_MATCH, "*") } if let Some(v) = args.if_none_match() { req = req.header(IF_NONE_MATCH, v) } // Set body let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn azdls_rename(&self, from: &str, to: &str) -> Result> { let source = build_abs_path(&self.root, from); let target = build_abs_path(&self.root, to); let url = format!( "{}/{}/{}", self.endpoint, self.filesystem, percent_encode_path(&target) ); let mut req = Request::put(&url) .header( X_MS_RENAME_SOURCE, format!("/{}/{}", self.filesystem, percent_encode_path(&source)), ) .header(CONTENT_LENGTH, 0) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } /// ref: https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/update pub fn azdls_update_request( &self, path: &str, size: Option, position: u64, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); // - close: Make this is the final action to this file. // - flush: Flush the file directly. let url = format!( "{}/{}/{}?action=append&close=true&flush=true&position={}", self.endpoint, self.filesystem, percent_encode_path(&p), position ); let mut req = Request::patch(&url); if let Some(size) = size { req = req.header(CONTENT_LENGTH, size) } // Set body let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn azdls_get_properties(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/{}/{}?action=getStatus", self.endpoint, self.filesystem, percent_encode_path(&p) ); let req = Request::head(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.send(req).await } pub async fn azdls_delete(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/{}/{}", self.endpoint, self.filesystem, percent_encode_path(&p) ); let req = Request::delete(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azdls_list( &self, path: &str, continuation: &str, limit: Option, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let mut url = format!( "{}/{}?resource=filesystem&recursive=false", self.endpoint, self.filesystem ); if !p.is_empty() { write!(url, "&directory={}", percent_encode_path(&p)) .expect("write into string must succeed"); } if let Some(limit) = limit { write!(url, "&maxResults={limit}").expect("write into string must succeed"); } if !continuation.is_empty() { write!(url, "&continuation={}", percent_encode_path(continuation)) .expect("write into string must succeed"); } let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azdls_ensure_parent_path(&self, path: &str) -> Result>> { let abs_target_path = path.trim_end_matches('/').to_string(); let abs_target_path = abs_target_path.as_str(); let mut parts: Vec<&str> = abs_target_path .split('/') .filter(|x| !x.is_empty()) .collect(); if !parts.is_empty() { parts.pop(); } if !parts.is_empty() { let parent_path = parts.join("/"); let mut req = self.azdls_create_request( &parent_path, "directory", &OpWrite::default(), Buffer::new(), )?; self.sign(&mut req).await?; Ok(Some(self.send(req).await?)) } else { Ok(None) } } } opendal-0.52.0/src/services/azdls/delete.rs000064400000000000000000000026371046102023000167010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct AzdlsDeleter { core: Arc, } impl AzdlsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for AzdlsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.azdls_delete(&path).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/azdls/docs.md000064400000000000000000000041301046102023000163310ustar 00000000000000As known as `abfs`, `azdls` or `azdls`. This service will visit the [ABFS](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-abfs-driver) URI supported by [Azure Data Lake Storage Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction). ## Notes `azdls` is different from `azfile` service which used to visit [Azure File Storage](https://azure.microsoft.com/en-us/services/storage/files/). ## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [x] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work dir for backend. - `filesystem`: Set the filesystem name for backend. - `endpoint`: Set the endpoint for backend. - `account_name`: Set the account_name for backend. - `account_key`: Set the account_key for backend. Refer to public API docs for more information. ## Example ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Azdls; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create azdls backend builder. let mut builder = Azdls::default() // Set the root for azdls, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/path/to/dir") // Set the filesystem name, this is required. .filesystem("test") // Set the endpoint, this is required. // // For examples: // - "https://accountname.dfs.core.windows.net" .endpoint("https://accountname.dfs.core.windows.net") // Set the account_name and account_key. // // OpenDAL will try load credential from the env. // If credential not set and no valid credential in env, OpenDAL will // send request without signing like anonymous user. .account_name("account_name") .account_key("account_key"); // `Accessor` provides the low level APIs, we will use `Operator` normally. let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/azdls/error.rs000064400000000000000000000066511046102023000165700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use bytes::Buf; use http::Response; use http::StatusCode; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// AzdlsError is the error returned by azure dfs service. #[derive(Default, Deserialize)] #[serde(default, rename_all = "PascalCase")] struct AzdlsError { code: String, message: String, query_parameter_name: String, query_parameter_value: String, reason: String, } impl Debug for AzdlsError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("AzdlsError"); de.field("code", &self.code); // replace `\n` to ` ` for better reading. de.field("message", &self.message.replace('\n', " ")); if !self.query_parameter_name.is_empty() { de.field("query_parameter_name", &self.query_parameter_name); } if !self.query_parameter_value.is_empty() { de.field("query_parameter_value", &self.query_parameter_value); } if !self.reason.is_empty() { de.field("reason", &self.reason); } de.finish() } } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::CONFLICT => { (ErrorKind::ConditionNotMatch, false) } StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let mut message = match de::from_reader::<_, AzdlsError>(bs.clone().reader()) { Ok(azdls_err) => format!("{azdls_err:?}"), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; // If there is no body here, fill with error code. if message.is_empty() { if let Some(v) = parts.headers.get("x-ms-error-code") { if let Ok(code) = v.to_str() { message = format!( "{:?}", AzdlsError { code: code.to_string(), ..Default::default() } ) } } } let mut err = Error::new(kind, &message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/azdls/lister.rs000064400000000000000000000124461046102023000167400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use serde::Deserialize; use serde_json::de; use super::core::AzdlsCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct AzdlsLister { core: Arc, path: String, limit: Option, } impl AzdlsLister { pub fn new(core: Arc, path: String, limit: Option) -> Self { Self { core, path, limit } } } impl oio::PageList for AzdlsLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .azdls_list(&self.path, &ctx.token, self.limit) .await?; // azdls will return not found for not-exist path. if resp.status() == http::StatusCode::NOT_FOUND { ctx.done = true; return Ok(()); } if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } // Return self at the first page. if ctx.token.is_empty() && !ctx.done { let e = oio::Entry::new(&self.path, Metadata::new(EntryMode::DIR)); ctx.entries.push_back(e); } // Check whether this list is done. if let Some(value) = resp.headers().get("x-ms-continuation") { let value = value.to_str().map_err(|err| { Error::new(ErrorKind::Unexpected, "header value is not valid string") .set_source(err) })?; ctx.token = value.to_string(); } else { ctx.token = "".to_string(); ctx.done = true; } let bs = resp.into_body(); let output: Output = de::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; for object in output.paths { // Azdls will return `"true"` and `"false"` for is_directory. let mode = if &object.is_directory == "true" { EntryMode::DIR } else { EntryMode::FILE }; let meta = Metadata::new(mode) // Keep fit with ETag header. .with_etag(format!("\"{}\"", &object.etag)) .with_content_length(object.content_length.parse().map_err(|err| { Error::new(ErrorKind::Unexpected, "content length is not valid integer") .set_source(err) })?) .with_last_modified(parse_datetime_from_rfc2822(&object.last_modified)?); let mut path = build_rel_path(&self.core.root, &object.name); if mode.is_dir() { path += "/" }; let de = oio::Entry::new(&path, meta); ctx.entries.push_back(de); } Ok(()) } } /// # Examples /// /// ```json /// {"paths":[{"contentLength":"1977097","etag":"0x8DACF9B0061305F","group":"$superuser","lastModified":"Sat, 26 Nov 2022 10:43:05 GMT","name":"c3b3ef48-7783-4946-81bc-dc07e1728878/d4ea21d7-a533-4011-8b1f-d0e566d63725","owner":"$superuser","permissions":"rw-r-----"}]} /// ``` #[derive(Default, Debug, Deserialize)] #[serde(default)] struct Output { paths: Vec, } #[derive(Default, Debug, Deserialize, PartialEq, Eq)] #[serde(default)] struct Path { #[serde(rename = "contentLength")] content_length: String, #[serde(rename = "etag")] etag: String, /// Azdls will return `"true"` and `"false"` for is_directory. #[serde(rename = "isDirectory")] is_directory: String, #[serde(rename = "lastModified")] last_modified: String, #[serde(rename = "name")] name: String, } #[cfg(test)] mod tests { use bytes::Bytes; use super::*; #[test] fn test_parse_path() { let bs = Bytes::from( r#"{"paths":[{"contentLength":"1977097","etag":"0x8DACF9B0061305F","group":"$superuser","lastModified":"Sat, 26 Nov 2022 10:43:05 GMT","name":"c3b3ef48-7783-4946-81bc-dc07e1728878/d4ea21d7-a533-4011-8b1f-d0e566d63725","owner":"$superuser","permissions":"rw-r-----"}]}"#, ); let out: Output = de::from_slice(&bs).expect("must success"); println!("{out:?}"); assert_eq!( out.paths[0], Path { content_length: "1977097".to_string(), etag: "0x8DACF9B0061305F".to_string(), is_directory: "".to_string(), last_modified: "Sat, 26 Nov 2022 10:43:05 GMT".to_string(), name: "c3b3ef48-7783-4946-81bc-dc07e1728878/d4ea21d7-a533-4011-8b1f-d0e566d63725" .to_string() } ); } } opendal-0.52.0/src/services/azdls/mod.rs000064400000000000000000000022601046102023000162060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-azdls")] mod core; #[cfg(feature = "services-azdls")] mod delete; #[cfg(feature = "services-azdls")] mod error; #[cfg(feature = "services-azdls")] mod lister; #[cfg(feature = "services-azdls")] mod writer; #[cfg(feature = "services-azdls")] mod backend; #[cfg(feature = "services-azdls")] pub use backend::AzdlsBuilder as Azdls; mod config; pub use config::AzdlsConfig; opendal-0.52.0/src/services/azdls/writer.rs000064400000000000000000000074051046102023000167510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::AzdlsCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type AzdlsWriters = TwoWays, oio::AppendWriter>; pub struct AzdlsWriter { core: Arc, op: OpWrite, path: String, } impl AzdlsWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { AzdlsWriter { core, op, path } } } impl oio::OneShotWrite for AzdlsWriter { async fn write_once(&self, bs: Buffer) -> Result { let mut req = self.core .azdls_create_request(&self.path, "file", &self.op, Buffer::new())?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => {} _ => { return Err(parse_error(resp).with_operation("Backend::azdls_create_request")); } } let mut req = self .core .azdls_update_request(&self.path, Some(bs.len() as u64), 0, bs)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::ACCEPTED => Ok(Metadata::default()), _ => Err(parse_error(resp).with_operation("Backend::azdls_update_request")), } } } impl oio::AppendWrite for AzdlsWriter { async fn offset(&self) -> Result { let resp = self.core.azdls_get_properties(&self.path).await?; let status = resp.status(); let headers = resp.headers(); match status { StatusCode::OK => Ok(parse_content_length(headers)?.unwrap_or_default()), StatusCode::NOT_FOUND => Ok(0), _ => Err(parse_error(resp)), } } async fn append(&self, offset: u64, size: u64, body: Buffer) -> Result { if offset == 0 { let mut req = self.core .azdls_create_request(&self.path, "file", &self.op, Buffer::new())?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => {} _ => { return Err(parse_error(resp).with_operation("Backend::azdls_create_request")); } } } let mut req = self .core .azdls_update_request(&self.path, Some(size), offset, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::ACCEPTED => Ok(Metadata::default()), _ => Err(parse_error(resp).with_operation("Backend::azdls_update_request")), } } } opendal-0.52.0/src/services/azfile/backend.rs000064400000000000000000000307661046102023000171670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use log::debug; use reqsign::AzureStorageConfig; use reqsign::AzureStorageLoader; use reqsign::AzureStorageSigner; use super::core::AzfileCore; use super::delete::AzfileDeleter; use super::error::parse_error; use super::lister::AzfileLister; use super::writer::AzfileWriter; use super::writer::AzfileWriters; use crate::raw::*; use crate::services::AzfileConfig; use crate::*; /// Default endpoint of Azure File services. const DEFAULT_AZFILE_ENDPOINT_SUFFIX: &str = "file.core.windows.net"; impl Configurator for AzfileConfig { type Builder = AzfileBuilder; fn into_builder(self) -> Self::Builder { AzfileBuilder { config: self, http_client: None, } } } /// Azure File services support. #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct AzfileBuilder { config: AzfileConfig, http_client: Option, } impl Debug for AzfileBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("AzfileBuilder"); ds.field("config", &self.config); ds.finish() } } impl AzfileBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set endpoint of this backend. pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()); } self } /// Set account_name of this backend. /// /// - If account_name is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn account_name(mut self, account_name: &str) -> Self { if !account_name.is_empty() { self.config.account_name = Some(account_name.to_string()); } self } /// Set account_key of this backend. /// /// - If account_key is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn account_key(mut self, account_key: &str) -> Self { if !account_key.is_empty() { self.config.account_key = Some(account_key.to_string()); } self } /// Set file share name of this backend. /// /// # Notes /// You can find more about from: pub fn share_name(mut self, share_name: &str) -> Self { if !share_name.is_empty() { self.config.share_name = share_name.to_string(); } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for AzfileBuilder { const SCHEME: Scheme = Scheme::Azfile; type Config = AzfileConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let endpoint = match &self.config.endpoint { Some(endpoint) => Ok(endpoint.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Azfile)), }?; debug!("backend use endpoint {}", &endpoint); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Azfile) })? }; let account_name_option = self .config .account_name .clone() .or_else(|| infer_account_name_from_endpoint(endpoint.as_str())); let account_name = match account_name_option { Some(account_name) => Ok(account_name), None => Err( Error::new(ErrorKind::ConfigInvalid, "account_name is empty") .with_operation("Builder::build") .with_context("service", Scheme::Azfile), ), }?; let config_loader = AzureStorageConfig { account_name: Some(account_name), account_key: self.config.account_key.clone(), sas_token: self.config.sas_token.clone(), ..Default::default() }; let cred_loader = AzureStorageLoader::new(config_loader); let signer = AzureStorageSigner::new(); Ok(AzfileBackend { core: Arc::new(AzfileCore { root, endpoint, loader: cred_loader, client, signer, share_name: self.config.share_name.clone(), }), }) } } fn infer_account_name_from_endpoint(endpoint: &str) -> Option { let endpoint: &str = endpoint .strip_prefix("http://") .or_else(|| endpoint.strip_prefix("https://")) .unwrap_or(endpoint); let mut parts = endpoint.splitn(2, '.'); let account_name = parts.next(); let endpoint_suffix = parts .next() .unwrap_or_default() .trim_end_matches('/') .to_lowercase(); if endpoint_suffix == DEFAULT_AZFILE_ENDPOINT_SUFFIX { account_name.map(|s| s.to_string()) } else { None } } /// Backend for azfile services. #[derive(Debug, Clone)] pub struct AzfileBackend { core: Arc, } impl Access for AzfileBackend { type Reader = HttpBody; type Writer = AzfileWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Azfile) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, read: true, write: true, create_dir: true, delete: true, rename: true, list: true, list_has_etag: true, list_has_last_modified: true, list_has_content_length: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { self.core.ensure_parent_dir_exists(path).await?; let resp = self.core.azfile_create_dir(path).await?; let status = resp.status(); match status { StatusCode::CREATED => Ok(RpCreateDir::default()), _ => { // we cannot just check status code because 409 Conflict has two meaning: // 1. If a directory by the same name is being deleted when Create Directory is called, the server returns status code 409 (Conflict) // 2. If a directory or file with the same name already exists, the operation fails with status code 409 (Conflict). // but we just need case 2 (already exists) // ref: https://learn.microsoft.com/en-us/rest/api/storageservices/create-directory if resp .headers() .get("x-ms-error-code") .map(|value| value.to_str().unwrap_or("")) .unwrap_or_else(|| "") == "ResourceAlreadyExists" { Ok(RpCreateDir::default()) } else { Err(parse_error(resp)) } } } } async fn stat(&self, path: &str, _: OpStat) -> Result { let resp = if path.ends_with('/') { self.core.azfile_get_directory_properties(path).await? } else { self.core.azfile_get_file_properties(path).await? }; let status = resp.status(); match status { StatusCode::OK => { let meta = parse_into_metadata(path, resp.headers())?; Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.azfile_read(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { self.core.ensure_parent_dir_exists(path).await?; let w = AzfileWriter::new(self.core.clone(), args.clone(), path.to_string()); let w = if args.append() { AzfileWriters::Two(oio::AppendWriter::new(w)) } else { AzfileWriters::One(oio::OneShotWriter::new(w)) }; Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(AzfileDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = AzfileLister::new(self.core.clone(), path.to_string(), args.limit()); Ok((RpList::default(), oio::PageLister::new(l))) } async fn rename(&self, from: &str, to: &str, _: OpRename) -> Result { self.core.ensure_parent_dir_exists(to).await?; let resp = self.core.azfile_rename(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpRename::default()), _ => Err(parse_error(resp)), } } } #[cfg(test)] mod tests { use super::*; #[test] fn test_infer_storage_name_from_endpoint() { let cases = vec![ ( "test infer account name from endpoint", "https://account.file.core.windows.net", "account", ), ( "test infer account name from endpoint with trailing slash", "https://account.file.core.windows.net/", "account", ), ]; for (desc, endpoint, expected) in cases { let account_name = infer_account_name_from_endpoint(endpoint); assert_eq!(account_name, Some(expected.to_string()), "{}", desc); } } } opendal-0.52.0/src/services/azfile/config.rs000064400000000000000000000040011046102023000170240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Azure File services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] pub struct AzfileConfig { /// The root path for azfile. pub root: Option, /// The endpoint for azfile. pub endpoint: Option, /// The share name for azfile. pub share_name: String, /// The account name for azfile. pub account_name: Option, /// The account key for azfile. pub account_key: Option, /// The sas token for azfile. pub sas_token: Option, } impl Debug for AzfileConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("AzfileConfig"); ds.field("root", &self.root); ds.field("share_name", &self.share_name); ds.field("endpoint", &self.endpoint); if self.account_name.is_some() { ds.field("account_name", &""); } if self.account_key.is_some() { ds.field("account_key", &""); } if self.sas_token.is_some() { ds.field("sas_token", &""); } ds.finish() } } opendal-0.52.0/src/services/azfile/core.rs000064400000000000000000000303311046102023000165140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::VecDeque; use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::Write; use http::header::CONTENT_DISPOSITION; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::header::RANGE; use http::HeaderName; use http::HeaderValue; use http::Request; use http::Response; use http::StatusCode; use reqsign::AzureStorageCredential; use reqsign::AzureStorageLoader; use reqsign::AzureStorageSigner; use super::error::parse_error; use crate::raw::*; use crate::*; const X_MS_VERSION: &str = "x-ms-version"; const X_MS_WRITE: &str = "x-ms-write"; const X_MS_FILE_RENAME_SOURCE: &str = "x-ms-file-rename-source"; const X_MS_CONTENT_LENGTH: &str = "x-ms-content-length"; const X_MS_TYPE: &str = "x-ms-type"; const X_MS_FILE_RENAME_REPLACE_IF_EXISTS: &str = "x-ms-file-rename-replace-if-exists"; pub struct AzfileCore { pub root: String, pub endpoint: String, pub share_name: String, pub client: HttpClient, pub loader: AzureStorageLoader, pub signer: AzureStorageSigner, } impl Debug for AzfileCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("AzfileCore") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("share_name", &self.share_name) .finish_non_exhaustive() } } impl AzfileCore { async fn load_credential(&self) -> Result { let cred = self .loader .load() .await .map_err(new_request_credential_error)?; if let Some(cred) = cred { Ok(cred) } else { Err(Error::new( ErrorKind::ConfigInvalid, "no valid credential found", )) } } pub async fn sign(&self, req: &mut Request) -> Result<()> { let cred = self.load_credential().await?; // Insert x-ms-version header for normal requests. req.headers_mut().insert( HeaderName::from_static(X_MS_VERSION), // consistent with azdls and azblob HeaderValue::from_static("2022-11-02"), ); self.signer.sign(req, &cred).map_err(new_request_sign_error) } #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } pub async fn azfile_read(&self, path: &str, range: BytesRange) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", self.endpoint, self.share_name, percent_encode_path(&p) ); let mut req = Request::get(&url); if !range.is_full() { req = req.header(RANGE, range.to_header()); } let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.fetch(req).await } pub async fn azfile_create_file( &self, path: &str, size: usize, args: &OpWrite, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_start_matches('/') .to_string(); let url = format!( "{}/{}/{}", self.endpoint, self.share_name, percent_encode_path(&p) ); let mut req = Request::put(&url); // x-ms-content-length specifies the maximum size for the file, up to 4 tebibytes (TiB) // https://learn.microsoft.com/en-us/rest/api/storageservices/create-file req = req.header(X_MS_CONTENT_LENGTH, size); req = req.header(X_MS_TYPE, "file"); // Content length must be 0 for create request. req = req.header(CONTENT_LENGTH, 0); if let Some(ty) = args.content_type() { req = req.header(CONTENT_TYPE, ty); } if let Some(pos) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, pos); } let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azfile_update( &self, path: &str, size: u64, position: u64, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_start_matches('/') .to_string(); let url = format!( "{}/{}/{}?comp=range", self.endpoint, self.share_name, percent_encode_path(&p) ); let mut req = Request::put(&url); req = req.header(CONTENT_LENGTH, size); req = req.header(X_MS_WRITE, "update"); req = req.header( RANGE, BytesRange::from(position..position + size).to_header(), ); let mut req = req.body(body).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azfile_get_file_properties(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", self.endpoint, self.share_name, percent_encode_path(&p) ); let req = Request::head(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azfile_get_directory_properties(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}?restype=directory", self.endpoint, self.share_name, percent_encode_path(&p) ); let req = Request::head(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azfile_rename(&self, path: &str, new_path: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_start_matches('/') .to_string(); let new_p = build_abs_path(&self.root, new_path) .trim_start_matches('/') .to_string(); let url = if path.ends_with('/') { format!( "{}/{}/{}?restype=directory&comp=rename", self.endpoint, self.share_name, percent_encode_path(&new_p) ) } else { format!( "{}/{}/{}?comp=rename", self.endpoint, self.share_name, percent_encode_path(&new_p) ) }; let mut req = Request::put(&url); req = req.header(CONTENT_LENGTH, 0); // x-ms-file-rename-source specifies the file or directory to be renamed. // the value must be a URL style path // the official document does not mention the URL style path // find the solution from the community FAQ and implementation of the Java-SDK // ref: https://learn.microsoft.com/en-us/answers/questions/799611/azure-file-service-rest-api(rename)?page=1 let source_url = format!( "{}/{}/{}", self.endpoint, self.share_name, percent_encode_path(&p) ); req = req.header(X_MS_FILE_RENAME_SOURCE, &source_url); req = req.header(X_MS_FILE_RENAME_REPLACE_IF_EXISTS, "true"); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azfile_create_dir(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_start_matches('/') .to_string(); let url = format!( "{}/{}/{}?restype=directory", self.endpoint, self.share_name, percent_encode_path(&p) ); let mut req = Request::put(&url); req = req.header(CONTENT_LENGTH, 0); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azfile_delete_file(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_start_matches('/') .to_string(); let url = format!( "{}/{}/{}", self.endpoint, self.share_name, percent_encode_path(&p) ); let req = Request::delete(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azfile_delete_dir(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_start_matches('/') .to_string(); let url = format!( "{}/{}/{}?restype=directory", self.endpoint, self.share_name, percent_encode_path(&p) ); let req = Request::delete(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn azfile_list( &self, path: &str, limit: &Option, continuation: &String, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_start_matches('/') .to_string(); let mut url = format!( "{}/{}/{}?restype=directory&comp=list&include=Timestamps,ETag", self.endpoint, self.share_name, percent_encode_path(&p), ); if !continuation.is_empty() { write!(url, "&marker={}", &continuation).expect("write into string must succeed"); } if let Some(limit) = limit { write!(url, "&maxresults={}", limit).expect("write into string must succeed"); } let req = Request::get(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn ensure_parent_dir_exists(&self, path: &str) -> Result<()> { let mut dirs = VecDeque::default(); // azure file service does not support recursive directory creation let mut p = path; while p != "/" { p = get_parent(p); dirs.push_front(p); } let mut pop_dir_count = dirs.len(); for dir in dirs.iter().rev() { let resp = self.azfile_get_directory_properties(dir).await?; if resp.status() == StatusCode::NOT_FOUND { pop_dir_count -= 1; continue; } break; } for dir in dirs.iter().skip(pop_dir_count) { let resp = self.azfile_create_dir(dir).await?; if resp.status() == StatusCode::CREATED { continue; } if resp .headers() .get("x-ms-error-code") .map(|value| value.to_str().unwrap_or("")) .unwrap_or_else(|| "") == "ResourceAlreadyExists" { continue; } return Err(parse_error(resp)); } Ok(()) } } opendal-0.52.0/src/services/azfile/delete.rs000064400000000000000000000030451046102023000170300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct AzfileDeleter { core: Arc, } impl AzfileDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for AzfileDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = if path.ends_with('/') { self.core.azfile_delete_dir(&path).await? } else { self.core.azfile_delete_file(&path).await? }; let status = resp.status(); match status { StatusCode::ACCEPTED | StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/azfile/docs.md000064400000000000000000000031471046102023000164750ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [x] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work dir for backend. - `endpoint`: Set the endpoint for backend. - `account_name`: Set the account_name for backend. - `account_key`: Set the account_key for backend. - `share_name`: Set the share_name for backend. Refer to public API docs for more information. ## Example ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Azfile; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create azfile backend builder. let mut builder = Azfile::default() // Set the root for azfile, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/path/to/dir") // Set the filesystem name, this is required. .share_name("test") // Set the endpoint, this is required. // // For examples: // - "https://accountname.file.core.windows.net" .endpoint("https://accountname.file.core.windows.net") // Set the account_name and account_key. // // OpenDAL will try load credential from the env. // If credential not set and no valid credential in env, OpenDAL will // send request without signing like anonymous user. .account_name("account_name") .account_key("account_key"); // `Accessor` provides the low level APIs, we will use `Operator` normally. let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/azfile/error.rs000064400000000000000000000066671046102023000167340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use bytes::Buf; use http::Response; use http::StatusCode; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// AzfileError is the error returned by azure file service. #[derive(Default, Deserialize)] #[serde(default, rename_all = "PascalCase")] struct AzfileError { code: String, message: String, query_parameter_name: String, query_parameter_value: String, reason: String, } impl Debug for AzfileError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("AzfileError"); de.field("code", &self.code); // replace `\n` to ` ` for better reading. de.field("message", &self.message.replace('\n', " ")); if !self.query_parameter_name.is_empty() { de.field("query_parameter_name", &self.query_parameter_name); } if !self.query_parameter_value.is_empty() { de.field("query_parameter_value", &self.query_parameter_value); } if !self.reason.is_empty() { de.field("reason", &self.reason); } de.finish() } } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::NOT_MODIFIED => { (ErrorKind::ConditionNotMatch, false) } StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let mut message = match de::from_reader::<_, AzfileError>(bs.clone().reader()) { Ok(azfile_err) => format!("{azfile_err:?}"), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; // If there is no body here, fill with error code. if message.is_empty() { if let Some(v) = parts.headers.get("x-ms-error-code") { if let Ok(code) = v.to_str() { message = format!( "{:?}", AzfileError { code: code.to_string(), ..Default::default() } ) } } } let mut err = Error::new(kind, &message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/azfile/lister.rs000064400000000000000000000152701046102023000170730ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use quick_xml::de; use serde::Deserialize; use super::core::AzfileCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct AzfileLister { core: Arc, path: String, limit: Option, } impl AzfileLister { pub fn new(core: Arc, path: String, limit: Option) -> Self { Self { core, path, limit } } } impl oio::PageList for AzfileLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .azfile_list(&self.path, &self.limit, &ctx.token) .await?; let status = resp.status(); if status != StatusCode::OK { if status == StatusCode::NOT_FOUND { ctx.done = true; return Ok(()); } return Err(parse_error(resp)); } // Return self at the first page. if ctx.token.is_empty() && !ctx.done { let e = oio::Entry::new(&self.path, Metadata::new(EntryMode::DIR)); ctx.entries.push_back(e); } let bs = resp.into_body(); let results: EnumerationResults = de::from_reader(bs.reader()).map_err(new_xml_deserialize_error)?; if results.next_marker.is_empty() { ctx.done = true; } else { ctx.token = results.next_marker; } for file in results.entries.file { let meta = Metadata::new(EntryMode::FILE) .with_etag(file.properties.etag) .with_content_length(file.properties.content_length.unwrap_or(0)) .with_last_modified(parse_datetime_from_rfc2822(&file.properties.last_modified)?); let path = self.path.clone().trim_start_matches('/').to_string() + &file.name; ctx.entries.push_back(oio::Entry::new(&path, meta)); } for dir in results.entries.directory { let meta = Metadata::new(EntryMode::DIR) .with_etag(dir.properties.etag) .with_last_modified(parse_datetime_from_rfc2822(&dir.properties.last_modified)?); let path = self.path.clone().trim_start_matches('/').to_string() + &dir.name + "/"; ctx.entries.push_back(oio::Entry::new(&path, meta)); } Ok(()) } } #[derive(Debug, Deserialize, PartialEq)] #[serde(rename_all = "PascalCase")] struct EnumerationResults { marker: Option, prefix: Option, max_results: Option, directory_id: Option, entries: Entries, #[serde(default)] next_marker: String, } #[derive(Debug, Deserialize, PartialEq)] #[serde(rename_all = "PascalCase")] struct Entries { #[serde(default)] file: Vec, #[serde(default)] directory: Vec, } #[derive(Debug, Deserialize, PartialEq)] #[serde(rename_all = "PascalCase")] struct File { #[serde(rename = "FileId")] file_id: String, name: String, properties: Properties, } #[derive(Debug, Deserialize, PartialEq)] #[serde(rename_all = "PascalCase")] struct Directory { #[serde(rename = "FileId")] file_id: String, name: String, properties: Properties, } #[derive(Debug, Deserialize, PartialEq)] #[serde(rename_all = "PascalCase")] struct Properties { #[serde(rename = "Content-Length")] content_length: Option, #[serde(rename = "CreationTime")] creation_time: String, #[serde(rename = "LastAccessTime")] last_access_time: String, #[serde(rename = "LastWriteTime")] last_write_time: String, #[serde(rename = "ChangeTime")] change_time: String, #[serde(rename = "Last-Modified")] last_modified: String, #[serde(rename = "Etag")] etag: String, } #[cfg(test)] mod tests { use quick_xml::de::from_str; use super::*; #[test] fn test_parse_list_result() { let xml = r#" string-value string-value 100 directory-id Rust By Example.pdf 13835093239654252544 5832374 2023-09-25T12:43:05.8483527Z 2023-09-25T12:43:05.8483527Z 2023-09-25T12:43:08.6337775Z 2023-09-25T12:43:08.6337775Z Mon, 25 Sep 2023 12:43:08 GMT \"0x8DBBDC4F8AC4AEF\" test_list_rich_dir 12105702186650959872 2023-10-15T12:03:40.7194774Z 2023-10-15T12:03:40.7194774Z 2023-10-15T12:03:40.7194774Z 2023-10-15T12:03:40.7194774Z Sun, 15 Oct 2023 12:03:40 GMT \"0x8DBCD76C58C3E96\" "#; let results: EnumerationResults = from_str(xml).unwrap(); assert_eq!(results.entries.file[0].name, "Rust By Example.pdf"); assert_eq!( results.entries.file[0].properties.etag, "\\\"0x8DBBDC4F8AC4AEF\\\"" ); assert_eq!(results.entries.directory[0].name, "test_list_rich_dir"); assert_eq!( results.entries.directory[0].properties.etag, "\\\"0x8DBCD76C58C3E96\\\"" ); } } opendal-0.52.0/src/services/azfile/mod.rs000064400000000000000000000022731046102023000163470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-azfile")] mod core; #[cfg(feature = "services-azfile")] mod delete; #[cfg(feature = "services-azfile")] mod error; #[cfg(feature = "services-azfile")] mod lister; #[cfg(feature = "services-azfile")] mod writer; #[cfg(feature = "services-azfile")] mod backend; #[cfg(feature = "services-azfile")] pub use backend::AzfileBuilder as Azfile; mod config; pub use config::AzfileConfig; opendal-0.52.0/src/services/azfile/writer.rs000064400000000000000000000056011046102023000171020ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::AzfileCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type AzfileWriters = TwoWays, oio::AppendWriter>; pub struct AzfileWriter { core: Arc, op: OpWrite, path: String, } impl AzfileWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { AzfileWriter { core, op, path } } } impl oio::OneShotWrite for AzfileWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self .core .azfile_create_file(&self.path, bs.len(), &self.op) .await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => {} _ => { return Err(parse_error(resp).with_operation("Backend::azfile_create_file")); } } let resp = self .core .azfile_update(&self.path, bs.len() as u64, 0, bs) .await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => Ok(Metadata::default()), _ => Err(parse_error(resp).with_operation("Backend::azfile_update")), } } } impl oio::AppendWrite for AzfileWriter { async fn offset(&self) -> Result { let resp = self.core.azfile_get_file_properties(&self.path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(parse_content_length(resp.headers())?.unwrap_or_default()), _ => Err(parse_error(resp)), } } async fn append(&self, offset: u64, size: u64, body: Buffer) -> Result { let resp = self .core .azfile_update(&self.path, size, offset, body) .await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => Ok(Metadata::default()), _ => Err(parse_error(resp).with_operation("Backend::azfile_update")), } } } opendal-0.52.0/src/services/b2/backend.rs000064400000000000000000000360051046102023000162100ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::Request; use http::Response; use http::StatusCode; use log::debug; use tokio::sync::RwLock; use super::core::constants; use super::core::parse_file_info; use super::core::B2Core; use super::core::B2Signer; use super::core::ListFileNamesResponse; use super::delete::B2Deleter; use super::error::parse_error; use super::lister::B2Lister; use super::writer::B2Writer; use super::writer::B2Writers; use crate::raw::*; use crate::services::B2Config; use crate::*; impl Configurator for B2Config { type Builder = B2Builder; fn into_builder(self) -> Self::Builder { B2Builder { config: self, http_client: None, } } } /// [b2](https://www.backblaze.com/cloud-storage) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct B2Builder { config: B2Config, http_client: Option, } impl Debug for B2Builder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("B2Builder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl B2Builder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// application_key_id of this backend. pub fn application_key_id(mut self, application_key_id: &str) -> Self { self.config.application_key_id = if application_key_id.is_empty() { None } else { Some(application_key_id.to_string()) }; self } /// application_key of this backend. pub fn application_key(mut self, application_key: &str) -> Self { self.config.application_key = if application_key.is_empty() { None } else { Some(application_key.to_string()) }; self } /// Set bucket name of this backend. /// You can find it in pub fn bucket(mut self, bucket: &str) -> Self { self.config.bucket = bucket.to_string(); self } /// Set bucket id of this backend. /// You can find it in pub fn bucket_id(mut self, bucket_id: &str) -> Self { self.config.bucket_id = bucket_id.to_string(); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for B2Builder { const SCHEME: Scheme = Scheme::B2; type Config = B2Config; /// Builds the backend and returns the result of B2Backend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle bucket. if self.config.bucket.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "bucket is empty") .with_operation("Builder::build") .with_context("service", Scheme::B2)); } debug!("backend use bucket {}", &self.config.bucket); // Handle bucket_id. if self.config.bucket_id.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "bucket_id is empty") .with_operation("Builder::build") .with_context("service", Scheme::B2)); } debug!("backend bucket_id {}", &self.config.bucket_id); let application_key_id = match &self.config.application_key_id { Some(application_key_id) => Ok(application_key_id.clone()), None => Err( Error::new(ErrorKind::ConfigInvalid, "application_key_id is empty") .with_operation("Builder::build") .with_context("service", Scheme::B2), ), }?; let application_key = match &self.config.application_key { Some(key_id) => Ok(key_id.clone()), None => Err( Error::new(ErrorKind::ConfigInvalid, "application_key is empty") .with_operation("Builder::build") .with_context("service", Scheme::B2), ), }?; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::B2) })? }; let signer = B2Signer { application_key_id, application_key, ..Default::default() }; Ok(B2Backend { core: Arc::new(B2Core { signer: Arc::new(RwLock::new(signer)), root, bucket: self.config.bucket.clone(), bucket_id: self.config.bucket_id.clone(), client, }), }) } } /// Backend for b2 services. #[derive(Debug, Clone)] pub struct B2Backend { core: Arc, } impl Access for B2Backend { type Reader = HttpBody; type Writer = B2Writers; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::B2) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_content_md5: true, stat_has_content_type: true, read: true, write: true, write_can_empty: true, write_can_multi: true, write_with_content_type: true, // The min multipart size of b2 is 5 MiB. // // ref: write_multi_min_size: Some(5 * 1024 * 1024), // The max multipart size of b2 is 5 Gb. // // ref: write_multi_max_size: if cfg!(target_pointer_width = "64") { Some(5 * 1024 * 1024 * 1024) } else { Some(usize::MAX) }, delete: true, copy: true, list: true, list_with_limit: true, list_with_start_after: true, list_with_recursive: true, list_has_content_length: true, list_has_content_md5: true, list_has_content_type: true, presign: true, presign_read: true, presign_write: true, presign_stat: true, shared: true, ..Default::default() }); am.into() } /// B2 have a get_file_info api required a file_id field, but field_id need call list api, list api also return file info /// So we call list api to get file info async fn stat(&self, path: &str, _args: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let delimiter = if path.ends_with('/') { Some("/") } else { None }; let resp = self .core .list_file_names(Some(path), delimiter, None, None) .await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: ListFileNamesResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; if resp.files.is_empty() { return Err(Error::new(ErrorKind::NotFound, "no such file or directory")); } let meta = parse_file_info(&resp.files[0]); Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self .core .download_file_by_name(path, args.range(), &args) .await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let concurrent = args.concurrent(); let executor = args.executor().cloned(); let writer = B2Writer::new(self.core.clone(), path, args); let w = oio::MultipartWriter::new(writer, executor, concurrent); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(B2Deleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { Ok(( RpList::default(), oio::PageLister::new(B2Lister::new( self.core.clone(), path, args.recursive(), args.limit(), args.start_after(), )), )) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self .core .list_file_names(Some(from), None, None, None) .await?; let status = resp.status(); let source_file_id = match status { StatusCode::OK => { let bs = resp.into_body(); let resp: ListFileNamesResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; if resp.files.is_empty() { return Err(Error::new(ErrorKind::NotFound, "no such file or directory")); } let file_id = resp.files[0].clone().file_id; Ok(file_id) } _ => Err(parse_error(resp)), }?; let Some(source_file_id) = source_file_id else { return Err(Error::new(ErrorKind::IsADirectory, "is a directory")); }; let resp = self.core.copy_file(source_file_id, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn presign(&self, path: &str, args: OpPresign) -> Result { match args.operation() { PresignOperation::Stat(_) => { let resp = self .core .get_download_authorization(path, args.expire()) .await?; let path = build_abs_path(&self.core.root, path); let auth_info = self.core.get_auth_info().await?; let url = format!( "{}/file/{}/{}?Authorization={}", auth_info.download_url, self.core.bucket, path, resp.authorization_token ); let req = Request::get(url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; // We don't need this request anymore, consume let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } PresignOperation::Read(_) => { let resp = self .core .get_download_authorization(path, args.expire()) .await?; let path = build_abs_path(&self.core.root, path); let auth_info = self.core.get_auth_info().await?; let url = format!( "{}/file/{}/{}?Authorization={}", auth_info.download_url, self.core.bucket, path, resp.authorization_token ); let req = Request::get(url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; // We don't need this request anymore, consume let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } PresignOperation::Write(_) => { let resp = self.core.get_upload_url().await?; let mut req = Request::post(&resp.upload_url); req = req.header(http::header::AUTHORIZATION, resp.authorization_token); req = req.header("X-Bz-File-Name", build_abs_path(&self.core.root, path)); req = req.header(http::header::CONTENT_TYPE, "b2/x-auto"); req = req.header(constants::X_BZ_CONTENT_SHA1, "do_not_verify"); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; // We don't need this request anymore, consume it directly. let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } } } } opendal-0.52.0/src/services/b2/config.rs000064400000000000000000000041701046102023000160640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for backblaze b2 services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct B2Config { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, /// keyID of this backend. /// /// - If application_key_id is set, we will take user's input first. /// - If not, we will try to load it from environment. pub application_key_id: Option, /// applicationKey of this backend. /// /// - If application_key is set, we will take user's input first. /// - If not, we will try to load it from environment. pub application_key: Option, /// bucket of this backend. /// /// required. pub bucket: String, /// bucket id of this backend. /// /// required. pub bucket_id: String, } impl Debug for B2Config { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("B2Config"); d.field("root", &self.root) .field("application_key_id", &self.application_key_id) .field("bucket_id", &self.bucket_id) .field("bucket", &self.bucket); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/b2/core.rs000064400000000000000000000511421046102023000155500ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use std::time::Duration; use bytes::Buf; use chrono::DateTime; use chrono::Utc; use http::header; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use serde::Serialize; use tokio::sync::RwLock; use self::constants::X_BZ_CONTENT_SHA1; use self::constants::X_BZ_FILE_NAME; use super::core::constants::X_BZ_PART_NUMBER; use super::error::parse_error; use crate::raw::*; use crate::*; pub(super) mod constants { pub const X_BZ_FILE_NAME: &str = "X-Bz-File-Name"; pub const X_BZ_CONTENT_SHA1: &str = "X-Bz-Content-Sha1"; pub const X_BZ_PART_NUMBER: &str = "X-Bz-Part-Number"; } /// Core of [b2](https://www.backblaze.com/cloud-storage) services support. #[derive(Clone)] pub struct B2Core { pub signer: Arc>, /// The root of this core. pub root: String, /// The bucket name of this backend. pub bucket: String, /// The bucket id of this backend. pub bucket_id: String, pub client: HttpClient, } impl Debug for B2Core { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("bucket", &self.bucket) .field("bucket_id", &self.bucket_id) .finish_non_exhaustive() } } impl B2Core { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } /// [b2_authorize_account](https://www.backblaze.com/apidocs/b2-authorize-account) pub async fn get_auth_info(&self) -> Result { { let signer = self.signer.read().await; if !signer.auth_info.authorization_token.is_empty() && signer.auth_info.expires_in > Utc::now() { let auth_info = signer.auth_info.clone(); return Ok(auth_info); } } { let mut signer = self.signer.write().await; let req = Request::get("https://api.backblazeb2.com/b2api/v2/b2_authorize_account") .header( header::AUTHORIZATION, format_authorization_by_basic( &signer.application_key_id, &signer.application_key, )?, ) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let token: AuthorizeAccountResponse = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; signer.auth_info = AuthInfo { authorization_token: token.authorization_token.clone(), api_url: token.api_url.clone(), download_url: token.download_url.clone(), // This authorization token is valid for at most 24 hours. expires_in: Utc::now() + chrono::TimeDelta::try_hours(20).expect("20 hours must be valid"), }; } _ => { return Err(parse_error(resp)); } } Ok(signer.auth_info.clone()) } } } impl B2Core { pub async fn download_file_by_name( &self, path: &str, range: BytesRange, _args: &OpRead, ) -> Result> { let path = build_abs_path(&self.root, path); let auth_info = self.get_auth_info().await?; // Construct headers to add to the request let url = format!( "{}/file/{}/{}", auth_info.download_url, self.bucket, percent_encode_path(&path) ); let mut req = Request::get(&url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub(super) async fn get_upload_url(&self) -> Result { let auth_info = self.get_auth_info().await?; let url = format!( "{}/b2api/v2/b2_get_upload_url?bucketId={}", auth_info.api_url, self.bucket_id ); let mut req = Request::get(&url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let resp = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; Ok(resp) } _ => Err(parse_error(resp)), } } pub async fn get_download_authorization( &self, path: &str, expire: Duration, ) -> Result { let path = build_abs_path(&self.root, path); let auth_info = self.get_auth_info().await?; // Construct headers to add to the request let url = format!( "{}/b2api/v2/b2_get_download_authorization", auth_info.api_url ); let mut req = Request::post(&url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); let body = GetDownloadAuthorizationRequest { bucket_id: self.bucket_id.clone(), file_name_prefix: path, valid_duration_in_seconds: expire.as_secs(), }; let body = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let body = bytes::Bytes::from(body); let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let resp = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; Ok(resp) } _ => Err(parse_error(resp)), } } pub async fn upload_file( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let resp = self.get_upload_url().await?; let p = build_abs_path(&self.root, path); let mut req = Request::post(resp.upload_url); req = req.header(X_BZ_FILE_NAME, percent_encode_path(&p)); req = req.header(header::AUTHORIZATION, resp.authorization_token); req = req.header(X_BZ_CONTENT_SHA1, "do_not_verify"); if let Some(size) = size { req = req.header(header::CONTENT_LENGTH, size.to_string()) } if let Some(mime) = args.content_type() { req = req.header(header::CONTENT_TYPE, mime) } else { req = req.header(header::CONTENT_TYPE, "b2/x-auto") } if let Some(pos) = args.content_disposition() { req = req.header(header::CONTENT_DISPOSITION, pos) } // Set body let req = req.body(body).map_err(new_request_build_error)?; self.send(req).await } pub async fn start_large_file(&self, path: &str, args: &OpWrite) -> Result> { let p = build_abs_path(&self.root, path); let auth_info = self.get_auth_info().await?; let url = format!("{}/b2api/v2/b2_start_large_file", auth_info.api_url); let mut req = Request::post(&url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); let mut start_large_file_request = StartLargeFileRequest { bucket_id: self.bucket_id.clone(), file_name: percent_encode_path(&p), content_type: "b2/x-auto".to_owned(), }; if let Some(mime) = args.content_type() { mime.clone_into(&mut start_large_file_request.content_type) } let body = serde_json::to_vec(&start_large_file_request).map_err(new_json_serialize_error)?; let body = bytes::Bytes::from(body); let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req).await } pub async fn get_upload_part_url(&self, file_id: &str) -> Result { let auth_info = self.get_auth_info().await?; let url = format!( "{}/b2api/v2/b2_get_upload_part_url?fileId={}", auth_info.api_url, file_id ); let mut req = Request::get(&url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let resp = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; Ok(resp) } _ => Err(parse_error(resp)), } } pub async fn upload_part( &self, file_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result> { let resp = self.get_upload_part_url(file_id).await?; let mut req = Request::post(resp.upload_url); req = req.header(X_BZ_PART_NUMBER, part_number.to_string()); req = req.header(header::CONTENT_LENGTH, size.to_string()); req = req.header(header::AUTHORIZATION, resp.authorization_token); req = req.header(X_BZ_CONTENT_SHA1, "do_not_verify"); // Set body let req = req.body(body).map_err(new_request_build_error)?; self.send(req).await } pub async fn finish_large_file( &self, file_id: &str, part_sha1_array: Vec, ) -> Result> { let auth_info = self.get_auth_info().await?; let url = format!("{}/b2api/v2/b2_finish_large_file", auth_info.api_url); let mut req = Request::post(&url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); let body = serde_json::to_vec(&FinishLargeFileRequest { file_id: file_id.to_owned(), part_sha1_array, }) .map_err(new_json_serialize_error)?; let body = bytes::Bytes::from(body); // Set body let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req).await } pub async fn cancel_large_file(&self, file_id: &str) -> Result> { let auth_info = self.get_auth_info().await?; let url = format!("{}/b2api/v2/b2_cancel_large_file", auth_info.api_url); let mut req = Request::post(&url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); let body = serde_json::to_vec(&CancelLargeFileRequest { file_id: file_id.to_owned(), }) .map_err(new_json_serialize_error)?; let body = bytes::Bytes::from(body); // Set body let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req).await } pub async fn list_file_names( &self, prefix: Option<&str>, delimiter: Option<&str>, limit: Option, start_after: Option, ) -> Result> { let auth_info = self.get_auth_info().await?; let mut url = format!( "{}/b2api/v2/b2_list_file_names?bucketId={}", auth_info.api_url, self.bucket_id ); if let Some(prefix) = prefix { let prefix = build_abs_path(&self.root, prefix); url.push_str(&format!("&prefix={}", percent_encode_path(&prefix))); } if let Some(limit) = limit { url.push_str(&format!("&maxFileCount={}", limit)); } if let Some(start_after) = start_after { url.push_str(&format!( "&startFileName={}", percent_encode_path(&start_after) )); } if let Some(delimiter) = delimiter { url.push_str(&format!("&delimiter={}", delimiter)); } let mut req = Request::get(&url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn copy_file(&self, source_file_id: String, to: &str) -> Result> { let to = build_abs_path(&self.root, to); let auth_info = self.get_auth_info().await?; let url = format!("{}/b2api/v2/b2_copy_file", auth_info.api_url); let mut req = Request::post(url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); let body = CopyFileRequest { source_file_id, file_name: to, }; let body = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let body = bytes::Bytes::from(body); // Set body let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req).await } pub async fn hide_file(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let auth_info = self.get_auth_info().await?; let url = format!("{}/b2api/v2/b2_hide_file", auth_info.api_url); let mut req = Request::post(url); req = req.header(header::AUTHORIZATION, auth_info.authorization_token); let body = HideFileRequest { bucket_id: self.bucket_id.clone(), file_name: path.to_string(), }; let body = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let body = bytes::Bytes::from(body); // Set body let req = req .body(Buffer::from(body)) .map_err(new_request_build_error)?; self.send(req).await } } #[derive(Clone)] pub struct B2Signer { /// The application_key_id of this core. pub application_key_id: String, /// The application_key of this core. pub application_key: String, pub auth_info: AuthInfo, } #[derive(Clone)] pub struct AuthInfo { pub authorization_token: String, /// The base URL to use for all API calls except for uploading and downloading files. pub api_url: String, /// The base URL to use for downloading files. pub download_url: String, pub expires_in: DateTime, } impl Default for B2Signer { fn default() -> Self { B2Signer { application_key: String::new(), application_key_id: String::new(), auth_info: AuthInfo { authorization_token: String::new(), api_url: String::new(), download_url: String::new(), expires_in: DateTime::::MIN_UTC, }, } } } /// Request of [b2_start_large_file](https://www.backblaze.com/apidocs/b2-start-large-file). #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub struct StartLargeFileRequest { pub bucket_id: String, pub file_name: String, pub content_type: String, } /// Response of [b2_start_large_file](https://www.backblaze.com/apidocs/b2-start-large-file). #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct StartLargeFileResponse { pub file_id: String, } /// Response of [b2_authorize_account](https://www.backblaze.com/apidocs/b2-authorize-account). #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct AuthorizeAccountResponse { /// An authorization token to use with all calls, other than b2_authorize_account, that need an Authorization header. This authorization token is valid for at most 24 hours. /// So we should call b2_authorize_account every 24 hours. pub authorization_token: String, pub api_url: String, pub download_url: String, } /// Response of [b2_get_upload_url](https://www.backblaze.com/apidocs/b2-get-upload-url). #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct GetUploadUrlResponse { /// The authorizationToken that must be used when uploading files to this bucket. /// This token is valid for 24 hours or until the uploadUrl endpoint rejects an upload, see b2_upload_file pub authorization_token: String, pub upload_url: String, } /// Response of [b2_get_upload_url](https://www.backblaze.com/apidocs/b2-get-upload-part-url). #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct GetUploadPartUrlResponse { /// The authorizationToken that must be used when uploading files to this bucket. /// This token is valid for 24 hours or until the uploadUrl endpoint rejects an upload, see b2_upload_file pub authorization_token: String, pub upload_url: String, } /// Response of [b2_upload_part](https://www.backblaze.com/apidocs/b2-upload-part). #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct UploadPartResponse { pub content_sha1: String, } /// Response of [b2_finish_large_file](https://www.backblaze.com/apidocs/b2-finish-large-file). #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub struct FinishLargeFileRequest { pub file_id: String, pub part_sha1_array: Vec, } /// Response of [b2_cancel_large_file](https://www.backblaze.com/apidocs/b2-cancel-large-file). #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub struct CancelLargeFileRequest { pub file_id: String, } /// Response of [list_file_names](https://www.backblaze.com/apidocs/b2-list-file-names). #[derive(Debug, Clone, Deserialize)] #[serde(rename_all = "camelCase")] pub struct ListFileNamesResponse { pub files: Vec, pub next_file_name: Option, } #[derive(Debug, Clone, Deserialize)] #[serde(rename_all = "camelCase")] pub struct File { pub file_id: Option, pub content_length: u64, pub content_md5: Option, pub content_type: Option, pub file_name: String, } pub(super) fn parse_file_info(file: &File) -> Metadata { if file.file_name.ends_with('/') { return Metadata::new(EntryMode::DIR); } let mut metadata = Metadata::new(EntryMode::FILE); metadata.set_content_length(file.content_length); if let Some(content_md5) = &file.content_md5 { metadata.set_content_md5(content_md5); } if let Some(content_type) = &file.content_type { metadata.set_content_type(content_type); } metadata } #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub struct CopyFileRequest { pub source_file_id: String, pub file_name: String, } #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub struct HideFileRequest { pub bucket_id: String, pub file_name: String, } #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub struct GetDownloadAuthorizationRequest { pub bucket_id: String, pub file_name_prefix: String, pub valid_duration_in_seconds: u64, } #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct GetDownloadAuthorizationResponse { pub authorization_token: String, } opendal-0.52.0/src/services/b2/delete.rs000064400000000000000000000032131046102023000160560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct B2Deleter { core: Arc, } impl B2Deleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for B2Deleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.hide_file(&path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => { let err = parse_error(resp); match err.kind() { ErrorKind::NotFound => Ok(()), // Representative deleted ErrorKind::AlreadyExists => Ok(()), _ => Err(err), } } } } } opendal-0.52.0/src/services/b2/docs.md000064400000000000000000000021421046102023000155200ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [x] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `key_id`: B2 application key keyID - `application_key` B2 application key applicationKey - `bucket` B2 bucket name - `bucket_id` B2 bucket_id You can refer to [`B2Builder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::B2; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = B2::default() // set the storage bucket for OpenDAL .root("/") // set the key_id for OpenDAL .application_key_id("xxxxxxxxxx") // set the key_id for OpenDAL .application_key("xxxxxxxxxx") // set the bucket name for OpenDAL .bucket("opendal") // set the bucket_id for OpenDAL .bucket_id("xxxxxxxxxxxxx"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/b2/error.rs000064400000000000000000000103471046102023000157530ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use serde::Deserialize; use crate::raw::*; use crate::*; /// the error response of b2 #[derive(Default, Debug, Deserialize)] #[allow(dead_code)] struct B2Error { status: u32, code: String, message: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (mut kind, mut retryable) = match parts.status.as_u16() { 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), 304 | 412 => (ErrorKind::ConditionNotMatch, false), // Service b2 could return 403, show the authorization error 401 => (ErrorKind::PermissionDenied, true), 429 => (ErrorKind::RateLimited, true), 500 | 502 | 503 | 504 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, b2_err) = serde_json::from_reader::<_, B2Error>(bs.clone().reader()) .map(|b2_err| (format!("{b2_err:?}"), Some(b2_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); if let Some(b2_err) = b2_err { (kind, retryable) = parse_b2_error_code(b2_err.code.as_str()).unwrap_or((kind, retryable)); }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } /// Returns the `Error kind` of this code and whether the error is retryable. pub(crate) fn parse_b2_error_code(code: &str) -> Option<(ErrorKind, bool)> { match code { "already_hidden" => Some((ErrorKind::AlreadyExists, false)), "no_such_file" => Some((ErrorKind::NotFound, false)), _ => None, } } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[test] fn test_parse_b2_error_code() { let code = "already_hidden"; assert_eq!( parse_b2_error_code(code), Some((crate::ErrorKind::AlreadyExists, false)) ); let code = "no_such_file"; assert_eq!( parse_b2_error_code(code), Some((crate::ErrorKind::NotFound, false)) ); let code = "not_found"; assert_eq!(parse_b2_error_code(code), None); } #[tokio::test] async fn test_parse_error() { let err_res = vec![ ( r#"{"status": 403, "code": "access_denied", "message":"The provided customer-managed encryption key is wrong."}"#, ErrorKind::PermissionDenied, StatusCode::FORBIDDEN, ), ( r#"{"status": 404, "code": "not_found", "message":"File is not in B2 Cloud Storage."}"#, ErrorKind::NotFound, StatusCode::NOT_FOUND, ), ( r#"{"status": 401, "code": "bad_auth_token", "message":"The auth token used is not valid. Call b2_authorize_account again to either get a new one, or an error message describing the problem."}"#, ErrorKind::PermissionDenied, StatusCode::UNAUTHORIZED, ), ]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/b2/lister.rs000064400000000000000000000061731046102023000161260ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use super::core::parse_file_info; use super::core::B2Core; use super::core::ListFileNamesResponse; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct B2Lister { core: Arc, path: String, delimiter: Option<&'static str>, limit: Option, /// B2 starts listing **after** this specified key start_after: Option, } impl B2Lister { pub fn new( core: Arc, path: &str, recursive: bool, limit: Option, start_after: Option<&str>, ) -> Self { let delimiter = if recursive { None } else { Some("/") }; Self { core, path: path.to_string(), delimiter, limit, start_after: start_after.map(String::from), } } } impl oio::PageList for B2Lister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .list_file_names( Some(&self.path), self.delimiter, self.limit, if ctx.token.is_empty() { self.start_after .as_ref() .map(|v| build_abs_path(&self.core.root, v)) } else { Some(ctx.token.clone()) }, ) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let output: ListFileNamesResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; if let Some(token) = output.next_file_name { ctx.token = token; } else { ctx.done = true; } for file in output.files { if let Some(start_after) = self.start_after.clone() { if build_abs_path(&self.core.root, &start_after) == file.file_name { continue; } } let file_name = file.file_name.clone(); let metadata = parse_file_info(&file); ctx.entries.push_back(oio::Entry::new( &build_rel_path(&self.core.root, &file_name), metadata, )) } Ok(()) } } opendal-0.52.0/src/services/b2/mod.rs000064400000000000000000000022231046102023000153730ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-b2")] mod core; #[cfg(feature = "services-b2")] mod delete; #[cfg(feature = "services-b2")] mod error; #[cfg(feature = "services-b2")] mod lister; #[cfg(feature = "services-b2")] mod writer; #[cfg(feature = "services-b2")] mod backend; #[cfg(feature = "services-b2")] pub use backend::B2Builder as B2; mod config; pub use config::B2Config; opendal-0.52.0/src/services/b2/writer.rs000064400000000000000000000103671046102023000161400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use super::core::B2Core; use super::core::StartLargeFileResponse; use super::core::UploadPartResponse; use super::error::parse_error; use crate::raw::*; use crate::*; pub type B2Writers = oio::MultipartWriter; pub struct B2Writer { core: Arc, op: OpWrite, path: String, } impl B2Writer { pub fn new(core: Arc, path: &str, op: OpWrite) -> Self { B2Writer { core, path: path.to_string(), op, } } } impl oio::MultipartWrite for B2Writer { async fn write_once(&self, size: u64, body: Buffer) -> Result { let resp = self .core .upload_file(&self.path, Some(size), &self.op, body) .await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn initiate_part(&self) -> Result { let resp = self.core.start_large_file(&self.path, &self.op).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let result: StartLargeFileResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; Ok(result.file_id) } _ => Err(parse_error(resp)), } } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result { // B2 requires part number must between [1..=10000] let part_number = part_number + 1; let resp = self .core .upload_part(upload_id, part_number, size, body) .await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let result: UploadPartResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; Ok(oio::MultipartPart { etag: result.content_sha1, part_number, checksum: None, }) } _ => Err(parse_error(resp)), } } async fn complete_part( &self, upload_id: &str, parts: &[oio::MultipartPart], ) -> Result { let part_sha1_array = parts .iter() .map(|p| { let binding = p.etag.clone(); let sha1 = binding.strip_prefix("unverified:"); let Some(sha1) = sha1 else { return "".to_string(); }; sha1.to_string() }) .collect(); let resp = self .core .finish_large_file(upload_id, part_sha1_array) .await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn abort_part(&self, upload_id: &str) -> Result<()> { let resp = self.core.cancel_large_file(upload_id).await?; match resp.status() { // b2 returns code 200 if abort succeeds. StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/cacache/backend.rs000064400000000000000000000101571046102023000172540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::str; use cacache; use crate::raw::adapters::kv; use crate::raw::Access; use crate::services::CacacheConfig; use crate::Builder; use crate::Error; use crate::ErrorKind; use crate::Scheme; use crate::*; impl Configurator for CacacheConfig { type Builder = CacacheBuilder; fn into_builder(self) -> Self::Builder { CacacheBuilder { config: self } } } /// cacache service support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct CacacheBuilder { config: CacacheConfig, } impl CacacheBuilder { /// Set the path to the cacache data directory. Will create if not exists. pub fn datadir(mut self, path: &str) -> Self { self.config.datadir = Some(path.into()); self } } impl Builder for CacacheBuilder { const SCHEME: Scheme = Scheme::Cacache; type Config = CacacheConfig; fn build(self) -> Result { let datadir_path = self.config.datadir.ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "datadir is required but not set") .with_context("service", Scheme::Cacache) })?; Ok(CacacheBackend::new(Adapter { datadir: datadir_path, })) } } /// Backend for cacache services. pub type CacacheBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { datadir: String, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.field("path", &self.datadir); ds.finish() } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Cacache, &self.datadir, Capability { read: true, write: true, delete: true, blocking: true, shared: false, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let result = cacache::read(&self.datadir, path) .await .map_err(parse_error)?; Ok(Some(Buffer::from(result))) } fn blocking_get(&self, path: &str) -> Result> { let result = cacache::read_sync(&self.datadir, path).map_err(parse_error)?; Ok(Some(Buffer::from(result))) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { cacache::write(&self.datadir, path, value.to_vec()) .await .map_err(parse_error)?; Ok(()) } fn blocking_set(&self, path: &str, value: Buffer) -> Result<()> { cacache::write_sync(&self.datadir, path, value.to_vec()).map_err(parse_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { cacache::remove(&self.datadir, path) .await .map_err(parse_error)?; Ok(()) } fn blocking_delete(&self, path: &str) -> Result<()> { cacache::remove_sync(&self.datadir, path).map_err(parse_error)?; Ok(()) } } fn parse_error(err: cacache::Error) -> Error { let kind = match err { cacache::Error::EntryNotFound(_, _) => ErrorKind::NotFound, _ => ErrorKind::Unexpected, }; Error::new(kind, "error from cacache").set_source(err) } opendal-0.52.0/src/services/cacache/config.rs000064400000000000000000000020731046102023000171300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// cacache service support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] pub struct CacacheConfig { /// That path to the cacache data directory. pub datadir: Option, } opendal-0.52.0/src/services/cacache/docs.md000064400000000000000000000011741046102023000165700ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] list - [ ] ~~presign~~ - [x] blocking ## Configuration - `datadir`: Set the path to the cacache data directory You can refer to [`CacacheBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Cacache; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Cacache::default().datadir("/tmp/opendal/cacache"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/cacache/mod.rs000064400000000000000000000017211046102023000164410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-cacache")] mod backend; #[cfg(feature = "services-cacache")] pub use backend::CacacheBuilder as Cacache; mod config; pub use config::CacacheConfig; opendal-0.52.0/src/services/chainsafe/backend.rs000064400000000000000000000175711046102023000176350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::Response; use http::StatusCode; use log::debug; use super::core::parse_info; use super::core::ChainsafeCore; use super::core::ObjectInfoResponse; use super::delete::ChainsafeDeleter; use super::error::parse_error; use super::lister::ChainsafeLister; use super::writer::ChainsafeWriter; use super::writer::ChainsafeWriters; use crate::raw::*; use crate::services::ChainsafeConfig; use crate::*; impl Configurator for ChainsafeConfig { type Builder = ChainsafeBuilder; fn into_builder(self) -> Self::Builder { ChainsafeBuilder { config: self, http_client: None, } } } /// [chainsafe](https://storage.chainsafe.io/) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct ChainsafeBuilder { config: ChainsafeConfig, http_client: Option, } impl Debug for ChainsafeBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("ChainsafeBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl ChainsafeBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// api_key of this backend. /// /// required. pub fn api_key(mut self, api_key: &str) -> Self { self.config.api_key = if api_key.is_empty() { None } else { Some(api_key.to_string()) }; self } /// Set bucket_id name of this backend. pub fn bucket_id(mut self, bucket_id: &str) -> Self { self.config.bucket_id = bucket_id.to_string(); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for ChainsafeBuilder { const SCHEME: Scheme = Scheme::Chainsafe; type Config = ChainsafeConfig; /// Builds the backend and returns the result of ChainsafeBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle bucket_id. if self.config.bucket_id.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "bucket_id is empty") .with_operation("Builder::build") .with_context("service", Scheme::Chainsafe)); } debug!("backend use bucket_id {}", &self.config.bucket_id); let api_key = match &self.config.api_key { Some(api_key) => Ok(api_key.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "api_key is empty") .with_operation("Builder::build") .with_context("service", Scheme::Chainsafe)), }?; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Chainsafe) })? }; Ok(ChainsafeBackend { core: Arc::new(ChainsafeCore { root, api_key, bucket_id: self.config.bucket_id.clone(), client, }), }) } } /// Backend for Chainsafe services. #[derive(Debug, Clone)] pub struct ChainsafeBackend { core: Arc, } impl Access for ChainsafeBackend { type Reader = HttpBody; type Writer = ChainsafeWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Chainsafe) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_content_type: true, read: true, create_dir: true, write: true, write_can_empty: true, delete: true, list: true, list_has_content_length: true, list_has_content_type: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let resp = self.core.create_dir(path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCreateDir::default()), // Allow 409 when creating a existing dir StatusCode::CONFLICT => Ok(RpCreateDir::default()), _ => Err(parse_error(resp)), } } async fn stat(&self, path: &str, _args: OpStat) -> Result { let resp = self.core.object_info(path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let output: ObjectInfoResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; Ok(RpStat::new(parse_info(output.content))) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.download_object(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = ChainsafeWriter::new(self.core.clone(), args, path.to_string()); let w = oio::OneShotWriter::new(writer); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(ChainsafeDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, _args: OpList) -> Result<(RpList, Self::Lister)> { let l = ChainsafeLister::new(self.core.clone(), path); Ok((RpList::default(), oio::PageLister::new(l))) } } opendal-0.52.0/src/services/chainsafe/config.rs000064400000000000000000000031331046102023000175000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Chainsafe services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct ChainsafeConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, /// api_key of this backend. pub api_key: Option, /// bucket_id of this backend. /// /// required. pub bucket_id: String, } impl Debug for ChainsafeConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("ChainsafeConfig"); d.field("root", &self.root) .field("bucket_id", &self.bucket_id); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/chainsafe/core.rs000064400000000000000000000153301046102023000171650ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use bytes::Bytes; use http::header; use http::Request; use http::Response; use serde::Deserialize; use serde_json::json; use crate::raw::*; use crate::*; /// Core of [chainsafe](https://storage.chainsafe.io/) services support. #[derive(Clone)] pub struct ChainsafeCore { /// The root of this core. pub root: String, /// The api_key of this core. pub api_key: String, /// The bucket id of this backend. pub bucket_id: String, pub client: HttpClient, } impl Debug for ChainsafeCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("bucket_id", &self.bucket_id) .finish_non_exhaustive() } } impl ChainsafeCore { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } } impl ChainsafeCore { pub async fn download_object( &self, path: &str, range: BytesRange, ) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://api.chainsafe.io/api/v1/bucket/{}/download", self.bucket_id ); let req_body = &json!({ "path": path, }); let body = Buffer::from(Bytes::from(req_body.to_string())); let req = Request::post(url) .header( header::AUTHORIZATION, format_authorization_by_bearer(&self.api_key)?, ) .header(header::RANGE, range.to_header()) .header(header::CONTENT_TYPE, "application/json") .body(body) .map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn object_info(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://api.chainsafe.io/api/v1/bucket/{}/file", self.bucket_id ); let req_body = &json!({ "path": path, }); let body = Buffer::from(Bytes::from(req_body.to_string())); let req = Request::post(url) .header( header::AUTHORIZATION, format_authorization_by_bearer(&self.api_key)?, ) .header(header::CONTENT_TYPE, "application/json") .body(body) .map_err(new_request_build_error)?; self.send(req).await } pub async fn delete_object(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://api.chainsafe.io/api/v1/bucket/{}/rm", self.bucket_id ); let req_body = &json!({ "paths": vec![path], }); let body = Buffer::from(Bytes::from(req_body.to_string())); let req = Request::post(url) .header( header::AUTHORIZATION, format_authorization_by_bearer(&self.api_key)?, ) .header(header::CONTENT_TYPE, "application/json") .body(body) .map_err(new_request_build_error)?; self.send(req).await } pub async fn upload_object(&self, path: &str, bs: Buffer) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://api.chainsafe.io/api/v1/bucket/{}/upload", self.bucket_id ); let file_part = FormDataPart::new("file").content(bs); let multipart = Multipart::new() .part(file_part) .part(FormDataPart::new("path").content(path)); let req = Request::post(url).header( header::AUTHORIZATION, format_authorization_by_bearer(&self.api_key)?, ); let req = multipart.apply(req)?; self.send(req).await } pub async fn list_objects(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://api.chainsafe.io/api/v1/bucket/{}/ls", self.bucket_id ); let req_body = &json!({ "path": path, }); let body = Buffer::from(Bytes::from(req_body.to_string())); let req = Request::post(url) .header( header::AUTHORIZATION, format_authorization_by_bearer(&self.api_key)?, ) .header(header::CONTENT_TYPE, "application/json") .body(body) .map_err(new_request_build_error)?; self.send(req).await } pub async fn create_dir(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://api.chainsafe.io/api/v1/bucket/{}/mkdir", self.bucket_id ); let req_body = &json!({ "path": path, }); let body = Buffer::from(Bytes::from(req_body.to_string())); let req = Request::post(url) .header( header::AUTHORIZATION, format_authorization_by_bearer(&self.api_key)?, ) .header(header::CONTENT_TYPE, "application/json") .body(body) .map_err(new_request_build_error)?; self.send(req).await } } #[derive(Debug, Deserialize)] pub struct Info { pub name: String, pub content_type: String, pub size: u64, } #[derive(Deserialize)] pub struct ObjectInfoResponse { pub content: Info, } pub(super) fn parse_info(info: Info) -> Metadata { let mode = if info.content_type == "application/chainsafe-files-directory" { EntryMode::DIR } else { EntryMode::FILE }; let mut md = Metadata::new(mode); md.set_content_length(info.size) .set_content_type(&info.content_type); md } opendal-0.52.0/src/services/chainsafe/delete.rs000064400000000000000000000030061046102023000174740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct ChainsafeDeleter { core: Arc, } impl ChainsafeDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for ChainsafeDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.delete_object(&path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), // Allow 404 when deleting a non-existing object StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/chainsafe/docs.md000064400000000000000000000016011046102023000171350ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `bucket_id` Chainsafe bucket_id - `api_key` Chainsafe api_key You can refer to [`ChainsafeBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Chainsafe; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Chainsafe::default() // set the storage root for OpenDAL .root("/") // set the bucket_id for OpenDAL .bucket_id("opendal") // set the api_key for OpenDAL .api_key("xxxxxxxxxxxxx"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/chainsafe/error.rs000064400000000000000000000062171046102023000173720ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use serde::Deserialize; use crate::raw::*; use crate::*; #[derive(Default, Debug, Deserialize)] #[allow(dead_code)] struct ChainsafeError { error: ChainsafeSubError, } #[derive(Default, Debug, Deserialize)] #[allow(dead_code)] struct ChainsafeSubError { code: i64, message: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status.as_u16() { 401 | 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), 304 | 412 => (ErrorKind::ConditionNotMatch, false), // https://github.com/apache/opendal/issues/4146 // https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/423 // We should retry it when we get 423 error. 423 => (ErrorKind::RateLimited, true), // Service like Upyun could return 499 error with a message like: // Client Disconnect, we should retry it. 499 => (ErrorKind::Unexpected, true), 500 | 502 | 503 | 504 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, _chainsafe_err) = serde_json::from_reader::<_, ChainsafeError>(bs.clone().reader()) .map(|chainsafe_err| (format!("{chainsafe_err:?}"), Some(chainsafe_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[test] fn test_parse_error() { let err_res = vec![( r#"{ "error": { "code": 404, "message": "path:test4, file doesn't exist" } }"#, ErrorKind::NotFound, StatusCode::NOT_FOUND, )]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/chainsafe/lister.rs000064400000000000000000000042741046102023000175440ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use super::core::parse_info; use super::core::ChainsafeCore; use super::core::Info; use super::error::parse_error; use crate::raw::oio::Entry; use crate::raw::*; use crate::*; pub struct ChainsafeLister { core: Arc, path: String, } impl ChainsafeLister { pub fn new(core: Arc, path: &str) -> Self { Self { core, path: path.to_string(), } } } impl oio::PageList for ChainsafeLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self.core.list_objects(&self.path).await?; match resp.status() { StatusCode::OK => { let bs = resp.into_body(); let output: Vec = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; for info in output { let mut path = build_abs_path(&normalize_root(&self.path), &info.name); let md = parse_info(info); if md.mode() == EntryMode::DIR { path = format!("{}/", path); } ctx.entries.push_back(Entry::new(&path, md)); } ctx.done = true; Ok(()) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/chainsafe/mod.rs000064400000000000000000000023301046102023000170100ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-chainsafe")] mod core; #[cfg(feature = "services-chainsafe")] mod delete; #[cfg(feature = "services-chainsafe")] mod error; #[cfg(feature = "services-chainsafe")] mod lister; #[cfg(feature = "services-chainsafe")] mod writer; #[cfg(feature = "services-chainsafe")] mod backend; #[cfg(feature = "services-chainsafe")] pub use backend::ChainsafeBuilder as Chainsafe; mod config; pub use config::ChainsafeConfig; opendal-0.52.0/src/services/chainsafe/writer.rs000064400000000000000000000031761046102023000175560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::ChainsafeCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type ChainsafeWriters = oio::OneShotWriter; pub struct ChainsafeWriter { core: Arc, _op: OpWrite, path: String, } impl ChainsafeWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { ChainsafeWriter { core, _op: op, path, } } } impl oio::OneShotWrite for ChainsafeWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self.core.upload_object(&self.path, bs).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/cloudflare_kv/backend.rs000064400000000000000000000246121046102023000205260ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use bytes::Buf; use http::header; use http::Request; use http::StatusCode; use serde::Deserialize; use super::error::parse_error; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::CloudflareKvConfig; use crate::ErrorKind; use crate::*; impl Configurator for CloudflareKvConfig { type Builder = CloudflareKvBuilder; fn into_builder(self) -> Self::Builder { CloudflareKvBuilder { config: self, http_client: None, } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct CloudflareKvBuilder { config: CloudflareKvConfig, /// The HTTP client used to communicate with CloudFlare. http_client: Option, } impl Debug for CloudflareKvBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("CloudFlareKvBuilder") .field("config", &self.config) .finish() } } impl CloudflareKvBuilder { /// Set the token used to authenticate with CloudFlare. pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()) } self } /// Set the account ID used to authenticate with CloudFlare. pub fn account_id(mut self, account_id: &str) -> Self { if !account_id.is_empty() { self.config.account_id = Some(account_id.to_string()) } self } /// Set the namespace ID. pub fn namespace_id(mut self, namespace_id: &str) -> Self { if !namespace_id.is_empty() { self.config.namespace_id = Some(namespace_id.to_string()) } self } /// Set the root within this backend. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } } impl Builder for CloudflareKvBuilder { const SCHEME: Scheme = Scheme::CloudflareKv; type Config = CloudflareKvConfig; fn build(self) -> Result { let authorization = match &self.config.token { Some(token) => format_authorization_by_bearer(token)?, None => return Err(Error::new(ErrorKind::ConfigInvalid, "token is required")), }; let Some(account_id) = self.config.account_id.clone() else { return Err(Error::new( ErrorKind::ConfigInvalid, "account_id is required", )); }; let Some(namespace_id) = self.config.namespace_id.clone() else { return Err(Error::new( ErrorKind::ConfigInvalid, "namespace_id is required", )); }; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::CloudflareKv) })? }; let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); let url_prefix = format!( r"https://api.cloudflare.com/client/v4/accounts/{}/storage/kv/namespaces/{}", account_id, namespace_id ); Ok(CloudflareKvBackend::new(Adapter { authorization, account_id, namespace_id, client, url_prefix, }) .with_normalized_root(root)) } } pub type CloudflareKvBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { authorization: String, account_id: String, namespace_id: String, client: HttpClient, url_prefix: String, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Adapter") .field("account_id", &self.account_id) .field("namespace_id", &self.namespace_id) .finish() } } impl Adapter { fn sign(&self, mut req: Request) -> Result> { req.headers_mut() .insert(header::AUTHORIZATION, self.authorization.parse().unwrap()); Ok(req) } } impl kv::Adapter for Adapter { type Scanner = kv::Scanner; fn info(&self) -> kv::Info { kv::Info::new( Scheme::CloudflareKv, &self.namespace_id, Capability { read: true, write: true, list: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let url = format!("{}/values/{}", self.url_prefix, path); let mut req = Request::get(&url); req = req.header(header::CONTENT_TYPE, "application/json"); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; req = self.sign(req)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Some(resp.into_body())), _ => Err(parse_error(resp)), } } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let url = format!("{}/values/{}", self.url_prefix, path); let req = Request::put(&url); let multipart = Multipart::new(); let multipart = multipart .part(FormDataPart::new("metadata").content(serde_json::Value::Null.to_string())) .part(FormDataPart::new("value").content(value.to_vec())); let mut req = multipart.apply(req)?; req = self.sign(req)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } async fn delete(&self, path: &str) -> Result<()> { let url = format!("{}/values/{}", self.url_prefix, path); let mut req = Request::delete(&url); req = req.header(header::CONTENT_TYPE, "application/json"); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; req = self.sign(req)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } async fn scan(&self, path: &str) -> Result { let mut url = format!("{}/keys", self.url_prefix); if !path.is_empty() { url = format!("{}?prefix={}", url, path); } let mut req = Request::get(&url); req = req.header(header::CONTENT_TYPE, "application/json"); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; req = self.sign(req)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let response: CfKvScanResponse = serde_json::from_reader(body.reader()).map_err(|e| { Error::new( ErrorKind::Unexpected, format!("failed to parse error response: {}", e), ) })?; Ok(Box::new(kv::ScanStdIter::new( response.result.into_iter().map(|r| Ok(r.name)), ))) } _ => Err(parse_error(resp)), } } } #[derive(Debug, Deserialize)] pub(super) struct CfKvResponse { pub(super) errors: Vec, } #[derive(Debug, Deserialize)] pub(super) struct CfKvScanResponse { result: Vec, // According to https://developers.cloudflare.com/api/operations/workers-kv-namespace-list-a-namespace'-s-keys, result_info is used to determine if there are more keys to be listed // result_info: Option, } #[derive(Debug, Deserialize)] struct CfKvScanResult { name: String, } // #[derive(Debug, Deserialize)] // struct CfKvResultInfo { // count: i64, // cursor: String, // } #[derive(Debug, Deserialize)] pub(super) struct CfKvError { pub(super) code: i32, } #[cfg(test)] mod test { use super::*; #[test] fn test_deserialize_scan_json_response() { let json_str = r#"{ "errors": [], "messages": [], "result": [ { "expiration": 1577836800, "metadata": { "someMetadataKey": "someMetadataValue" }, "name": "My-Key" } ], "success": true, "result_info": { "count": 1, "cursor": "6Ck1la0VxJ0djhidm1MdX2FyDGxLKVeeHZZmORS_8XeSuhz9SjIJRaSa2lnsF01tQOHrfTGAP3R5X1Kv5iVUuMbNKhWNAXHOl6ePB0TUL8nw" } }"#; let response: CfKvScanResponse = serde_json::from_slice(json_str.as_bytes()).unwrap(); assert_eq!(response.result.len(), 1); assert_eq!(response.result[0].name, "My-Key"); // assert!(response.result_info.is_some()); // if let Some(result_info) = response.result_info { // assert_eq!(result_info.count, 1); // assert_eq!(result_info.cursor, "6Ck1la0VxJ0djhidm1MdX2FyDGxLKVeeHZZmORS_8XeSuhz9SjIJRaSa2lnsF01tQOHrfTGAP3R5X1Kv5iVUuMbNKhWNAXHOl6ePB0TUL8nw"); // } } #[test] fn test_deserialize_json_response() { let json_str = r#"{ "errors": [], "messages": [], "result": {}, "success": true }"#; let response: CfKvResponse = serde_json::from_slice(json_str.as_bytes()).unwrap(); assert_eq!(response.errors.len(), 0); } } opendal-0.52.0/src/services/cloudflare_kv/config.rs000064400000000000000000000034351046102023000204040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Cloudflare KV Service Support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] pub struct CloudflareKvConfig { /// The token used to authenticate with CloudFlare. pub token: Option, /// The account ID used to authenticate with CloudFlare. Used as URI path parameter. pub account_id: Option, /// The namespace ID. Used as URI path parameter. pub namespace_id: Option, /// Root within this backend. pub root: Option, } impl Debug for CloudflareKvConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("CloudflareKvConfig"); ds.field("root", &self.root); ds.field("account_id", &self.account_id); ds.field("namespace_id", &self.namespace_id); if self.token.is_some() { ds.field("token", &""); } ds.finish() } } opendal-0.52.0/src/services/cloudflare_kv/docs.md000064400000000000000000000006241046102023000200400ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `token`: Set the token of cloudflare api - `account_id`: Set the account identifier of cloudflare - `namespace_id`: Set the namespace identifier of d1 opendal-0.52.0/src/services/cloudflare_kv/error.rs000064400000000000000000000053171046102023000202710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use http::StatusCode; use serde_json::de; use super::backend::CfKvError; use super::backend::CfKvResponse; use crate::raw::*; use crate::*; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (mut kind, mut retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), // Some services (like owncloud) return 403 while file locked. StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, true), // Allowing retry for resource locked. StatusCode::LOCKED => (ErrorKind::Unexpected, true), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, err) = de::from_reader::<_, CfKvResponse>(bs.clone().reader()) .map(|err| (format!("{err:?}"), Some(err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); if let Some(err) = err { (kind, retryable) = parse_cfkv_error_code(err.errors).unwrap_or((kind, retryable)); } let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } pub(super) fn parse_cfkv_error_code(errors: Vec) -> Option<(ErrorKind, bool)> { if errors.is_empty() { return None; } match errors[0].code { // The request is malformed: failed to decode id. 7400 => Some((ErrorKind::Unexpected, false)), // no such column: Xxxx. 7500 => Some((ErrorKind::NotFound, false)), // Authentication error. 10000 => Some((ErrorKind::PermissionDenied, false)), _ => None, } } opendal-0.52.0/src/services/cloudflare_kv/mod.rs000064400000000000000000000020431046102023000177100ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-cloudflare-kv")] mod error; #[cfg(feature = "services-cloudflare-kv")] mod backend; #[cfg(feature = "services-cloudflare-kv")] pub use backend::CloudflareKvBuilder as CloudflareKv; mod config; pub use config::CloudflareKvConfig; opendal-0.52.0/src/services/compfs/backend.rs000064400000000000000000000174431046102023000172010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io::Cursor; use std::sync::Arc; use compio::dispatcher::Dispatcher; use compio::fs::OpenOptions; use super::core::CompfsCore; use super::delete::CompfsDeleter; use super::lister::CompfsLister; use super::reader::CompfsReader; use super::writer::CompfsWriter; use crate::raw::oio::OneShotDeleter; use crate::raw::*; use crate::services::CompfsConfig; use crate::*; impl Configurator for CompfsConfig { type Builder = CompfsBuilder; fn into_builder(self) -> Self::Builder { CompfsBuilder { config: self } } } /// [`compio`]-based file system support. #[derive(Debug, Clone, Default)] pub struct CompfsBuilder { config: CompfsConfig, } impl CompfsBuilder { /// Set root for Compfs pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } } impl Builder for CompfsBuilder { const SCHEME: Scheme = Scheme::Compfs; type Config = CompfsConfig; fn build(self) -> Result { let root = match self.config.root { Some(root) => Ok(root), None => Err(Error::new( ErrorKind::ConfigInvalid, "root is not specified", )), }?; // If root dir does not exist, we must create it. if let Err(e) = std::fs::metadata(&root) { if e.kind() == std::io::ErrorKind::NotFound { std::fs::create_dir_all(&root).map_err(|e| { Error::new(ErrorKind::Unexpected, "create root dir failed") .with_operation("Builder::build") .with_context("root", root.as_str()) .set_source(e) })?; } } let dispatcher = Dispatcher::new().map_err(|_| { Error::new( ErrorKind::Unexpected, "failed to initiate compio dispatcher", ) })?; let core = CompfsCore { root: root.into(), dispatcher, buf_pool: oio::PooledBuf::new(16), }; Ok(CompfsBackend { core: Arc::new(core), }) } } #[derive(Debug)] pub struct CompfsBackend { core: Arc, } impl Access for CompfsBackend { type Reader = CompfsReader; type Writer = CompfsWriter; type Lister = Option; type Deleter = OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Compfs) .set_root(&self.core.root.to_string_lossy()) .set_native_capability(Capability { stat: true, stat_has_last_modified: true, read: true, write: true, write_can_empty: true, write_can_multi: true, create_dir: true, delete: true, list: true, copy: true, rename: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let path = self.core.prepare_path(path); self.core .exec(move || async move { compio::fs::create_dir_all(path).await }) .await?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _: OpStat) -> Result { let path = self.core.prepare_path(path); let meta = self .core .exec(move || async move { compio::fs::metadata(path).await }) .await?; let ty = meta.file_type(); let mode = if ty.is_dir() { EntryMode::DIR } else if ty.is_file() { EntryMode::FILE } else { EntryMode::Unknown }; let last_mod = meta.modified().map_err(new_std_io_error)?.into(); let ret = Metadata::new(mode).with_last_modified(last_mod); Ok(RpStat::new(ret)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), OneShotDeleter::new(CompfsDeleter::new(self.core.clone())), )) } async fn copy(&self, from: &str, to: &str, _: OpCopy) -> Result { let from = self.core.prepare_path(from); let to = self.core.prepare_path(to); self.core .exec(move || async move { let from = OpenOptions::new().read(true).open(from).await?; let to = OpenOptions::new().write(true).create(true).open(to).await?; let (mut from, mut to) = (Cursor::new(from), Cursor::new(to)); compio::io::copy(&mut from, &mut to).await?; Ok(()) }) .await?; Ok(RpCopy::default()) } async fn rename(&self, from: &str, to: &str, _: OpRename) -> Result { let from = self.core.prepare_path(from); let to = self.core.prepare_path(to); self.core .exec(move || async move { compio::fs::rename(from, to).await }) .await?; Ok(RpRename::default()) } async fn read(&self, path: &str, op: OpRead) -> Result<(RpRead, Self::Reader)> { let path = self.core.prepare_path(path); let file = self .core .exec(|| async move { compio::fs::OpenOptions::new().read(true).open(&path).await }) .await?; let r = CompfsReader::new(self.core.clone(), file, op.range()); Ok((RpRead::new(), r)) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let path = self.core.prepare_path(path); let append = args.append(); let file = self .core .exec(move || async move { compio::fs::OpenOptions::new() .create(true) .write(true) .truncate(!append) .open(path) .await }) .await .map(Cursor::new)?; let w = CompfsWriter::new(self.core.clone(), file); Ok((RpWrite::new(), w)) } async fn list(&self, path: &str, _: OpList) -> Result<(RpList, Self::Lister)> { let path = self.core.prepare_path(path); let read_dir = match self .core .exec_blocking(move || std::fs::read_dir(path)) .await? { Ok(rd) => rd, Err(e) => { return if e.kind() == std::io::ErrorKind::NotFound { Ok((RpList::default(), None)) } else { Err(new_std_io_error(e)) }; } }; let lister = CompfsLister::new(self.core.clone(), read_dir); Ok((RpList::default(), Some(lister))) } } opendal-0.52.0/src/services/compfs/config.rs000064400000000000000000000021511046102023000170450ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// compio-based file system support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] pub struct CompfsConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, } opendal-0.52.0/src/services/compfs/core.rs000064400000000000000000000104101046102023000165250ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use std::path::PathBuf; use compio::buf::IoBuf; use compio::dispatcher::Dispatcher; use crate::raw::*; use crate::*; unsafe impl IoBuf for Buffer { fn as_buf_ptr(&self) -> *const u8 { self.current().as_ptr() } fn buf_len(&self) -> usize { self.current().len() } fn buf_capacity(&self) -> usize { // `Bytes` doesn't expose uninitialized capacity, so treat it as the same as `len` self.current().len() } } #[derive(Debug)] pub(super) struct CompfsCore { pub root: PathBuf, pub dispatcher: Dispatcher, pub buf_pool: oio::PooledBuf, } impl CompfsCore { pub fn prepare_path(&self, path: &str) -> PathBuf { self.root.join(path.trim_end_matches('/')) } pub async fn exec(&self, f: Fn) -> crate::Result where Fn: FnOnce() -> Fut + Send + 'static, Fut: Future> + 'static, R: Send + 'static, { self.dispatcher .dispatch(f) .map_err(|_| Error::new(ErrorKind::Unexpected, "compio spawn io task failed"))? .await .map_err(|_| Error::new(ErrorKind::Unexpected, "compio task cancelled"))? .map_err(new_std_io_error) } pub async fn exec_blocking(&self, f: Fn) -> Result where Fn: FnOnce() -> R + Send + 'static, R: Send + 'static, { self.dispatcher .dispatch_blocking(f) .map_err(|_| Error::new(ErrorKind::Unexpected, "compio spawn blocking task failed"))? .await .map_err(|_| Error::new(ErrorKind::Unexpected, "compio task cancelled")) } } // TODO: impl IoVectoredBuf for Buffer // impl IoVectoredBuf for Buffer { // fn as_dyn_bufs(&self) -> impl Iterator {} // // fn owned_iter(self) -> Result>, Self> { // Ok(OwnedIter::new(BufferIter { // current: self.current(), // buf: self, // })) // } // } // #[derive(Debug, Clone)] // struct BufferIter { // buf: Buffer, // current: Bytes, // } // impl IntoInner for BufferIter { // type Inner = Buffer; // // fn into_inner(self) -> Self::Inner { // self.buf // } // } // impl OwnedIterator for BufferIter { // fn next(mut self) -> Result { // let Some(current) = self.buf.next() else { // return Err(self.buf); // }; // self.current = current; // Ok(self) // } // // fn current(&self) -> &dyn IoBuf { // &self.current // } // } #[cfg(test)] mod tests { use bytes::Buf; use bytes::Bytes; use rand::thread_rng; use rand::Rng; use super::*; fn setup_buffer() -> (Buffer, usize, Bytes) { let mut rng = thread_rng(); let bs = (0..100) .map(|_| { let len = rng.gen_range(1..100); let mut buf = vec![0; len]; rng.fill(&mut buf[..]); Bytes::from(buf) }) .collect::>(); let total_size = bs.iter().map(|b| b.len()).sum::(); let total_content = bs.iter().flatten().copied().collect::(); let buf = Buffer::from(bs); (buf, total_size, total_content) } #[test] fn test_io_buf() { let (buf, _len, _bytes) = setup_buffer(); let slice = IoBuf::as_slice(&buf); assert_eq!(slice, buf.current().chunk()) } } opendal-0.52.0/src/services/compfs/delete.rs000064400000000000000000000031011046102023000170360ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use crate::raw::*; use crate::*; use std::sync::Arc; pub struct CompfsDeleter { core: Arc, } impl CompfsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for CompfsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { if path.ends_with('/') { let path = self.core.prepare_path(&path); self.core .exec(move || async move { compio::fs::remove_dir(path).await }) .await?; } else { let path = self.core.prepare_path(&path); self.core .exec(move || async move { compio::fs::remove_file(path).await }) .await?; } Ok(()) } } opendal-0.52.0/src/services/compfs/lister.rs000064400000000000000000000050201046102023000171000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fs::ReadDir; use std::path::Path; use std::sync::Arc; use super::core::CompfsCore; use crate::raw::*; use crate::*; #[derive(Debug)] pub struct CompfsLister { core: Arc, read_dir: Option, } impl CompfsLister { pub(super) fn new(core: Arc, read_dir: ReadDir) -> Self { Self { core, read_dir: Some(read_dir), } } } fn next_entry(read_dir: &mut ReadDir, root: &Path) -> std::io::Result> { let Some(entry) = read_dir.next().transpose()? else { return Ok(None); }; let path = entry.path(); let rel_path = normalize_path( &path .strip_prefix(root) .expect("cannot fail because the prefix is iterated") .to_string_lossy() .replace('\\', "/"), ); let file_type = entry.file_type()?; let entry = if file_type.is_file() { oio::Entry::new(&rel_path, Metadata::new(EntryMode::FILE)) } else if file_type.is_dir() { oio::Entry::new(&format!("{rel_path}/"), Metadata::new(EntryMode::DIR)) } else { oio::Entry::new(&rel_path, Metadata::new(EntryMode::Unknown)) }; Ok(Some(entry)) } impl oio::List for CompfsLister { async fn next(&mut self) -> Result> { let Some(mut read_dir) = self.read_dir.take() else { return Ok(None); }; let root = self.core.root.clone(); let (entry, read_dir) = self .core .exec_blocking(move || { let entry = next_entry(&mut read_dir, &root).map_err(new_std_io_error); (entry, read_dir) }) .await?; self.read_dir = Some(read_dir); entry } } opendal-0.52.0/src/services/compfs/mod.rs000064400000000000000000000022731046102023000163640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-compfs")] mod core; #[cfg(feature = "services-compfs")] mod delete; #[cfg(feature = "services-compfs")] mod lister; #[cfg(feature = "services-compfs")] mod reader; #[cfg(feature = "services-compfs")] mod writer; #[cfg(feature = "services-compfs")] mod backend; #[cfg(feature = "services-compfs")] pub use backend::CompfsBuilder as Compfs; mod config; pub use config::CompfsConfig; opendal-0.52.0/src/services/compfs/reader.rs000064400000000000000000000042521046102023000170460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use compio::buf::buf_try; use compio::io::AsyncReadAt; use super::core::CompfsCore; use crate::raw::*; use crate::*; #[derive(Debug)] pub struct CompfsReader { core: Arc, file: compio::fs::File, offset: u64, end: Option, } impl CompfsReader { pub(super) fn new(core: Arc, file: compio::fs::File, range: BytesRange) -> Self { Self { core, file, offset: range.offset(), end: range.size().map(|v| v + range.offset()), } } } impl oio::Read for CompfsReader { async fn read(&mut self) -> Result { let pos = self.offset; if let Some(end) = self.end { if end >= pos { return Ok(Buffer::new()); } } let mut bs = self.core.buf_pool.get(); // reserve 64KB buffer by default, we should allow user to configure this or make it adaptive. bs.reserve(64 * 1024); let f = self.file.clone(); let (n, mut bs) = self .core .exec(move || async move { let (n, bs) = buf_try!(@try f.read_at(bs, pos).await); Ok((n, bs)) }) .await?; let frozen = bs.split_to(n).freeze(); self.offset += frozen.len() as u64; self.core.buf_pool.put(bs); Ok(Buffer::from(frozen)) } } opendal-0.52.0/src/services/compfs/writer.rs000064400000000000000000000042031046102023000171140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io::Cursor; use std::sync::Arc; use compio::buf::buf_try; use compio::fs::File; use compio::io::AsyncWriteExt; use super::core::CompfsCore; use crate::raw::*; use crate::*; #[derive(Debug)] pub struct CompfsWriter { core: Arc, file: Cursor, } impl CompfsWriter { pub(super) fn new(core: Arc, file: Cursor) -> Self { Self { core, file } } } impl oio::Write for CompfsWriter { /// FIXME /// /// the write_all doesn't work correctly if `bs` is non-contiguous. /// /// The IoBuf::buf_len() only returns the length of the current buffer. async fn write(&mut self, bs: Buffer) -> Result<()> { let mut file = self.file.clone(); self.core .exec(move || async move { buf_try!(@try file.write_all(bs).await); Ok(()) }) .await?; Ok(()) } async fn close(&mut self) -> Result { let f = self.file.clone(); self.core .exec(move || async move { f.get_ref().sync_all().await }) .await?; let f = self.file.clone(); self.core .exec(move || async move { f.into_inner().close().await }) .await?; Ok(Metadata::default()) } async fn abort(&mut self) -> Result<()> { Ok(()) } } opendal-0.52.0/src/services/cos/backend.rs000064400000000000000000000335261046102023000164760ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::sync::Arc; use http::Response; use http::StatusCode; use http::Uri; use log::debug; use reqsign::TencentCosConfig; use reqsign::TencentCosCredentialLoader; use reqsign::TencentCosSigner; use super::core::*; use super::delete::CosDeleter; use super::error::parse_error; use super::lister::{CosLister, CosListers, CosObjectVersionsLister}; use super::writer::CosWriter; use super::writer::CosWriters; use crate::raw::oio::PageLister; use crate::raw::*; use crate::services::CosConfig; use crate::*; impl Configurator for CosConfig { type Builder = CosBuilder; fn into_builder(self) -> Self::Builder { CosBuilder { config: self, http_client: None, } } } /// Tencent-Cloud COS services support. #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct CosBuilder { config: CosConfig, http_client: Option, } impl Debug for CosBuilder { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("CosBuilder") .field("config", &self.config) .finish() } } impl CosBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set endpoint of this backend. /// /// NOTE: no bucket or account id in endpoint, we will trim them if exists. /// /// # Examples /// /// - `https://cos.ap-singapore.myqcloud.com` pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()); } self } /// Set secret_id of this backend. /// - If it is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn secret_id(mut self, secret_id: &str) -> Self { if !secret_id.is_empty() { self.config.secret_id = Some(secret_id.to_string()); } self } /// Set secret_key of this backend. /// - If it is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn secret_key(mut self, secret_key: &str) -> Self { if !secret_key.is_empty() { self.config.secret_key = Some(secret_key.to_string()); } self } /// Set bucket of this backend. /// The param is required. pub fn bucket(mut self, bucket: &str) -> Self { if !bucket.is_empty() { self.config.bucket = Some(bucket.to_string()); } self } /// Set bucket versioning status for this backend pub fn enable_versioning(mut self, enabled: bool) -> Self { self.config.enable_versioning = enabled; self } /// Disable config load so that opendal will not load config from /// environment. /// /// For examples: /// /// - envs like `TENCENTCLOUD_SECRET_ID` pub fn disable_config_load(mut self) -> Self { self.config.disable_config_load = true; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for CosBuilder { const SCHEME: Scheme = Scheme::Cos; type Config = CosConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let bucket = match &self.config.bucket { Some(bucket) => Ok(bucket.to_string()), None => Err( Error::new(ErrorKind::ConfigInvalid, "The bucket is misconfigured") .with_context("service", Scheme::Cos), ), }?; debug!("backend use bucket {}", &bucket); let uri = match &self.config.endpoint { Some(endpoint) => endpoint.parse::().map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "endpoint is invalid") .with_context("service", Scheme::Cos) .with_context("endpoint", endpoint) .set_source(err) }), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_context("service", Scheme::Cos)), }?; let scheme = match uri.scheme_str() { Some(scheme) => scheme.to_string(), None => "https".to_string(), }; // If endpoint contains bucket name, we should trim them. let endpoint = uri.host().unwrap().replace(&format!("//{bucket}."), "//"); debug!("backend use endpoint {}", &endpoint); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Cos) })? }; let mut cfg = TencentCosConfig::default(); if !self.config.disable_config_load { cfg = cfg.from_env(); } if let Some(v) = self.config.secret_id { cfg.secret_id = Some(v); } if let Some(v) = self.config.secret_key { cfg.secret_key = Some(v); } let cred_loader = TencentCosCredentialLoader::new(GLOBAL_REQWEST_CLIENT.clone(), cfg); let signer = TencentCosSigner::new(); Ok(CosBackend { core: Arc::new(CosCore { bucket: bucket.clone(), root, endpoint: format!("{}://{}.{}", &scheme, &bucket, &endpoint), enable_versioning: self.config.enable_versioning, signer, loader: cred_loader, client, }), }) } } /// Backend for Tencent-Cloud COS services. #[derive(Debug, Clone)] pub struct CosBackend { core: Arc, } impl Access for CosBackend { type Reader = HttpBody; type Writer = CosWriters; type Lister = CosListers; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Cos) .set_root(&self.core.root) .set_name(&self.core.bucket) .set_native_capability(Capability { stat: true, stat_with_if_match: true, stat_with_if_none_match: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_with_version: self.core.enable_versioning, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, stat_has_version: true, stat_has_user_metadata: true, read: true, read_with_if_match: true, read_with_if_none_match: true, read_with_version: self.core.enable_versioning, write: true, write_can_empty: true, write_can_append: true, write_can_multi: true, write_with_content_type: true, write_with_cache_control: true, write_with_content_disposition: true, // Cos doesn't support forbid overwrite while version has been enabled. write_with_if_not_exists: !self.core.enable_versioning, // The min multipart size of COS is 1 MiB. // // ref: write_multi_min_size: Some(1024 * 1024), // The max multipart size of COS is 5 GiB. // // ref: write_multi_max_size: if cfg!(target_pointer_width = "64") { Some(5 * 1024 * 1024 * 1024) } else { Some(usize::MAX) }, write_with_user_metadata: true, delete: true, delete_with_version: self.core.enable_versioning, copy: true, list: true, list_with_recursive: true, list_with_versions: self.core.enable_versioning, list_with_deleted: self.core.enable_versioning, list_has_content_length: true, presign: true, presign_stat: true, presign_read: true, presign_write: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, args: OpStat) -> Result { let resp = self.core.cos_head_object(path, &args).await?; let status = resp.status(); match status { StatusCode::OK => { let headers = resp.headers(); let mut meta = parse_into_metadata(path, headers)?; let user_meta = parse_prefixed_headers(headers, "x-cos-meta-"); if !user_meta.is_empty() { meta.with_user_metadata(user_meta); } if let Some(v) = parse_header_to_str(headers, "x-cos-version-id")? { meta.set_version(v); } Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.cos_get_object(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = CosWriter::new(self.core.clone(), path, args.clone()); let w = if args.append() { CosWriters::Two(oio::AppendWriter::new(writer)) } else { CosWriters::One(oio::MultipartWriter::new( writer, args.executor().cloned(), args.concurrent(), )) }; Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(CosDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = if args.versions() || args.deleted() { TwoWays::Two(PageLister::new(CosObjectVersionsLister::new( self.core.clone(), path, args, ))) } else { TwoWays::One(PageLister::new(CosLister::new( self.core.clone(), path, args.recursive(), args.limit(), ))) }; Ok((RpList::default(), l)) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.cos_copy_object(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn presign(&self, path: &str, args: OpPresign) -> Result { let mut req = match args.operation() { PresignOperation::Stat(v) => self.core.cos_head_object_request(path, v)?, PresignOperation::Read(v) => { self.core .cos_get_object_request(path, BytesRange::default(), v)? } PresignOperation::Write(v) => { self.core .cos_put_object_request(path, None, v, Buffer::new())? } }; self.core.sign_query(&mut req, args.expire()).await?; // We don't need this request anymore, consume it directly. let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } } opendal-0.52.0/src/services/cos/config.rs000064400000000000000000000036551046102023000163540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Tencent-Cloud COS services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] pub struct CosConfig { /// Root of this backend. pub root: Option, /// Endpoint of this backend. pub endpoint: Option, /// Secret ID of this backend. pub secret_id: Option, /// Secret key of this backend. pub secret_key: Option, /// Bucket of this backend. pub bucket: Option, /// is bucket versioning enabled for this bucket pub enable_versioning: bool, /// Disable config load so that opendal will not load config from pub disable_config_load: bool, } impl Debug for CosConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("CosConfig") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("secret_id", &"") .field("secret_key", &"") .field("bucket", &self.bucket) .finish_non_exhaustive() } } opendal-0.52.0/src/services/cos/core.rs000064400000000000000000000522011046102023000160260ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::Write; use std::time::Duration; use bytes::Bytes; use http::header::CACHE_CONTROL; use http::header::CONTENT_DISPOSITION; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::header::IF_MATCH; use http::header::IF_NONE_MATCH; use http::Request; use http::Response; use reqsign::TencentCosCredential; use reqsign::TencentCosCredentialLoader; use reqsign::TencentCosSigner; use serde::Deserialize; use serde::Serialize; use crate::raw::*; use crate::*; pub mod constants { pub const COS_QUERY_VERSION_ID: &str = "versionId"; } pub struct CosCore { pub bucket: String, pub root: String, pub endpoint: String, pub enable_versioning: bool, pub signer: TencentCosSigner, pub loader: TencentCosCredentialLoader, pub client: HttpClient, } impl Debug for CosCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .finish_non_exhaustive() } } impl CosCore { async fn load_credential(&self) -> Result> { let cred = self .loader .load() .await .map_err(new_request_credential_error)?; if let Some(cred) = cred { return Ok(Some(cred)); } Err(Error::new( ErrorKind::PermissionDenied, "no valid credential found and anonymous access is not allowed", )) } pub async fn sign(&self, req: &mut Request) -> Result<()> { let cred = if let Some(cred) = self.load_credential().await? { cred } else { return Ok(()); }; self.signer.sign(req, &cred).map_err(new_request_sign_error) } pub async fn sign_query(&self, req: &mut Request, duration: Duration) -> Result<()> { let cred = if let Some(cred) = self.load_credential().await? { cred } else { return Ok(()); }; self.signer .sign_query(req, duration, &cred) .map_err(new_request_sign_error) } #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } } impl CosCore { pub async fn cos_get_object( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let mut req = self.cos_get_object_request(path, range, args)?; self.sign(&mut req).await?; self.client.fetch(req).await } pub fn cos_get_object_request( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let mut query_args = Vec::new(); if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::COS_QUERY_VERSION_ID, percent_decode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let mut req = Request::get(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if !range.is_full() { req = req.header(http::header::RANGE, range.to_header()) } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub fn cos_put_object_request( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let mut req = Request::put(&url); if let Some(size) = size { req = req.header(CONTENT_LENGTH, size) } if let Some(cache_control) = args.cache_control() { req = req.header(CACHE_CONTROL, cache_control) } if let Some(pos) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, pos) } if let Some(mime) = args.content_type() { req = req.header(CONTENT_TYPE, mime) } // For a bucket which has never enabled versioning, you may use it to // specify whether to prohibit overwriting the object with the same name // when uploading the object: // // When the x-cos-forbid-overwrite is specified as true, overwriting the object // with the same name will be prohibited. // // ref: https://www.tencentcloud.com/document/product/436/7749 if args.if_not_exists() { req = req.header("x-cos-forbid-overwrite", "true") } // Set user metadata headers. if let Some(user_metadata) = args.user_metadata() { for (key, value) in user_metadata { req = req.header(format!("x-cos-meta-{key}"), value) } } let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn cos_head_object(&self, path: &str, args: &OpStat) -> Result> { let mut req = self.cos_head_object_request(path, args)?; self.sign(&mut req).await?; self.send(req).await } pub fn cos_head_object_request(&self, path: &str, args: &OpStat) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let mut query_args = Vec::new(); if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::COS_QUERY_VERSION_ID, percent_decode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let mut req = Request::head(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub async fn cos_delete_object(&self, path: &str, args: &OpDelete) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let mut query_args = Vec::new(); if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::COS_QUERY_VERSION_ID, percent_decode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let req = Request::delete(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub fn cos_append_object_request( &self, path: &str, position: u64, size: u64, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?append&position={}", self.endpoint, percent_encode_path(&p), position ); let mut req = Request::post(&url); req = req.header(CONTENT_LENGTH, size); if let Some(mime) = args.content_type() { req = req.header(CONTENT_TYPE, mime); } if let Some(pos) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, pos); } if let Some(cache_control) = args.cache_control() { req = req.header(CACHE_CONTROL, cache_control) } let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn cos_copy_object(&self, from: &str, to: &str) -> Result> { let source = build_abs_path(&self.root, from); let target = build_abs_path(&self.root, to); let source = format!("/{}/{}", self.bucket, percent_encode_path(&source)); let url = format!("{}/{}", self.endpoint, percent_encode_path(&target)); let mut req = Request::put(&url) .header("x-cos-copy-source", &source) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn cos_list_objects( &self, path: &str, next_marker: &str, delimiter: &str, limit: Option, ) -> Result> { let p = build_abs_path(&self.root, path); let mut queries = vec![]; if !p.is_empty() { queries.push(format!("prefix={}", percent_encode_path(&p))); } if !delimiter.is_empty() { queries.push(format!("delimiter={delimiter}")); } if let Some(limit) = limit { queries.push(format!("max-keys={limit}")); } if !next_marker.is_empty() { queries.push(format!("marker={next_marker}")); } let url = if queries.is_empty() { self.endpoint.to_string() } else { format!("{}?{}", self.endpoint, queries.join("&")) }; let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn cos_initiate_multipart_upload( &self, path: &str, args: &OpWrite, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}?uploads", self.endpoint, percent_encode_path(&p)); let mut req = Request::post(&url); if let Some(mime) = args.content_type() { req = req.header(CONTENT_TYPE, mime) } if let Some(content_disposition) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, content_disposition) } if let Some(cache_control) = args.cache_control() { req = req.header(CACHE_CONTROL, cache_control) } // Set user metadata headers. if let Some(user_metadata) = args.user_metadata() { for (key, value) in user_metadata { req = req.header(format!("x-cos-meta-{key}"), value) } } let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn cos_upload_part_request( &self, path: &str, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?partNumber={}&uploadId={}", self.endpoint, percent_encode_path(&p), part_number, percent_encode_path(upload_id) ); let mut req = Request::put(&url); req = req.header(CONTENT_LENGTH, size); // Set body let mut req = req.body(body).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn cos_complete_multipart_upload( &self, path: &str, upload_id: &str, parts: Vec, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?uploadId={}", self.endpoint, percent_encode_path(&p), percent_encode_path(upload_id) ); let req = Request::post(&url); let content = quick_xml::se::to_string(&CompleteMultipartUploadRequest { part: parts }) .map_err(new_xml_deserialize_error)?; // Make sure content length has been set to avoid post with chunked encoding. let req = req.header(CONTENT_LENGTH, content.len()); // Set content-type to `application/xml` to avoid mixed with form post. let req = req.header(CONTENT_TYPE, "application/xml"); let mut req = req .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } /// Abort an on-going multipart upload. pub async fn cos_abort_multipart_upload( &self, path: &str, upload_id: &str, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?uploadId={}", self.endpoint, percent_encode_path(&p), percent_encode_path(upload_id) ); let mut req = Request::delete(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn cos_list_object_versions( &self, prefix: &str, delimiter: &str, limit: Option, key_marker: &str, version_id_marker: &str, ) -> Result> { let p = build_abs_path(&self.root, prefix); let mut url = format!("{}?versions", self.endpoint); if !p.is_empty() { write!(url, "&prefix={}", percent_encode_path(p.as_str())) .expect("write into string must succeed"); } if !delimiter.is_empty() { write!(url, "&delimiter={}", delimiter).expect("write into string must succeed"); } if let Some(limit) = limit { write!(url, "&max-keys={}", limit).expect("write into string must succeed"); } if !key_marker.is_empty() { write!(url, "&key-marker={}", percent_encode_path(key_marker)) .expect("write into string must succeed"); } if !version_id_marker.is_empty() { write!( url, "&version-id-marker={}", percent_encode_path(version_id_marker) ) .expect("write into string must succeed"); } let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } } /// Result of CreateMultipartUpload #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct InitiateMultipartUploadResult { pub upload_id: String, } /// Request of CompleteMultipartUploadRequest #[derive(Default, Debug, Serialize)] #[serde(default, rename = "CompleteMultipartUpload", rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequest { pub part: Vec, } #[derive(Clone, Default, Debug, Serialize)] #[serde(default, rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequestPart { #[serde(rename = "PartNumber")] pub part_number: usize, /// # TODO /// /// quick-xml will do escape on `"` which leads to our serialized output is /// not the same as aws s3's example. /// /// Ideally, we could use `serialize_with` to address this (buf failed) /// /// ```ignore /// #[derive(Default, Debug, Serialize)] /// #[serde(default, rename_all = "PascalCase")] /// struct CompleteMultipartUploadRequestPart { /// #[serde(rename = "PartNumber")] /// part_number: usize, /// #[serde(rename = "ETag", serialize_with = "partial_escape")] /// etag: String, /// } /// /// fn partial_escape(s: &str, ser: S) -> Result /// where /// S: serde::Serializer, /// { /// ser.serialize_str(&String::from_utf8_lossy( /// &quick_xml::escape::partial_escape(s.as_bytes()), /// )) /// } /// ``` /// /// ref: #[serde(rename = "ETag")] pub etag: String, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectsOutput { pub name: String, pub prefix: String, pub contents: Vec, pub common_prefixes: Vec, pub marker: String, pub next_marker: Option, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct CommonPrefix { pub prefix: String, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectsOutputContent { pub key: String, pub size: u64, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct OutputCommonPrefix { pub prefix: String, } /// Output of ListObjectVersions #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectVersionsOutput { pub is_truncated: Option, pub next_key_marker: Option, pub next_version_id_marker: Option, pub common_prefixes: Vec, pub version: Vec, pub delete_marker: Vec, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct ListObjectVersionsOutputVersion { pub key: String, pub version_id: String, pub is_latest: bool, pub size: u64, pub last_modified: String, #[serde(rename = "ETag")] pub etag: Option, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct ListObjectVersionsOutputDeleteMarker { pub key: String, pub version_id: String, pub is_latest: bool, pub last_modified: String, } #[cfg(test)] mod tests { use bytes::Buf; use super::*; #[test] fn test_parse_xml() { let bs = bytes::Bytes::from( r#" examplebucket obj obj002 obj004 1000 false obj002 2015-07-01T02:11:19.775Z "a72e382246ac83e86bd203389849e71d" 9 b4bf1b36d9ca43d984fbcb9491b6fce9 STANDARD obj003 2015-07-01T02:11:19.775Z "a72e382246ac83e86bd203389849e71d" 10 b4bf1b36d9ca43d984fbcb9491b6fce9 STANDARD hello world "#, ); let out: ListObjectsOutput = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert_eq!(out.name, "examplebucket".to_string()); assert_eq!(out.prefix, "obj".to_string()); assert_eq!(out.marker, "obj002".to_string()); assert_eq!(out.next_marker, Some("obj004".to_string()),); assert_eq!( out.contents .iter() .map(|v| v.key.clone()) .collect::>(), ["obj002", "obj003"], ); assert_eq!( out.contents.iter().map(|v| v.size).collect::>(), [9, 10], ); assert_eq!( out.common_prefixes .iter() .map(|v| v.prefix.clone()) .collect::>(), ["hello", "world"], ) } } opendal-0.52.0/src/services/cos/delete.rs000064400000000000000000000027031046102023000163420ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct CosDeleter { core: Arc, } impl CosDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for CosDeleter { async fn delete_once(&self, path: String, args: OpDelete) -> Result<()> { let resp = self.core.cos_delete_object(&path, &args).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT | StatusCode::ACCEPTED | StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/cos/docs.md000064400000000000000000000024511046102023000160040ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `bucket`: Set the container name for backend - `endpoint`: Customizable endpoint setting - `access_key_id`: Set the access_key_id for backend. - `secret_access_key`: Set the secret_access_key for backend. You can refer to [`CosBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Cos; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Cos::default() // set the storage bucket for OpenDAL .bucket("test") // set the endpoint for OpenDAL .endpoint("https://cos.ap-singapore.myqcloud.com") // Set the access_key_id and secret_access_key. // // OpenDAL will try load credential from the env. // If credential not set and no valid credential in env, OpenDAL will // send request without signing like anonymous user. .secret_id("secret_id") .secret_key("secret_access_key"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/cos/error.rs000064400000000000000000000065671046102023000162450ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use http::StatusCode; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// CosError is the error returned by cos service. #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] struct CosError { code: String, message: String, resource: String, request_id: String, host_id: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::NOT_MODIFIED | StatusCode::CONFLICT => { (ErrorKind::ConditionNotMatch, false) } StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), // COS could return `520 Origin Error` errors which should be retried. v if v.as_u16() == 520 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = match de::from_reader::<_, CosError>(bs.clone().reader()) { Ok(cos_error) => format!("{cos_error:?}"), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod tests { use super::*; #[test] fn test_parse_error() { let bs = bytes::Bytes::from( r#" NoSuchKey The resource you requested does not exist /example-bucket/object 001B21A61C6C0000013402C4616D5285 RkRCRDJENDc5MzdGQkQ4OUY3MTI4NTQ3NDk2Mjg0M0FBQUFBQUFBYmJiYmJiYmJD "#, ); let out: CosError = de::from_reader(bs.reader()).expect("must success"); println!("{out:?}"); assert_eq!(out.code, "NoSuchKey"); assert_eq!(out.message, "The resource you requested does not exist"); assert_eq!(out.resource, "/example-bucket/object"); assert_eq!(out.request_id, "001B21A61C6C0000013402C4616D5285"); assert_eq!( out.host_id, "RkRCRDJENDc5MzdGQkQ4OUY3MTI4NTQ3NDk2Mjg0M0FBQUFBQUFBYmJiYmJiYmJD" ); } } opendal-0.52.0/src/services/cos/lister.rs000064400000000000000000000171041046102023000164030ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use quick_xml::de; use super::core::*; use super::error::parse_error; use crate::raw::oio::PageContext; use crate::raw::*; use crate::EntryMode; use crate::Error; use crate::Metadata; use crate::Result; pub type CosListers = TwoWays, oio::PageLister>; pub struct CosLister { core: Arc, path: String, delimiter: &'static str, limit: Option, } impl CosLister { pub fn new(core: Arc, path: &str, recursive: bool, limit: Option) -> Self { let delimiter = if recursive { "" } else { "/" }; Self { core, path: path.to_string(), delimiter, limit, } } } impl oio::PageList for CosLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .cos_list_objects(&self.path, &ctx.token, self.delimiter, self.limit) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let output: ListObjectsOutput = de::from_reader(bs.reader()).map_err(new_xml_deserialize_error)?; // Try our best to check whether this list is done. // // - Check `next_marker` ctx.done = match output.next_marker.as_ref() { None => true, Some(next_marker) => next_marker.is_empty(), }; ctx.token = output.next_marker.clone().unwrap_or_default(); for prefix in output.common_prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix.prefix), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de); } for object in output.contents { let mut path = build_rel_path(&self.core.root, &object.key); if path.is_empty() { path = "/".to_string(); } let meta = Metadata::new(EntryMode::from_path(&path)).with_content_length(object.size); let de = oio::Entry::with(path, meta); ctx.entries.push_back(de); } Ok(()) } } /// refer: https://cloud.tencent.com/document/product/436/35521 pub struct CosObjectVersionsLister { core: Arc, prefix: String, args: OpList, delimiter: &'static str, abs_start_after: Option, } impl CosObjectVersionsLister { pub fn new(core: Arc, path: &str, args: OpList) -> Self { let delimiter = if args.recursive() { "" } else { "/" }; let abs_start_after = args .start_after() .map(|start_after| build_abs_path(&core.root, start_after)); Self { core, prefix: path.to_string(), args, delimiter, abs_start_after, } } } impl oio::PageList for CosObjectVersionsLister { async fn next_page(&self, ctx: &mut PageContext) -> Result<()> { let markers = ctx.token.rsplit_once(" "); let (key_marker, version_id_marker) = if let Some(data) = markers { data } else if let Some(start_after) = &self.abs_start_after { (start_after.as_str(), "") } else { ("", "") }; let resp = self .core .cos_list_object_versions( &self.prefix, self.delimiter, self.args.limit(), key_marker, version_id_marker, ) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let body = resp.into_body(); let output: ListObjectVersionsOutput = de::from_reader(body.reader()) .map_err(new_xml_deserialize_error) // Allow Cos list to retry on XML deserialization errors. // // This is because the Cos list API may return incomplete XML data under high load. // We are confident that our XML decoding logic is correct. When this error occurs, // we allow retries to obtain the correct data. .map_err(Error::set_temporary)?; ctx.done = if let Some(is_truncated) = output.is_truncated { !is_truncated } else { false }; ctx.token = format!( "{} {}", output.next_key_marker.unwrap_or_default(), output.next_version_id_marker.unwrap_or_default() ); for prefix in output.common_prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix.prefix), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de); } for version_object in output.version { // `list` must be additive, so we need to include the latest version object // even if `versions` is not enabled. // // Here we skip all non-latest version objects if `versions` is not enabled. if !(self.args.versions() || version_object.is_latest) { continue; } let mut path = build_rel_path(&self.core.root, &version_object.key); if path.is_empty() { path = "/".to_owned(); } let mut meta = Metadata::new(EntryMode::from_path(&path)); meta.set_version(&version_object.version_id); meta.set_is_current(version_object.is_latest); meta.set_content_length(version_object.size); meta.set_last_modified(parse_datetime_from_rfc3339( version_object.last_modified.as_str(), )?); if let Some(etag) = version_object.etag { meta.set_etag(&etag); meta.set_content_md5(etag.trim_matches('"')); } let entry = oio::Entry::new(&path, meta); ctx.entries.push_back(entry); } if self.args.deleted() { for delete_marker in output.delete_marker { let mut path = build_rel_path(&self.core.root, &delete_marker.key); if path.is_empty() { path = "/".to_owned(); } let mut meta = Metadata::new(EntryMode::FILE); meta.set_version(&delete_marker.version_id); meta.set_is_deleted(true); meta.set_is_current(delete_marker.is_latest); meta.set_last_modified(parse_datetime_from_rfc3339( delete_marker.last_modified.as_str(), )?); let entry = oio::Entry::new(&path, meta); ctx.entries.push_back(entry); } } Ok(()) } } opendal-0.52.0/src/services/cos/mod.rs000064400000000000000000000022351046102023000156570ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-cos")] mod core; #[cfg(feature = "services-cos")] mod delete; #[cfg(feature = "services-cos")] mod error; #[cfg(feature = "services-cos")] mod lister; #[cfg(feature = "services-cos")] mod writer; #[cfg(feature = "services-cos")] mod backend; #[cfg(feature = "services-cos")] pub use backend::CosBuilder as Cos; mod config; pub use config::CosConfig; opendal-0.52.0/src/services/cos/writer.rs000064400000000000000000000133761046102023000164240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; pub type CosWriters = TwoWays, oio::AppendWriter>; pub struct CosWriter { core: Arc, op: OpWrite, path: String, } impl CosWriter { pub fn new(core: Arc, path: &str, op: OpWrite) -> Self { CosWriter { core, path: path.to_string(), op, } } } impl oio::MultipartWrite for CosWriter { async fn write_once(&self, size: u64, body: Buffer) -> Result { let mut req = self .core .cos_put_object_request(&self.path, Some(size), &self.op, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn initiate_part(&self) -> Result { let resp = self .core .cos_initiate_multipart_upload(&self.path, &self.op) .await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let result: InitiateMultipartUploadResult = quick_xml::de::from_reader(bytes::Buf::reader(bs)) .map_err(new_xml_deserialize_error)?; Ok(result.upload_id) } _ => Err(parse_error(resp)), } } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result { // COS requires part number must between [1..=10000] let part_number = part_number + 1; let resp = self .core .cos_upload_part_request(&self.path, upload_id, part_number, size, body) .await?; let status = resp.status(); match status { StatusCode::OK => { let etag = parse_etag(resp.headers())? .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "ETag not present in returning response", ) })? .to_string(); Ok(oio::MultipartPart { part_number, etag, checksum: None, }) } _ => Err(parse_error(resp)), } } async fn complete_part( &self, upload_id: &str, parts: &[oio::MultipartPart], ) -> Result { let parts = parts .iter() .map(|p| CompleteMultipartUploadRequestPart { part_number: p.part_number, etag: p.etag.clone(), }) .collect(); let resp = self .core .cos_complete_multipart_upload(&self.path, upload_id, parts) .await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn abort_part(&self, upload_id: &str) -> Result<()> { let resp = self .core .cos_abort_multipart_upload(&self.path, upload_id) .await?; match resp.status() { // cos returns code 204 if abort succeeds. // Reference: https://www.tencentcloud.com/document/product/436/7740 StatusCode::NO_CONTENT => Ok(()), _ => Err(parse_error(resp)), } } } impl oio::AppendWrite for CosWriter { async fn offset(&self) -> Result { let resp = self .core .cos_head_object(&self.path, &OpStat::default()) .await?; let status = resp.status(); match status { StatusCode::OK => { let content_length = parse_content_length(resp.headers())?.ok_or_else(|| { Error::new( ErrorKind::Unexpected, "Content-Length not present in returning response", ) })?; Ok(content_length) } StatusCode::NOT_FOUND => Ok(0), _ => Err(parse_error(resp)), } } async fn append(&self, offset: u64, size: u64, body: Buffer) -> Result { let mut req = self .core .cos_append_object_request(&self.path, offset, size, &self.op, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/d1/backend.rs000064400000000000000000000234021046102023000162060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use http::header; use http::Request; use http::StatusCode; use serde_json::Value; use super::error::parse_error; use super::model::D1Response; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::D1Config; use crate::ErrorKind; use crate::*; impl Configurator for D1Config { type Builder = D1Builder; fn into_builder(self) -> Self::Builder { D1Builder { config: self, http_client: None, } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct D1Builder { config: D1Config, http_client: Option, } impl Debug for D1Builder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("D1Builder") .field("config", &self.config) .finish() } } impl D1Builder { /// Set api token for the cloudflare d1 service. /// /// create a api token from [here](https://dash.cloudflare.com/profile/api-tokens) pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()); } self } /// Set the account identifier for the cloudflare d1 service. /// /// get the account identifier from Workers & Pages -> Overview -> Account ID /// If not specified, it will return an error when building. pub fn account_id(mut self, account_id: &str) -> Self { if !account_id.is_empty() { self.config.account_id = Some(account_id.to_string()); } self } /// Set the database identifier for the cloudflare d1 service. /// /// get the database identifier from Workers & Pages -> D1 -> [Your Database] -> Database ID /// If not specified, it will return an error when building. pub fn database_id(mut self, database_id: &str) -> Self { if !database_id.is_empty() { self.config.database_id = Some(database_id.to_string()); } self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the table name of the d1 service to read/write. /// /// If not specified, it will return an error when building. pub fn table(mut self, table: &str) -> Self { if !table.is_empty() { self.config.table = Some(table.to_owned()); } self } /// Set the key field name of the d1 service to read/write. /// /// Default to `key` if not specified. pub fn key_field(mut self, key_field: &str) -> Self { if !key_field.is_empty() { self.config.key_field = Some(key_field.to_string()); } self } /// Set the value field name of the d1 service to read/write. /// /// Default to `value` if not specified. pub fn value_field(mut self, value_field: &str) -> Self { if !value_field.is_empty() { self.config.value_field = Some(value_field.to_string()); } self } } impl Builder for D1Builder { const SCHEME: Scheme = Scheme::D1; type Config = D1Config; fn build(self) -> Result { let mut authorization = None; let config = self.config; if let Some(token) = config.token { authorization = Some(format_authorization_by_bearer(&token)?) } let Some(account_id) = config.account_id else { return Err(Error::new( ErrorKind::ConfigInvalid, "account_id is required", )); }; let Some(database_id) = config.database_id.clone() else { return Err(Error::new( ErrorKind::ConfigInvalid, "database_id is required", )); }; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::D1) })? }; let Some(table) = config.table.clone() else { return Err(Error::new(ErrorKind::ConfigInvalid, "table is required")); }; let key_field = config .key_field .clone() .unwrap_or_else(|| "key".to_string()); let value_field = config .value_field .clone() .unwrap_or_else(|| "value".to_string()); let root = normalize_root( config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); Ok(D1Backend::new(Adapter { authorization, account_id, database_id, client, table, key_field, value_field, }) .with_normalized_root(root)) } } pub type D1Backend = kv::Backend; #[derive(Clone)] pub struct Adapter { authorization: Option, account_id: String, database_id: String, client: HttpClient, table: String, key_field: String, value_field: String, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("D1Adapter"); ds.field("table", &self.table); ds.field("key_field", &self.key_field); ds.field("value_field", &self.value_field); ds.finish() } } impl Adapter { fn create_d1_query_request(&self, sql: &str, params: Vec) -> Result> { let p = format!( "/accounts/{}/d1/database/{}/query", self.account_id, self.database_id ); let url: String = format!( "{}{}", "https://api.cloudflare.com/client/v4", percent_encode_path(&p) ); let mut req = Request::post(&url); if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth); } req = req.header(header::CONTENT_TYPE, "application/json"); let json = serde_json::json!({ "sql": sql, "params": params, }); let body = serde_json::to_vec(&json).map_err(new_json_serialize_error)?; req.body(Buffer::from(body)) .map_err(new_request_build_error) } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::D1, &self.table, Capability { read: true, write: true, // Cloudflare D1 supports 1MB as max in write_total. // refer to https://developers.cloudflare.com/d1/platform/limits/ write_total_max_size: Some(1000 * 1000), shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let query = format!( "SELECT {} FROM {} WHERE {} = ? LIMIT 1", self.value_field, self.table, self.key_field ); let req = self.create_d1_query_request(&query, vec![path.into()])?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { let body = resp.into_body(); let bs = body.to_bytes(); let d1_response = D1Response::parse(&bs)?; Ok(d1_response.get_result(&self.value_field)) } _ => Err(parse_error(resp)), } } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let table = &self.table; let key_field = &self.key_field; let value_field = &self.value_field; let query = format!( "INSERT INTO {table} ({key_field}, {value_field}) \ VALUES (?, ?) \ ON CONFLICT ({key_field}) \ DO UPDATE SET {value_field} = EXCLUDED.{value_field}", ); let params = vec![path.into(), value.to_vec().into()]; let req = self.create_d1_query_request(&query, params)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok(()), _ => Err(parse_error(resp)), } } async fn delete(&self, path: &str) -> Result<()> { let query = format!("DELETE FROM {} WHERE {} = ?", self.table, self.key_field); let req = self.create_d1_query_request(&query, vec![path.into()])?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/d1/config.rs000064400000000000000000000037311046102023000160670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for [Cloudflare D1](https://developers.cloudflare.com/d1) backend support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct D1Config { /// Set the token of cloudflare api. pub token: Option, /// Set the account id of cloudflare api. pub account_id: Option, /// Set the database id of cloudflare api. pub database_id: Option, /// Set the working directory of OpenDAL. pub root: Option, /// Set the table of D1 Database. pub table: Option, /// Set the key field of D1 Database. pub key_field: Option, /// Set the value field of D1 Database. pub value_field: Option, } impl Debug for D1Config { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("D1Config"); ds.field("root", &self.root); ds.field("table", &self.table); ds.field("key_field", &self.key_field); ds.field("value_field", &self.value_field); ds.finish_non_exhaustive() } } opendal-0.52.0/src/services/d1/docs.md000064400000000000000000000017411046102023000155250ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `token`: Set the token of cloudflare api - `account_id`: Set the account id of cloudflare api - `database_id`: Set the database id of cloudflare api - `table`: Set the table of D1 Database - `key_field`: Set the key field of D1 Database - `value_field`: Set the value field of D1 Database ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::D1; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = D1::default() .token("token") .account_id("account_id") .database_id("database_id") .table("table") .key_field("key_field") .value_field("value_field"); let op = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/d1/error.rs000064400000000000000000000053151046102023000157530ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use http::StatusCode; use serde_json::de; use super::model::D1Error; use super::model::D1Response; use crate::raw::*; use crate::*; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (mut kind, mut retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), // Some services (like owncloud) return 403 while file locked. StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, true), // Allowing retry for resource locked. StatusCode::LOCKED => (ErrorKind::Unexpected, true), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, d1_err) = de::from_reader::<_, D1Response>(bs.clone().reader()) .map(|d1_err| (format!("{d1_err:?}"), Some(d1_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); if let Some(d1_err) = d1_err { (kind, retryable) = parse_d1_error_code(d1_err.errors).unwrap_or((kind, retryable)); } let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } pub fn parse_d1_error_code(errors: Vec) -> Option<(ErrorKind, bool)> { if errors.is_empty() { return None; } match errors[0].code { // The request is malformed: failed to decode id. 7400 => Some((ErrorKind::Unexpected, false)), // no such column: Xxxx. 7500 => Some((ErrorKind::NotFound, false)), // Authentication error. 10000 => Some((ErrorKind::PermissionDenied, false)), _ => None, } } opendal-0.52.0/src/services/d1/mod.rs000064400000000000000000000020171046102023000153750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-d1")] mod error; #[cfg(feature = "services-d1")] mod model; #[cfg(feature = "services-d1")] mod backend; #[cfg(feature = "services-d1")] pub use backend::D1Builder as D1; mod config; pub use config::D1Config; opendal-0.52.0/src/services/d1/model.rs000064400000000000000000000071221046102023000157200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use bytes::Bytes; use serde::Deserialize; use serde::Serialize; use serde_json::Map; use serde_json::Value; use crate::Buffer; use crate::Error; /// response data from d1 #[derive(Deserialize, Debug)] pub struct D1Response { pub result: Vec, pub success: bool, pub errors: Vec, } impl D1Response { pub fn parse(bs: &Bytes) -> Result { let response: D1Response = serde_json::from_slice(bs).map_err(|e| { Error::new( crate::ErrorKind::Unexpected, format!("failed to parse error response: {}", e), ) })?; if !response.success { return Err(Error::new( crate::ErrorKind::Unexpected, String::from_utf8_lossy(bs), )); } Ok(response) } pub fn get_result(&self, key: &str) -> Option { if self.result.is_empty() || self.result[0].results.is_empty() { return None; } let result = &self.result[0].results[0]; let value = result.get(key); match value { Some(Value::Array(s)) => { let mut v = Vec::new(); for i in s { if let Value::Number(n) = i { v.push(n.as_u64().unwrap() as u8); } } Some(Buffer::from(v)) } _ => None, } } } #[derive(Deserialize, Debug)] pub struct D1Result { pub results: Vec>, } #[derive(Clone, Deserialize, Debug, Serialize)] pub struct D1Error { pub message: String, pub code: i32, } #[cfg(test)] mod test { use super::*; #[test] fn test_deserialize_get_object_json_response() { let data = r#" { "result": [ { "results": [ { "CustomerId": "4", "CompanyName": "Around the Horn", "ContactName": "Thomas Hardy" } ], "success": true, "meta": { "served_by": "v3-prod", "duration": 0.2147, "changes": 0, "last_row_id": 0, "changed_db": false, "size_after": 2162688, "rows_read": 3, "rows_written": 2 } } ], "success": true, "errors": [], "messages": [] }"#; let response: D1Response = serde_json::from_str(data).unwrap(); println!("{:?}", response.result[0].results[0]); } } opendal-0.52.0/src/services/dashmap/backend.rs000064400000000000000000000103701046102023000173170ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use dashmap::DashMap; use crate::raw::adapters::typed_kv; use crate::raw::Access; use crate::services::DashmapConfig; use crate::*; impl Configurator for DashmapConfig { type Builder = DashmapBuilder; fn into_builder(self) -> Self::Builder { DashmapBuilder { config: self } } } /// [dashmap](https://github.com/xacrimon/dashmap) backend support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct DashmapBuilder { config: DashmapConfig, } impl DashmapBuilder { /// Set the root for dashmap. pub fn root(mut self, path: &str) -> Self { self.config.root = if path.is_empty() { None } else { Some(path.to_string()) }; self } } impl Builder for DashmapBuilder { const SCHEME: Scheme = Scheme::Dashmap; type Config = DashmapConfig; fn build(self) -> Result { let mut backend = DashmapBackend::new(Adapter { inner: DashMap::default(), }); if let Some(v) = self.config.root { backend = backend.with_root(&v); } Ok(backend) } } /// Backend is used to serve `Accessor` support in dashmap. pub type DashmapBackend = typed_kv::Backend; #[derive(Clone)] pub struct Adapter { inner: DashMap, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("DashmapAdapter") .field("size", &self.inner.len()) .finish_non_exhaustive() } } impl typed_kv::Adapter for Adapter { fn info(&self) -> typed_kv::Info { typed_kv::Info::new( Scheme::Dashmap, &format!("{:?}", &self.inner as *const _), typed_kv::Capability { get: true, set: true, scan: true, delete: true, shared: false, }, ) } async fn get(&self, path: &str) -> Result> { self.blocking_get(path) } fn blocking_get(&self, path: &str) -> Result> { match self.inner.get(path) { None => Ok(None), Some(bs) => Ok(Some(bs.value().to_owned())), } } async fn set(&self, path: &str, value: typed_kv::Value) -> Result<()> { self.blocking_set(path, value) } fn blocking_set(&self, path: &str, value: typed_kv::Value) -> Result<()> { self.inner.insert(path.to_string(), value); Ok(()) } async fn delete(&self, path: &str) -> Result<()> { self.blocking_delete(path) } fn blocking_delete(&self, path: &str) -> Result<()> { self.inner.remove(path); Ok(()) } async fn scan(&self, path: &str) -> Result> { self.blocking_scan(path) } fn blocking_scan(&self, path: &str) -> Result> { let keys = self.inner.iter().map(|kv| kv.key().to_string()); if path.is_empty() { Ok(keys.collect()) } else { Ok(keys.filter(|k| k.starts_with(path)).collect()) } } } #[cfg(test)] mod tests { use super::*; #[test] fn test_accessor_metadata_name() { let b1 = DashmapBuilder::default().build().unwrap(); assert_eq!(b1.info().name(), b1.info().name()); let b2 = DashmapBuilder::default().build().unwrap(); assert_ne!(b1.info().name(), b2.info().name()) } } opendal-0.52.0/src/services/dashmap/config.rs000064400000000000000000000021211046102023000171700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// [dashmap](https://github.com/xacrimon/dashmap) backend support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] pub struct DashmapConfig { /// The root path for dashmap. pub root: Option, } opendal-0.52.0/src/services/dashmap/docs.md000064400000000000000000000002571046102023000166370ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] list - [ ] presign - [ ] blocking opendal-0.52.0/src/services/dashmap/mod.rs000064400000000000000000000017211046102023000165070ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-dashmap")] mod backend; #[cfg(feature = "services-dashmap")] pub use backend::DashmapBuilder as Dashmap; mod config; pub use config::DashmapConfig; opendal-0.52.0/src/services/dbfs/backend.rs000064400000000000000000000202001046102023000166110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::StatusCode; use log::debug; use serde::Deserialize; use super::core::DbfsCore; use super::delete::DbfsDeleter; use super::error::parse_error; use super::lister::DbfsLister; use super::writer::DbfsWriter; use crate::raw::*; use crate::services::DbfsConfig; use crate::*; impl Configurator for DbfsConfig { type Builder = DbfsBuilder; fn into_builder(self) -> Self::Builder { DbfsBuilder { config: self } } } /// [Dbfs](https://docs.databricks.com/api/azure/workspace/dbfs)'s REST API support. #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct DbfsBuilder { config: DbfsConfig, } impl Debug for DbfsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("DbfsBuilder"); ds.field("config", &self.config); ds.finish() } } impl DbfsBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set endpoint of this backend. /// /// Endpoint must be full uri, e.g. /// /// - Azure: `https://adb-1234567890123456.78.azuredatabricks.net` /// - Aws: `https://dbc-123a5678-90bc.cloud.databricks.com` pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.trim_end_matches('/').to_string()) }; self } /// Set the token of this backend. pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()); } self } } impl Builder for DbfsBuilder { const SCHEME: Scheme = Scheme::Dbfs; type Config = DbfsConfig; /// Build a DbfsBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let endpoint = match &self.config.endpoint { Some(endpoint) => Ok(endpoint.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Dbfs)), }?; debug!("backend use endpoint: {}", &endpoint); let token = match self.config.token { Some(token) => token, None => { return Err(Error::new( ErrorKind::ConfigInvalid, "missing token for Dbfs", )); } }; let client = HttpClient::new()?; Ok(DbfsBackend { core: Arc::new(DbfsCore { root, endpoint: endpoint.to_string(), token, client, }), }) } } /// Backend for DBFS service #[derive(Debug, Clone)] pub struct DbfsBackend { core: Arc, } impl Access for DbfsBackend { type Reader = (); type Writer = oio::OneShotWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Dbfs) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, write: true, create_dir: true, delete: true, rename: true, list: true, list_has_last_modified: true, list_has_content_length: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let resp = self.core.dbfs_create_dir(path).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(RpCreateDir::default()), _ => Err(parse_error(resp)), } } async fn stat(&self, path: &str, _: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let resp = self.core.dbfs_get_status(path).await?; let status = resp.status(); match status { StatusCode::OK => { let mut meta = parse_into_metadata(path, resp.headers())?; let bs = resp.into_body(); let decoded_response: DbfsStatus = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; meta.set_last_modified(parse_datetime_from_from_timestamp_millis( decoded_response.modification_time, )?); match decoded_response.is_dir { true => meta.set_mode(EntryMode::DIR), false => { meta.set_mode(EntryMode::FILE); meta.set_content_length(decoded_response.file_size as u64) } }; Ok(RpStat::new(meta)) } StatusCode::NOT_FOUND if path.ends_with('/') => { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } _ => Err(parse_error(resp)), } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { Ok(( RpWrite::default(), oio::OneShotWriter::new(DbfsWriter::new(self.core.clone(), args, path.to_string())), )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(DbfsDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, _args: OpList) -> Result<(RpList, Self::Lister)> { let l = DbfsLister::new(self.core.clone(), path.to_string()); Ok((RpList::default(), oio::PageLister::new(l))) } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { self.core.dbfs_ensure_parent_path(to).await?; let resp = self.core.dbfs_rename(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpRename::default()), _ => Err(parse_error(resp)), } } } #[derive(Deserialize)] struct DbfsStatus { // Not used fields. // path: String, is_dir: bool, file_size: i64, modification_time: i64, } opendal-0.52.0/src/services/dbfs/config.rs000064400000000000000000000031011046102023000164700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// [Dbfs](https://docs.databricks.com/api/azure/workspace/dbfs)'s REST API support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] pub struct DbfsConfig { /// The root for dbfs. pub root: Option, /// The endpoint for dbfs. pub endpoint: Option, /// The token for dbfs. pub token: Option, } impl Debug for DbfsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("DbfsConfig"); ds.field("root", &self.root); ds.field("endpoint", &self.endpoint); if self.token.is_some() { ds.field("token", &""); } ds.finish() } } opendal-0.52.0/src/services/dbfs/core.rs000064400000000000000000000141371046102023000161660ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use base64::prelude::BASE64_STANDARD; use base64::Engine; use bytes::Bytes; use http::header; use http::Request; use http::Response; use http::StatusCode; use serde_json::json; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct DbfsCore { pub root: String, pub endpoint: String, pub token: String, pub client: HttpClient, } impl Debug for DbfsCore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("DbfsCore") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("token", &self.token) .finish_non_exhaustive() } } impl DbfsCore { pub async fn dbfs_create_dir(&self, path: &str) -> Result> { let url = format!("{}/api/2.0/dbfs/mkdirs", self.endpoint); let mut req = Request::post(&url); let auth_header_content = format!("Bearer {}", self.token); req = req.header(header::AUTHORIZATION, auth_header_content); let p = build_rooted_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let req_body = &json!({ "path": percent_encode_path(&p), }); let body = Buffer::from(Bytes::from(req_body.to_string())); let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn dbfs_delete(&self, path: &str) -> Result> { let url = format!("{}/api/2.0/dbfs/delete", self.endpoint); let mut req = Request::post(&url); let auth_header_content = format!("Bearer {}", self.token); req = req.header(header::AUTHORIZATION, auth_header_content); let p = build_rooted_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let request_body = &json!({ "path": percent_encode_path(&p), // TODO: support recursive toggle, should we add a new field in OpDelete? "recursive": true, }); let body = Buffer::from(Bytes::from(request_body.to_string())); let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn dbfs_rename(&self, from: &str, to: &str) -> Result> { let source = build_rooted_abs_path(&self.root, from); let target = build_rooted_abs_path(&self.root, to); let url = format!("{}/api/2.0/dbfs/move", self.endpoint); let mut req = Request::post(&url); let auth_header_content = format!("Bearer {}", self.token); req = req.header(header::AUTHORIZATION, auth_header_content); let req_body = &json!({ "source_path": percent_encode_path(&source), "destination_path": percent_encode_path(&target), }); let body = Buffer::from(Bytes::from(req_body.to_string())); let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn dbfs_list(&self, path: &str) -> Result> { let p = build_rooted_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/api/2.0/dbfs/list?path={}", self.endpoint, percent_encode_path(&p) ); let mut req = Request::get(&url); let auth_header_content = format!("Bearer {}", self.token); req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub fn dbfs_create_file_request(&self, path: &str, body: Bytes) -> Result> { let url = format!("{}/api/2.0/dbfs/put", self.endpoint); let contents = BASE64_STANDARD.encode(body); let mut req = Request::post(&url); let auth_header_content = format!("Bearer {}", self.token); req = req.header(header::AUTHORIZATION, auth_header_content); let req_body = &json!({ "path": path, "contents": contents, "overwrite": true, }); let body = Buffer::from(Bytes::from(req_body.to_string())); req.body(body).map_err(new_request_build_error) } pub async fn dbfs_get_status(&self, path: &str) -> Result> { let p = build_rooted_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/api/2.0/dbfs/get-status?path={}", &self.endpoint, percent_encode_path(&p) ); let mut req = Request::get(&url); let auth_header_content = format!("Bearer {}", self.token); req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn dbfs_ensure_parent_path(&self, path: &str) -> Result<()> { let resp = self.dbfs_get_status(path).await?; match resp.status() { StatusCode::OK => return Ok(()), StatusCode::NOT_FOUND => { self.dbfs_create_dir(path).await?; } _ => return Err(parse_error(resp)), } Ok(()) } } opendal-0.52.0/src/services/dbfs/delete.rs000064400000000000000000000027151046102023000164770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct DbfsDeleter { core: Arc, } impl DbfsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for DbfsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.dbfs_delete(&path).await?; let status = resp.status(); match status { // NOTE: Server will return 200 even if the path doesn't exist. StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/dbfs/docs.md000064400000000000000000000027441046102023000161430ustar 00000000000000This service will visit the [DBFS API](https://docs.databricks.com/api/azure/workspace/dbfs) supported by [Databricks File System](https://docs.databricks.com/en/dbfs/index.html). ## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [x] rename - [x] list - [ ] ~~presign~~ - [ ] blocking ## Configurations - `root`: Set the work directory for backend. - `endpoint`: Set the endpoint for backend. - `token`: Databricks personal access token. Refer to [`DbfsBuilder`]'s public API docs for more information. ## Examples ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Dbfs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Dbfs::default() // set the root for Dbfs, all operations will happen under this root // // Note: // if the root is not exists, the builder will automatically create the // root directory for you // if the root exists and is a directory, the builder will continue working // if the root exists and is a folder, the builder will fail on building backend .root("/path/to/dir") // set the endpoint of Dbfs workspace .endpoint("https://adb-1234567890123456.78.azuredatabricks.net") // set the personal access token for builder .token("access_token"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/dbfs/error.rs000064400000000000000000000046331046102023000163670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use http::Response; use http::StatusCode; use serde::Deserialize; use crate::raw::*; use crate::*; /// DbfsError is the error returned by DBFS service. #[derive(Default, Deserialize)] struct DbfsError { error_code: String, message: String, } impl Debug for DbfsError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("DbfsError"); de.field("error_code", &self.error_code); // replace `\n` to ` ` for better reading. de.field("message", &self.message.replace('\n', " ")); de.finish() } } pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED => (ErrorKind::ConditionNotMatch, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = match serde_json::from_slice::(&bs) { Ok(dbfs_error) => format!("{:?}", dbfs_error.message), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/dbfs/lister.rs000064400000000000000000000057731046102023000165460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use serde::Deserialize; use super::core::DbfsCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct DbfsLister { core: Arc, path: String, } impl DbfsLister { pub fn new(core: Arc, path: String) -> Self { Self { core, path } } } impl oio::PageList for DbfsLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let response = self.core.dbfs_list(&self.path).await?; let status_code = response.status(); if !status_code.is_success() { if status_code == StatusCode::NOT_FOUND { ctx.done = true; return Ok(()); } let error = parse_error(response); return Err(error); } let bytes = response.into_body(); let decoded_response: DbfsOutputList = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; ctx.done = true; for status in decoded_response.files { let entry: oio::Entry = match status.is_dir { true => { let normalized_path = format!("{}/", &status.path); let mut meta = Metadata::new(EntryMode::DIR); meta.set_last_modified(parse_datetime_from_from_timestamp_millis( status.modification_time, )?); oio::Entry::new(&normalized_path, meta) } false => { let mut meta = Metadata::new(EntryMode::FILE); meta.set_last_modified(parse_datetime_from_from_timestamp_millis( status.modification_time, )?); meta.set_content_length(status.file_size as u64); oio::Entry::new(&status.path, meta) } }; ctx.entries.push_back(entry); } Ok(()) } } #[derive(Debug, Deserialize)] struct DbfsOutputList { files: Vec, } #[derive(Debug, Deserialize)] struct DbfsStatus { path: String, is_dir: bool, file_size: i64, modification_time: i64, } opendal-0.52.0/src/services/dbfs/mod.rs000064400000000000000000000022461046102023000160130ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-dbfs")] mod core; #[cfg(feature = "services-dbfs")] mod delete; #[cfg(feature = "services-dbfs")] mod error; #[cfg(feature = "services-dbfs")] mod lister; #[cfg(feature = "services-dbfs")] mod writer; #[cfg(feature = "services-dbfs")] mod backend; #[cfg(feature = "services-dbfs")] pub use backend::DbfsBuilder as Dbfs; mod config; pub use config::DbfsConfig; opendal-0.52.0/src/services/dbfs/writer.rs000064400000000000000000000036721046102023000165540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::DbfsCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct DbfsWriter { core: Arc, path: String, } impl DbfsWriter { const MAX_SIMPLE_SIZE: usize = 1024 * 1024; pub fn new(core: Arc, _op: OpWrite, path: String) -> Self { DbfsWriter { core, path } } } impl oio::OneShotWrite for DbfsWriter { async fn write_once(&self, bs: Buffer) -> Result { let size = bs.len(); // MAX_BLOCK_SIZE_EXCEEDED will be thrown if this limit(1MB) is exceeded. if size >= Self::MAX_SIMPLE_SIZE { return Err(Error::new( ErrorKind::Unsupported, "AppendWrite has not been implemented for Dbfs", )); } let req = self .core .dbfs_create_file_request(&self.path, bs.to_bytes())?; let resp = self.core.client.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/dropbox/backend.rs000064400000000000000000000164071046102023000173660ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::sync::Arc; use bytes::Buf; use http::Response; use http::StatusCode; use super::core::*; use super::delete::DropboxDeleter; use super::error::*; use super::lister::DropboxLister; use super::writer::DropboxWriter; use crate::raw::*; use crate::*; #[derive(Clone, Debug)] pub struct DropboxBackend { pub core: Arc, } impl Access for DropboxBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut ma = AccessorInfo::default(); ma.set_scheme(Scheme::Dropbox) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_last_modified: true, stat_has_content_length: true, read: true, write: true, create_dir: true, delete: true, list: true, list_with_recursive: true, list_has_last_modified: true, list_has_content_length: true, copy: true, rename: true, shared: true, ..Default::default() }); ma.into() } async fn create_dir(&self, path: &str, _args: OpCreateDir) -> Result { // Check if the folder already exists. let resp = self.core.dropbox_get_metadata(path).await?; if StatusCode::OK == resp.status() { let bytes = resp.into_body(); let decoded_response: DropboxMetadataResponse = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; if "folder" == decoded_response.tag { return Ok(RpCreateDir::default()); } if "file" == decoded_response.tag { return Err(Error::new( ErrorKind::NotADirectory, format!("it's not a directory {}", path), )); } } let res = self.core.dropbox_create_folder(path).await?; Ok(res) } async fn stat(&self, path: &str, _: OpStat) -> Result { let resp = self.core.dropbox_get_metadata(path).await?; let status = resp.status(); match status { StatusCode::OK => { let bytes = resp.into_body(); let decoded_response: DropboxMetadataResponse = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; let entry_mode: EntryMode = match decoded_response.tag.as_str() { "file" => EntryMode::FILE, "folder" => EntryMode::DIR, _ => EntryMode::Unknown, }; let mut metadata = Metadata::new(entry_mode); // Only set last_modified and size if entry_mode is FILE, because Dropbox API // returns last_modified and size only for files. // FYI: https://www.dropbox.com/developers/documentation/http/documentation#files-get_metadata if entry_mode == EntryMode::FILE { let date_utc_last_modified = parse_datetime_from_rfc3339(&decoded_response.client_modified)?; metadata.set_last_modified(date_utc_last_modified); if let Some(size) = decoded_response.size { metadata.set_content_length(size); } else { return Err(Error::new( ErrorKind::Unexpected, format!("no size found for file {}", path), )); } } Ok(RpStat::new(metadata)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.dropbox_get(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { Ok(( RpWrite::default(), oio::OneShotWriter::new(DropboxWriter::new( self.core.clone(), args, String::from(path), )), )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(DropboxDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { Ok(( RpList::default(), oio::PageLister::new(DropboxLister::new( self.core.clone(), path.to_string(), args.recursive(), args.limit(), )), )) } async fn copy(&self, from: &str, to: &str, _: OpCopy) -> Result { let resp = self.core.dropbox_copy(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => { let err = parse_error(resp); match err.kind() { ErrorKind::NotFound => Ok(RpCopy::default()), _ => Err(err), } } } } async fn rename(&self, from: &str, to: &str, _: OpRename) -> Result { let resp = self.core.dropbox_move(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpRename::default()), _ => { let err = parse_error(resp); match err.kind() { ErrorKind::NotFound => Ok(RpRename::default()), _ => Err(err), } } } } } opendal-0.52.0/src/services/dropbox/builder.rs000064400000000000000000000143661046102023000174270ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use chrono::DateTime; use chrono::Utc; use tokio::sync::Mutex; use super::backend::DropboxBackend; use super::core::DropboxCore; use super::core::DropboxSigner; use crate::raw::*; use crate::services::DropboxConfig; use crate::*; impl Configurator for DropboxConfig { type Builder = DropboxBuilder; fn into_builder(self) -> Self::Builder { DropboxBuilder { config: self, http_client: None, } } } /// [Dropbox](https://www.dropbox.com/) backend support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct DropboxBuilder { config: DropboxConfig, http_client: Option, } impl Debug for DropboxBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Builder") .field("root", &self.config.root) .finish() } } impl DropboxBuilder { /// Set the root directory for dropbox. /// /// Default to `/` if not set. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Access token is used for temporary access to the Dropbox API. /// /// You can get the access token from [Dropbox App Console](https://www.dropbox.com/developers/apps) /// /// NOTE: this token will be expired in 4 hours. /// If you are trying to use the Dropbox service in a long time, please set a refresh_token instead. pub fn access_token(mut self, access_token: &str) -> Self { self.config.access_token = Some(access_token.to_string()); self } /// Refresh token is used for long term access to the Dropbox API. /// /// You can get the refresh token via OAuth 2.0 Flow of Dropbox. /// /// OpenDAL will use this refresh token to get a new access token when the old one is expired. pub fn refresh_token(mut self, refresh_token: &str) -> Self { self.config.refresh_token = Some(refresh_token.to_string()); self } /// Set the client id for Dropbox. /// /// This is required for OAuth 2.0 Flow to refresh the access token. pub fn client_id(mut self, client_id: &str) -> Self { self.config.client_id = Some(client_id.to_string()); self } /// Set the client secret for Dropbox. /// /// This is required for OAuth 2.0 Flow with refresh the access token. pub fn client_secret(mut self, client_secret: &str) -> Self { self.config.client_secret = Some(client_secret.to_string()); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, http_client: HttpClient) -> Self { self.http_client = Some(http_client); self } } impl Builder for DropboxBuilder { const SCHEME: Scheme = Scheme::Dropbox; type Config = DropboxConfig; fn build(self) -> Result { let root = normalize_root(&self.config.root.unwrap_or_default()); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Dropbox) })? }; let signer = match (self.config.access_token, self.config.refresh_token) { (Some(access_token), None) => DropboxSigner { access_token, // We will never expire user specified token. expires_in: DateTime::::MAX_UTC, ..Default::default() }, (None, Some(refresh_token)) => { let client_id = self.config.client_id.ok_or_else(|| { Error::new( ErrorKind::ConfigInvalid, "client_id must be set when refresh_token is set", ) .with_context("service", Scheme::Dropbox) })?; let client_secret = self.config.client_secret.ok_or_else(|| { Error::new( ErrorKind::ConfigInvalid, "client_secret must be set when refresh_token is set", ) .with_context("service", Scheme::Dropbox) })?; DropboxSigner { refresh_token, client_id, client_secret, ..Default::default() } } (Some(_), Some(_)) => { return Err(Error::new( ErrorKind::ConfigInvalid, "access_token and refresh_token can not be set at the same time", ) .with_context("service", Scheme::Dropbox)) } (None, None) => { return Err(Error::new( ErrorKind::ConfigInvalid, "access_token or refresh_token must be set", ) .with_context("service", Scheme::Dropbox)) } }; Ok(DropboxBackend { core: Arc::new(DropboxCore { root, signer: Arc::new(Mutex::new(signer)), client, }), }) } } opendal-0.52.0/src/services/dropbox/config.rs000064400000000000000000000031741046102023000172410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for [Dropbox](https://www.dropbox.com/) backend support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct DropboxConfig { /// root path for dropbox. pub root: Option, /// access token for dropbox. pub access_token: Option, /// refresh_token for dropbox. pub refresh_token: Option, /// client_id for dropbox. pub client_id: Option, /// client_secret for dropbox. pub client_secret: Option, } impl Debug for DropboxConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("DropBoxConfig") .field("root", &self.root) .finish_non_exhaustive() } } opendal-0.52.0/src/services/dropbox/core.rs000064400000000000000000000375541046102023000167350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::default::Default; use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use bytes::Bytes; use chrono::DateTime; use chrono::Utc; use http::header; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use serde::Serialize; use tokio::sync::Mutex; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct DropboxCore { pub root: String, pub client: HttpClient, pub signer: Arc>, } impl Debug for DropboxCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("DropboxCore") .field("root", &self.root) .finish() } } impl DropboxCore { fn build_path(&self, path: &str) -> String { let path = build_rooted_abs_path(&self.root, path); // For dropbox, even the path is a directory, // we still need to remove the trailing slash. path.trim_end_matches('/').to_string() } pub async fn sign(&self, req: &mut Request) -> Result<()> { let mut signer = self.signer.lock().await; // Access token is valid, use it directly. if !signer.access_token.is_empty() && signer.expires_in > Utc::now() { let value = format!("Bearer {}", signer.access_token) .parse() .expect("token must be valid header value"); req.headers_mut().insert(header::AUTHORIZATION, value); return Ok(()); } // Refresh invalid token. let url = "https://api.dropboxapi.com/oauth2/token".to_string(); let content = format!( "grant_type=refresh_token&refresh_token={}&client_id={}&client_secret={}", signer.refresh_token, signer.client_id, signer.client_secret ); let bs = Bytes::from(content); let request = Request::post(&url) .header(CONTENT_TYPE, "application/x-www-form-urlencoded") .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; let resp = self.client.send(request).await?; let body = resp.into_body(); let token: DropboxTokenResponse = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; // Update signer after token refreshed. signer.access_token.clone_from(&token.access_token); // Refresh it 2 minutes earlier. signer.expires_in = Utc::now() + chrono::TimeDelta::try_seconds(token.expires_in as i64) .expect("expires_in must be valid seconds") - chrono::TimeDelta::try_seconds(120).expect("120 must be valid seconds"); let value = format!("Bearer {}", token.access_token) .parse() .expect("token must be valid header value"); req.headers_mut().insert(header::AUTHORIZATION, value); Ok(()) } pub async fn dropbox_get( &self, path: &str, range: BytesRange, _: &OpRead, ) -> Result> { let url: String = "https://content.dropboxapi.com/2/files/download".to_string(); let download_args = DropboxDownloadArgs { path: build_rooted_abs_path(&self.root, path), }; let request_payload = serde_json::to_string(&download_args).map_err(new_json_serialize_error)?; let mut req = Request::post(&url) .header("Dropbox-API-Arg", request_payload) .header(CONTENT_LENGTH, 0); if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } let mut request = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut request).await?; self.client.fetch(request).await } pub async fn dropbox_update( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let url = "https://content.dropboxapi.com/2/files/upload".to_string(); let dropbox_update_args = DropboxUploadArgs { path: build_rooted_abs_path(&self.root, path), ..Default::default() }; let mut request_builder = Request::post(&url); if let Some(size) = size { request_builder = request_builder.header(CONTENT_LENGTH, size); } request_builder = request_builder.header( CONTENT_TYPE, args.content_type().unwrap_or("application/octet-stream"), ); let mut request = request_builder .header( "Dropbox-API-Arg", serde_json::to_string(&dropbox_update_args).map_err(new_json_serialize_error)?, ) .body(body) .map_err(new_request_build_error)?; self.sign(&mut request).await?; self.client.send(request).await } pub async fn dropbox_delete(&self, path: &str) -> Result> { let url = "https://api.dropboxapi.com/2/files/delete_v2".to_string(); let args = DropboxDeleteArgs { path: self.build_path(path), }; let bs = Bytes::from(serde_json::to_string(&args).map_err(new_json_serialize_error)?); let mut request = Request::post(&url) .header(CONTENT_TYPE, "application/json") .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; self.sign(&mut request).await?; self.client.send(request).await } pub async fn dropbox_create_folder(&self, path: &str) -> Result { let url = "https://api.dropboxapi.com/2/files/create_folder_v2".to_string(); let args = DropboxCreateFolderArgs { path: self.build_path(path), }; let bs = Bytes::from(serde_json::to_string(&args).map_err(new_json_serialize_error)?); let mut request = Request::post(&url) .header(CONTENT_TYPE, "application/json") .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; self.sign(&mut request).await?; let resp = self.client.send(request).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCreateDir::default()), _ => { let err = parse_error(resp); match err.kind() { ErrorKind::AlreadyExists => Ok(RpCreateDir::default()), _ => Err(err), } } } } pub async fn dropbox_list( &self, path: &str, recursive: bool, limit: Option, ) -> Result> { let url = "https://api.dropboxapi.com/2/files/list_folder".to_string(); // The default settings here align with the DropboxAPI default settings. // Refer: https://www.dropbox.com/developers/documentation/http/documentation#files-list_folder let args = DropboxListArgs { path: self.build_path(path), recursive, limit: limit.unwrap_or(1000), }; let bs = Bytes::from(serde_json::to_string(&args).map_err(new_json_serialize_error)?); let mut request = Request::post(&url) .header(CONTENT_TYPE, "application/json") .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; self.sign(&mut request).await?; self.client.send(request).await } pub async fn dropbox_list_continue(&self, cursor: &str) -> Result> { let url = "https://api.dropboxapi.com/2/files/list_folder/continue".to_string(); let args = DropboxListContinueArgs { cursor: cursor.to_string(), }; let bs = Bytes::from(serde_json::to_string(&args).map_err(new_json_serialize_error)?); let mut request = Request::post(&url) .header(CONTENT_TYPE, "application/json") .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; self.sign(&mut request).await?; self.client.send(request).await } pub async fn dropbox_copy(&self, from: &str, to: &str) -> Result> { let url = "https://api.dropboxapi.com/2/files/copy_v2".to_string(); let args = DropboxCopyArgs { from_path: self.build_path(from), to_path: self.build_path(to), }; let bs = Bytes::from(serde_json::to_string(&args).map_err(new_json_serialize_error)?); let mut request = Request::post(&url) .header(CONTENT_TYPE, "application/json") .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; self.sign(&mut request).await?; self.client.send(request).await } pub async fn dropbox_move(&self, from: &str, to: &str) -> Result> { let url = "https://api.dropboxapi.com/2/files/move_v2".to_string(); let args = DropboxMoveArgs { from_path: self.build_path(from), to_path: self.build_path(to), }; let bs = Bytes::from(serde_json::to_string(&args).map_err(new_json_serialize_error)?); let mut request = Request::post(&url) .header(CONTENT_TYPE, "application/json") .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; self.sign(&mut request).await?; self.client.send(request).await } pub async fn dropbox_get_metadata(&self, path: &str) -> Result> { let url = "https://api.dropboxapi.com/2/files/get_metadata".to_string(); let args = DropboxMetadataArgs { path: self.build_path(path), ..Default::default() }; let bs = Bytes::from(serde_json::to_string(&args).map_err(new_json_serialize_error)?); let mut request = Request::post(&url) .header(CONTENT_TYPE, "application/json") .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; self.sign(&mut request).await?; self.client.send(request).await } } #[derive(Clone)] pub struct DropboxSigner { pub client_id: String, pub client_secret: String, pub refresh_token: String, pub access_token: String, pub expires_in: DateTime, } impl Default for DropboxSigner { fn default() -> Self { DropboxSigner { refresh_token: String::new(), client_id: String::new(), client_secret: String::new(), access_token: String::new(), expires_in: DateTime::::MIN_UTC, } } } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxDownloadArgs { path: String, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxUploadArgs { path: String, mode: String, mute: bool, autorename: bool, strict_conflict: bool, } impl Default for DropboxUploadArgs { fn default() -> Self { DropboxUploadArgs { mode: "overwrite".to_string(), path: "".to_string(), mute: true, autorename: false, strict_conflict: false, } } } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxDeleteArgs { path: String, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxDeleteBatchEntry { path: String, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxDeleteBatchArgs { entries: Vec, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxDeleteBatchCheckArgs { async_job_id: String, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxCreateFolderArgs { path: String, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxListArgs { path: String, recursive: bool, limit: usize, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxListContinueArgs { cursor: String, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxCopyArgs { from_path: String, to_path: String, } #[derive(Clone, Debug, Deserialize, Serialize)] struct DropboxMoveArgs { from_path: String, to_path: String, } #[derive(Default, Clone, Debug, Deserialize, Serialize)] struct DropboxMetadataArgs { include_deleted: bool, include_has_explicit_shared_members: bool, include_media_info: bool, path: String, } #[derive(Clone, Deserialize)] struct DropboxTokenResponse { access_token: String, expires_in: usize, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxMetadataResponse { #[serde(rename(deserialize = ".tag"))] pub tag: String, pub client_modified: String, pub content_hash: Option, pub file_lock_info: Option, pub has_explicit_shared_members: Option, pub id: String, pub is_downloadable: Option, pub name: String, pub path_display: String, pub path_lower: String, pub property_groups: Option>, pub rev: Option, pub server_modified: Option, pub sharing_info: Option, pub size: Option, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxMetadataFileLockInfo { pub created: Option, pub is_lockholder: bool, pub lockholder_name: Option, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxMetadataPropertyGroup { pub fields: Vec, pub template_id: String, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxMetadataPropertyGroupField { pub name: String, pub value: String, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxMetadataSharingInfo { pub modified_by: Option, pub parent_shared_folder_id: Option, pub read_only: Option, pub shared_folder_id: Option, pub traverse_only: Option, pub no_access: Option, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxListResponse { pub entries: Vec, pub cursor: String, pub has_more: bool, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxDeleteBatchResponse { #[serde(rename(deserialize = ".tag"))] pub tag: String, pub async_job_id: Option, pub entries: Option>, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxDeleteBatchResponseEntry { #[serde(rename(deserialize = ".tag"))] pub tag: String, pub metadata: Option, } #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxDeleteBatchFailureResponseCause { #[serde(rename(deserialize = ".tag"))] pub tag: String, } opendal-0.52.0/src/services/dropbox/delete.rs000064400000000000000000000031031046102023000172260ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct DropboxDeleter { core: Arc, } impl DropboxDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for DropboxDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.dropbox_delete(&path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => { let err = parse_error(resp); match err.kind() { ErrorKind::NotFound => Ok(()), _ => Err(err), } } } } } opendal-0.52.0/src/services/dropbox/docs.md000064400000000000000000000027201046102023000166740ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [x] batch - [ ] blocking ## Configuration - `root`: Set the work directory for this backend. ### Credentials related #### Just provide Access Token (Temporary) - `access_token`: set the access_token for this backend. Please notice its expiration. #### Or provide Client ID and Client Secret and refresh token (Long Term) If you want to let OpenDAL to refresh the access token automatically, please provide the following fields: - `refresh_token`: set the refresh_token for dropbox api - `client_id`: set the client_id for dropbox api - `client_secret`: set the client_secret for dropbox api OpenDAL is a library, it cannot do the first step of OAuth2 for you. You need to get authorization code from user by calling Dropbox's authorize url and exchange it for refresh token. Please refer to [Dropbox OAuth2 Guide](https://www.dropbox.com/developers/reference/oauth-guide) for more information. You can refer to [`DropboxBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::raw::OpWrite; use opendal::services::Dropbox; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Dropbox::default() .root("/opendal") .access_token(""); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/dropbox/error.rs000064400000000000000000000061301046102023000171200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use serde::Deserialize; use crate::raw::*; use crate::*; #[derive(Default, Debug, Deserialize)] #[serde(default)] pub struct DropboxErrorResponse { pub error_summary: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (mut kind, mut retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::TOO_MANY_REQUESTS => (ErrorKind::RateLimited, true), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, dropbox_err) = serde_json::from_slice::(&bs) .map(|dropbox_err| (format!("{dropbox_err:?}"), Some(dropbox_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); if let Some(dropbox_err) = dropbox_err { (kind, retryable) = parse_dropbox_error_summary(&dropbox_err.error_summary).unwrap_or((kind, retryable)); } let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } /// We cannot get the error type from the response header when the status code is 409. /// Because Dropbox API v2 will put error summary in the response body, /// we need to parse it to get the correct error type and then error kind. /// /// See pub fn parse_dropbox_error_summary(summary: &str) -> Option<(ErrorKind, bool)> { if summary.starts_with("path/not_found") || summary.starts_with("path_lookup/not_found") || summary.starts_with("from_lookup/not_found") { Some((ErrorKind::NotFound, false)) } else if summary.starts_with("path/conflict") { Some((ErrorKind::AlreadyExists, false)) } else if summary.starts_with("too_many_write_operations") { Some((ErrorKind::RateLimited, true)) } else { None } } opendal-0.52.0/src/services/dropbox/lister.rs000064400000000000000000000072251046102023000172770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct DropboxLister { core: Arc, path: String, recursive: bool, limit: Option, } impl DropboxLister { pub fn new( core: Arc, path: String, recursive: bool, limit: Option, ) -> Self { Self { core, path, recursive, limit, } } } impl oio::PageList for DropboxLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { // The token is set when obtaining entries and returning `has_more` flag. // When the token exists, we should retrieve more entries using the Dropbox continue API. // Refer: https://www.dropbox.com/developers/documentation/http/documentation#files-list_folder-continue let response = if !ctx.token.is_empty() { self.core.dropbox_list_continue(&ctx.token).await? } else { self.core .dropbox_list(&self.path, self.recursive, self.limit) .await? }; let status_code = response.status(); if !status_code.is_success() { let error = parse_error(response); let result = match error.kind() { ErrorKind::NotFound => Ok(()), _ => Err(error), }; ctx.done = true; return result; } let bytes = response.into_body(); let decoded_response: DropboxListResponse = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; for entry in decoded_response.entries { let entry_mode = match entry.tag.as_str() { "file" => EntryMode::FILE, "folder" => EntryMode::DIR, _ => EntryMode::Unknown, }; let mut name = entry.name; let mut meta = Metadata::new(entry_mode); // Dropbox will return folder names that do not end with '/'. if entry_mode == EntryMode::DIR && !name.ends_with('/') { name.push('/'); } // The behavior here aligns with Dropbox's stat function. if entry_mode == EntryMode::FILE { let date_utc_last_modified = parse_datetime_from_rfc3339(&entry.client_modified)?; meta.set_last_modified(date_utc_last_modified); if let Some(size) = entry.size { meta.set_content_length(size); } } ctx.entries.push_back(oio::Entry::with(name, meta)); } if decoded_response.has_more { ctx.token = decoded_response.cursor; ctx.done = false; } else { ctx.done = true; } Ok(()) } } opendal-0.52.0/src/services/dropbox/mod.rs000064400000000000000000000023661046102023000165550ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-dropbox")] mod backend; #[cfg(feature = "services-dropbox")] mod core; #[cfg(feature = "services-dropbox")] mod delete; #[cfg(feature = "services-dropbox")] mod error; #[cfg(feature = "services-dropbox")] mod lister; #[cfg(feature = "services-dropbox")] mod writer; #[cfg(feature = "services-dropbox")] mod builder; #[cfg(feature = "services-dropbox")] pub use builder::DropboxBuilder as Dropbox; mod config; pub use config::DropboxConfig; opendal-0.52.0/src/services/dropbox/writer.rs000064400000000000000000000030731046102023000173060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::DropboxCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct DropboxWriter { core: Arc, op: OpWrite, path: String, } impl DropboxWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { DropboxWriter { core, op, path } } } impl oio::OneShotWrite for DropboxWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self .core .dropbox_update(&self.path, Some(bs.len()), &self.op, bs) .await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/etcd/backend.rs000064400000000000000000000230671046102023000166300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::vec; use bb8::PooledConnection; use bb8::RunError; use etcd_client::Certificate; use etcd_client::Client; use etcd_client::ConnectOptions; use etcd_client::Error as EtcdError; use etcd_client::GetOptions; use etcd_client::Identity; use etcd_client::TlsOptions; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::EtcdConfig; use crate::*; const DEFAULT_ETCD_ENDPOINTS: &str = "http://127.0.0.1:2379"; impl Configurator for EtcdConfig { type Builder = EtcdBuilder; fn into_builder(self) -> Self::Builder { EtcdBuilder { config: self } } } /// [Etcd](https://etcd.io/) services support. #[doc = include_str!("docs.md")] #[derive(Clone, Default)] pub struct EtcdBuilder { config: EtcdConfig, } impl Debug for EtcdBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Builder"); ds.field("config", &self.config); ds.finish() } } impl EtcdBuilder { /// set the network address of etcd service. /// /// default: "http://127.0.0.1:2379" pub fn endpoints(mut self, endpoints: &str) -> Self { if !endpoints.is_empty() { self.config.endpoints = Some(endpoints.to_owned()); } self } /// set the username for etcd /// /// default: no username pub fn username(mut self, username: &str) -> Self { if !username.is_empty() { self.config.username = Some(username.to_owned()); } self } /// set the password for etcd /// /// default: no password pub fn password(mut self, password: &str) -> Self { if !password.is_empty() { self.config.password = Some(password.to_owned()); } self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the certificate authority file path. /// /// default is None pub fn ca_path(mut self, ca_path: &str) -> Self { if !ca_path.is_empty() { self.config.ca_path = Some(ca_path.to_string()) } self } /// Set the certificate file path. /// /// default is None pub fn cert_path(mut self, cert_path: &str) -> Self { if !cert_path.is_empty() { self.config.cert_path = Some(cert_path.to_string()) } self } /// Set the key file path. /// /// default is None pub fn key_path(mut self, key_path: &str) -> Self { if !key_path.is_empty() { self.config.key_path = Some(key_path.to_string()) } self } } impl Builder for EtcdBuilder { const SCHEME: Scheme = Scheme::Etcd; type Config = EtcdConfig; fn build(self) -> Result { let endpoints = self .config .endpoints .clone() .unwrap_or_else(|| DEFAULT_ETCD_ENDPOINTS.to_string()); let endpoints: Vec = endpoints.split(',').map(|s| s.to_string()).collect(); let mut options = ConnectOptions::new(); if self.config.ca_path.is_some() && self.config.cert_path.is_some() && self.config.key_path.is_some() { let ca = self.load_pem(self.config.ca_path.clone().unwrap().as_str())?; let key = self.load_pem(self.config.key_path.clone().unwrap().as_str())?; let cert = self.load_pem(self.config.cert_path.clone().unwrap().as_str())?; let tls_options = TlsOptions::default() .ca_certificate(Certificate::from_pem(ca)) .identity(Identity::from_pem(cert, key)); options = options.with_tls(tls_options); } if let Some(username) = self.config.username.clone() { options = options.with_user( username, self.config.password.clone().unwrap_or("".to_string()), ); } let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); let client = OnceCell::new(); Ok(EtcdBackend::new(Adapter { endpoints, client, options, }) .with_normalized_root(root)) } } impl EtcdBuilder { fn load_pem(&self, path: &str) -> Result { std::fs::read_to_string(path) .map_err(|err| Error::new(ErrorKind::Unexpected, "invalid file path").set_source(err)) } } /// Backend for etcd services. pub type EtcdBackend = kv::Backend; #[derive(Clone)] pub struct Manager { endpoints: Vec, options: ConnectOptions, } #[async_trait::async_trait] impl bb8::ManageConnection for Manager { type Connection = Client; type Error = Error; async fn connect(&self) -> Result { let conn = Client::connect(self.endpoints.clone(), Some(self.options.clone())) .await .map_err(format_etcd_error)?; Ok(conn) } async fn is_valid(&self, conn: &mut Self::Connection) -> Result<(), Self::Error> { let _ = conn.status().await.map_err(format_etcd_error)?; Ok(()) } /// Always allow reuse conn. fn has_broken(&self, _: &mut Self::Connection) -> bool { false } } #[derive(Clone)] pub struct Adapter { endpoints: Vec, options: ConnectOptions, client: OnceCell>, } // implement `Debug` manually, or password may be leaked. impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.field("endpoints", &self.endpoints.join(",")); ds.field("options", &self.options.clone()); ds.finish() } } impl Adapter { async fn conn(&self) -> Result> { let client = self .client .get_or_try_init(|| async { bb8::Pool::builder() .max_size(64) .build(Manager { endpoints: self.endpoints.clone(), options: self.options.clone(), }) .await }) .await?; client.get_owned().await.map_err(|err| match err { RunError::User(err) => err, RunError::TimedOut => { Error::new(ErrorKind::Unexpected, "connection request: timeout").set_temporary() } }) } } impl kv::Adapter for Adapter { type Scanner = kv::ScanStdIter>>; fn info(&self) -> kv::Info { kv::Info::new( Scheme::Etcd, &self.endpoints.join(","), Capability { read: true, write: true, list: true, shared: true, ..Default::default() }, ) } async fn get(&self, key: &str) -> Result> { let mut client = self.conn().await?; let resp = client.get(key, None).await.map_err(format_etcd_error)?; if let Some(kv) = resp.kvs().first() { Ok(Some(Buffer::from(kv.value().to_vec()))) } else { Ok(None) } } async fn set(&self, key: &str, value: Buffer) -> Result<()> { let mut client = self.conn().await?; let _ = client .put(key, value.to_vec(), None) .await .map_err(format_etcd_error)?; Ok(()) } async fn delete(&self, key: &str) -> Result<()> { let mut client = self.conn().await?; let _ = client.delete(key, None).await.map_err(format_etcd_error)?; Ok(()) } async fn scan(&self, path: &str) -> Result { let mut client = self.conn().await?; let get_options = Some(GetOptions::new().with_prefix().with_keys_only()); let resp = client .get(path, get_options) .await .map_err(format_etcd_error)?; let mut res = Vec::default(); for kv in resp.kvs() { let v = kv.key_str().map(String::from).map_err(|err| { Error::new(ErrorKind::Unexpected, "store key is not valid utf-8 string") .set_source(err) })?; res.push(Ok(v)); } Ok(kv::ScanStdIter::new(res.into_iter())) } } pub fn format_etcd_error(e: EtcdError) -> Error { Error::new(ErrorKind::Unexpected, e.to_string().as_str()) .set_source(e) .set_temporary() } opendal-0.52.0/src/services/etcd/config.rs000064400000000000000000000056411046102023000165040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Etcd services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct EtcdConfig { /// network address of the Etcd services. /// If use https, must set TLS options: `ca_path`, `cert_path`, `key_path`. /// e.g. "127.0.0.1:23790,127.0.0.1:23791,127.0.0.1:23792" or "http://127.0.0.1:23790,http://127.0.0.1:23791,http://127.0.0.1:23792" or "https://127.0.0.1:23790,https://127.0.0.1:23791,https://127.0.0.1:23792" /// /// default is "http://127.0.0.1:2379" pub endpoints: Option, /// the username to connect etcd service. /// /// default is None pub username: Option, /// the password for authentication /// /// default is None pub password: Option, /// the working directory of the etcd service. Can be "/path/to/dir" /// /// default is "/" pub root: Option, /// certificate authority file path /// /// default is None pub ca_path: Option, /// cert path /// /// default is None pub cert_path: Option, /// key path /// /// default is None pub key_path: Option, } impl Debug for EtcdConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("EtcdConfig"); ds.field("root", &self.root); if let Some(endpoints) = self.endpoints.clone() { ds.field("endpoints", &endpoints); } if let Some(username) = self.username.clone() { ds.field("username", &username); } if self.password.is_some() { ds.field("password", &""); } if let Some(ca_path) = self.ca_path.clone() { ds.field("ca_path", &ca_path); } if let Some(cert_path) = self.cert_path.clone() { ds.field("cert_path", &cert_path); } if let Some(key_path) = self.key_path.clone() { ds.field("key_path", &key_path); } ds.finish() } } opendal-0.52.0/src/services/etcd/docs.md000064400000000000000000000017271046102023000161440ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `endpoints`: Set the network address of etcd servers - `username`: Set the username of Etcd - `password`: Set the password for authentication - `ca_path`: Set the ca path to the etcd connection - `cert_path`: Set the cert path to the etcd connection - `key_path`: Set the key path to the etcd connection You can refer to [`EtcdBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Etcd; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Etcd::default(); // this will build a Operator accessing etcd which runs on http://127.0.0.1:2379 let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/etcd/mod.rs000064400000000000000000000017021046102023000160100ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-etcd")] mod backend; #[cfg(feature = "services-etcd")] pub use backend::EtcdBuilder as Etcd; mod config; pub use config::EtcdConfig; opendal-0.52.0/src/services/foundationdb/backend.rs000064400000000000000000000122451046102023000203610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use foundationdb::api::NetworkAutoStop; use foundationdb::Database; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::FoundationdbConfig; use crate::Builder; use crate::Error; use crate::ErrorKind; use crate::Scheme; use crate::*; impl Configurator for FoundationdbConfig { type Builder = FoundationdbBuilder; fn into_builder(self) -> Self::Builder { FoundationdbBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct FoundationdbBuilder { config: FoundationdbConfig, } impl FoundationdbBuilder { /// Set the root for Foundationdb. pub fn root(mut self, path: &str) -> Self { self.config.root = Some(path.into()); self } /// Set the config path for Foundationdb. If not set, will fallback to use default pub fn config_path(mut self, path: &str) -> Self { self.config.config_path = Some(path.into()); self } } impl Builder for FoundationdbBuilder { const SCHEME: Scheme = Scheme::Foundationdb; type Config = FoundationdbConfig; fn build(self) -> Result { let _network = Arc::new(unsafe { foundationdb::boot() }); let db; if let Some(cfg_path) = &self.config.config_path { db = Database::from_path(cfg_path).map_err(|e| { Error::new(ErrorKind::ConfigInvalid, "open foundation db") .with_context("service", Scheme::Foundationdb) .set_source(e) })?; } else { db = Database::default().map_err(|e| { Error::new(ErrorKind::ConfigInvalid, "open foundation db") .with_context("service", Scheme::Foundationdb) .set_source(e) })? } let db = Arc::new(db); let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); Ok(FoundationdbBackend::new(Adapter { db, _network }).with_normalized_root(root)) } } /// Backend for Foundationdb services. pub type FoundationdbBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { db: Arc, _network: Arc, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.finish() } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Foundationdb, "foundationdb", Capability { read: true, write: true, delete: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let transaction = self.db.create_trx().expect("Unable to create transaction"); match transaction.get(path.as_bytes(), false).await { Ok(slice) => match slice { Some(data) => Ok(Some(Buffer::from(data.to_vec()))), None => Err(Error::new( ErrorKind::NotFound, "foundationdb: key not found", )), }, Err(_) => Err(Error::new( ErrorKind::NotFound, "foundationdb: key not found", )), } } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let transaction = self.db.create_trx().expect("Unable to create transaction"); transaction.set(path.as_bytes(), &value.to_vec()); match transaction.commit().await { Ok(_) => Ok(()), Err(e) => Err(parse_transaction_commit_error(e)), } } async fn delete(&self, path: &str) -> Result<()> { let transaction = self.db.create_trx().expect("Unable to create transaction"); transaction.clear(path.as_bytes()); match transaction.commit().await { Ok(_) => Ok(()), Err(e) => Err(parse_transaction_commit_error(e)), } } } fn parse_transaction_commit_error(e: foundationdb::TransactionCommitError) -> Error { Error::new(ErrorKind::Unexpected, e.to_string().as_str()) .with_context("service", Scheme::Foundationdb) } opendal-0.52.0/src/services/foundationdb/config.rs000064400000000000000000000027761046102023000202470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// [foundationdb](https://www.foundationdb.org/) service support. ///Config for FoundationDB. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct FoundationdbConfig { ///root of the backend. pub root: Option, ///config_path for the backend. pub config_path: Option, } impl Debug for FoundationdbConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("FoundationConfig"); ds.field("root", &self.root); ds.field("config_path", &self.config_path); ds.finish() } } opendal-0.52.0/src/services/foundationdb/docs.md000064400000000000000000000020501046102023000176670ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking **Note**: As for [Known Limitations - FoundationDB](https://apple.github.io/foundationdb/known-limitations), keys cannot exceed 10,000 bytes in size, and values cannot exceed 100,000 bytes in size. Errors will be raised by OpenDAL if these limits are exceeded. ## Configuration - `root`: Set the work directory for this backend. - `config_path`: Set the configuration path for foundationdb. If not provided, the default configuration path will be used. You can refer to [`FoundationdbBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Foundationdb; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Foundationdb::default() .config_path("/etc/foundationdb/foundationdb.conf"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/foundationdb/mod.rs000064400000000000000000000017521046102023000175520ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-foundationdb")] mod backend; #[cfg(feature = "services-foundationdb")] pub use backend::FoundationdbBuilder as Foundationdb; mod config; pub use config::FoundationdbConfig; opendal-0.52.0/src/services/fs/backend.rs000064400000000000000000000451641046102023000163230ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io::SeekFrom; use std::path::Path; use std::path::PathBuf; use std::sync::Arc; use chrono::DateTime; use log::debug; use super::core::*; use super::delete::FsDeleter; use super::lister::FsLister; use super::reader::FsReader; use super::writer::FsWriter; use super::writer::FsWriters; use crate::raw::*; use crate::services::FsConfig; use crate::*; impl Configurator for FsConfig { type Builder = FsBuilder; fn into_builder(self) -> Self::Builder { FsBuilder { config: self } } } /// POSIX file system support. #[doc = include_str!("docs.md")] #[derive(Default, Debug)] pub struct FsBuilder { config: FsConfig, } impl FsBuilder { /// Set root for backend. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set temp dir for atomic write. /// /// # Notes /// /// - When append is enabled, we will not use atomic write /// to avoid data loss and performance issue. pub fn atomic_write_dir(mut self, dir: &str) -> Self { if !dir.is_empty() { self.config.atomic_write_dir = Some(dir.to_string()); } self } } impl Builder for FsBuilder { const SCHEME: Scheme = Scheme::Fs; type Config = FsConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = match self.config.root.map(PathBuf::from) { Some(root) => Ok(root), None => Err(Error::new( ErrorKind::ConfigInvalid, "root is not specified", )), }?; debug!("backend use root {}", root.to_string_lossy()); // If root dir is not exist, we must create it. if let Err(e) = std::fs::metadata(&root) { if e.kind() == std::io::ErrorKind::NotFound { std::fs::create_dir_all(&root).map_err(|e| { Error::new(ErrorKind::Unexpected, "create root dir failed") .with_operation("Builder::build") .with_context("root", root.to_string_lossy()) .set_source(e) })?; } } let atomic_write_dir = self.config.atomic_write_dir.map(PathBuf::from); // If atomic write dir is not exist, we must create it. if let Some(d) = &atomic_write_dir { if let Err(e) = std::fs::metadata(d) { if e.kind() == std::io::ErrorKind::NotFound { std::fs::create_dir_all(d).map_err(|e| { Error::new(ErrorKind::Unexpected, "create atomic write dir failed") .with_operation("Builder::build") .with_context("atomic_write_dir", d.to_string_lossy()) .set_source(e) })?; } } } // Canonicalize the root directory. This should work since we already know that we can // get the metadata of the path. let root = root.canonicalize().map_err(|e| { Error::new( ErrorKind::Unexpected, "canonicalize of root directory failed", ) .with_operation("Builder::build") .with_context("root", root.to_string_lossy()) .set_source(e) })?; // Canonicalize the atomic_write_dir directory. This should work since we already know that // we can get the metadata of the path. let atomic_write_dir = atomic_write_dir .map(|p| { p.canonicalize().map(Some).map_err(|e| { Error::new( ErrorKind::Unexpected, "canonicalize of atomic_write_dir directory failed", ) .with_operation("Builder::build") .with_context("root", root.to_string_lossy()) .set_source(e) }) }) .unwrap_or(Ok(None))?; Ok(FsBackend { core: Arc::new(FsCore { root, atomic_write_dir, buf_pool: oio::PooledBuf::new(16).with_initial_capacity(256 * 1024), }), }) } } /// Backend is used to serve `Accessor` support for posix-like fs. #[derive(Debug, Clone)] pub struct FsBackend { core: Arc, } impl Access for FsBackend { type Reader = FsReader; type Writer = FsWriters; type Lister = Option>; type Deleter = oio::OneShotDeleter; type BlockingReader = FsReader; type BlockingWriter = FsWriter; type BlockingLister = Option>; type BlockingDeleter = oio::OneShotDeleter; fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Fs) .set_root(&self.core.root.to_string_lossy()) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, write: true, write_can_empty: true, write_can_append: true, write_can_multi: true, write_with_if_not_exists: true, write_has_last_modified: true, create_dir: true, delete: true, list: true, copy: true, rename: true, blocking: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let p = self.core.root.join(path.trim_end_matches('/')); tokio::fs::create_dir_all(&p) .await .map_err(new_std_io_error)?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _: OpStat) -> Result { let p = self.core.root.join(path.trim_end_matches('/')); let meta = tokio::fs::metadata(&p).await.map_err(new_std_io_error)?; let mode = if meta.is_dir() { EntryMode::DIR } else if meta.is_file() { EntryMode::FILE } else { EntryMode::Unknown }; let m = Metadata::new(mode) .with_content_length(meta.len()) .with_last_modified( meta.modified() .map(DateTime::from) .map_err(new_std_io_error)?, ); Ok(RpStat::new(m)) } /// # Notes /// /// There are three ways to get the total file length: /// /// - call std::fs::metadata directly and then open. (400ns) /// - open file first, and then use `f.metadata()` (300ns) /// - open file first, and then use `seek`. (100ns) /// /// Benchmark could be found [here](https://gist.github.com/Xuanwo/48f9cfbc3022ea5f865388bb62e1a70f) async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let p = self.core.root.join(path.trim_end_matches('/')); let mut f = tokio::fs::OpenOptions::new() .read(true) .open(&p) .await .map_err(new_std_io_error)?; if args.range().offset() != 0 { use tokio::io::AsyncSeekExt; f.seek(SeekFrom::Start(args.range().offset())) .await .map_err(new_std_io_error)?; } let r = FsReader::new( self.core.clone(), f, args.range().size().unwrap_or(u64::MAX) as _, ); Ok((RpRead::new(), r)) } async fn write(&self, path: &str, op: OpWrite) -> Result<(RpWrite, Self::Writer)> { let (target_path, tmp_path) = if let Some(atomic_write_dir) = &self.core.atomic_write_dir { let target_path = self .core .ensure_write_abs_path(&self.core.root, path) .await?; let tmp_path = self .core .ensure_write_abs_path(atomic_write_dir, &tmp_file_of(path)) .await?; // If the target file exists, we should append to the end of it directly. if op.append() && tokio::fs::try_exists(&target_path) .await .map_err(new_std_io_error)? { (target_path, None) } else { (target_path, Some(tmp_path)) } } else { let p = self .core .ensure_write_abs_path(&self.core.root, path) .await?; (p, None) }; let mut open_options = tokio::fs::OpenOptions::new(); if op.if_not_exists() { open_options.create_new(true); } else { open_options.create(true); } open_options.write(true); if op.append() { open_options.append(true); } else { open_options.truncate(true); } let f = open_options .open(tmp_path.as_ref().unwrap_or(&target_path)) .await .map_err(|e| { match e.kind() { std::io::ErrorKind::AlreadyExists => { // Map io AlreadyExists to opendal ConditionNotMatch Error::new( ErrorKind::ConditionNotMatch, "The file already exists in the filesystem", ) .set_source(e) } _ => new_std_io_error(e), } })?; let w = FsWriter::new(target_path, tmp_path, f); let w = if op.append() { FsWriters::One(w) } else { FsWriters::Two(oio::PositionWriter::new( w, op.executor().cloned(), op.concurrent(), )) }; Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(FsDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, _: OpList) -> Result<(RpList, Self::Lister)> { let p = self.core.root.join(path.trim_end_matches('/')); let f = match tokio::fs::read_dir(&p).await { Ok(rd) => rd, Err(e) => { return if e.kind() == std::io::ErrorKind::NotFound { Ok((RpList::default(), None)) } else { Err(new_std_io_error(e)) }; } }; let rd = FsLister::new(&self.core.root, path, f); Ok((RpList::default(), Some(rd))) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let from = self.core.root.join(from.trim_end_matches('/')); // try to get the metadata of the source file to ensure it exists tokio::fs::metadata(&from).await.map_err(new_std_io_error)?; let to = self .core .ensure_write_abs_path(&self.core.root, to.trim_end_matches('/')) .await?; tokio::fs::copy(from, to).await.map_err(new_std_io_error)?; Ok(RpCopy::default()) } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let from = self.core.root.join(from.trim_end_matches('/')); // try to get the metadata of the source file to ensure it exists tokio::fs::metadata(&from).await.map_err(new_std_io_error)?; let to = self .core .ensure_write_abs_path(&self.core.root, to.trim_end_matches('/')) .await?; tokio::fs::rename(from, to) .await .map_err(new_std_io_error)?; Ok(RpRename::default()) } fn blocking_create_dir(&self, path: &str, _: OpCreateDir) -> Result { let p = self.core.root.join(path.trim_end_matches('/')); std::fs::create_dir_all(p).map_err(new_std_io_error)?; Ok(RpCreateDir::default()) } fn blocking_stat(&self, path: &str, _: OpStat) -> Result { let p = self.core.root.join(path.trim_end_matches('/')); let meta = std::fs::metadata(p).map_err(new_std_io_error)?; let mode = if meta.is_dir() { EntryMode::DIR } else if meta.is_file() { EntryMode::FILE } else { EntryMode::Unknown }; let m = Metadata::new(mode) .with_content_length(meta.len()) .with_last_modified( meta.modified() .map(DateTime::from) .map_err(new_std_io_error)?, ); Ok(RpStat::new(m)) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let p = self.core.root.join(path.trim_end_matches('/')); let mut f = std::fs::OpenOptions::new() .read(true) .open(p) .map_err(new_std_io_error)?; if args.range().offset() != 0 { use std::io::Seek; f.seek(SeekFrom::Start(args.range().offset())) .map_err(new_std_io_error)?; } let r = FsReader::new( self.core.clone(), f, args.range().size().unwrap_or(u64::MAX) as _, ); Ok((RpRead::new(), r)) } fn blocking_write(&self, path: &str, op: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let (target_path, tmp_path) = if let Some(atomic_write_dir) = &self.core.atomic_write_dir { let target_path = self .core .blocking_ensure_write_abs_path(&self.core.root, path)?; let tmp_path = self .core .blocking_ensure_write_abs_path(atomic_write_dir, &tmp_file_of(path))?; // If the target file exists, we should append to the end of it directly. if op.append() && Path::new(&target_path) .try_exists() .map_err(new_std_io_error)? { (target_path, None) } else { (target_path, Some(tmp_path)) } } else { let p = self .core .blocking_ensure_write_abs_path(&self.core.root, path)?; (p, None) }; let mut f = std::fs::OpenOptions::new(); if op.if_not_exists() { f.create_new(true); } else { f.create(true); } f.write(true); if op.append() { f.append(true); } else { f.truncate(true); } let f = f .open(tmp_path.as_ref().unwrap_or(&target_path)) .map_err(|e| { match e.kind() { std::io::ErrorKind::AlreadyExists => { // Map io AlreadyExists to opendal ConditionNotMatch Error::new( ErrorKind::ConditionNotMatch, "The file already exists in the filesystem", ) .set_source(e) } _ => new_std_io_error(e), } })?; Ok((RpWrite::new(), FsWriter::new(target_path, tmp_path, f))) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(FsDeleter::new(self.core.clone())), )) } fn blocking_list(&self, path: &str, _: OpList) -> Result<(RpList, Self::BlockingLister)> { let p = self.core.root.join(path.trim_end_matches('/')); let f = match std::fs::read_dir(p) { Ok(rd) => rd, Err(e) => { return if e.kind() == std::io::ErrorKind::NotFound { Ok((RpList::default(), None)) } else { Err(new_std_io_error(e)) }; } }; let rd = FsLister::new(&self.core.root, path, f); Ok((RpList::default(), Some(rd))) } fn blocking_copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let from = self.core.root.join(from.trim_end_matches('/')); // try to get the metadata of the source file to ensure it exists std::fs::metadata(&from).map_err(new_std_io_error)?; let to = self .core .blocking_ensure_write_abs_path(&self.core.root, to.trim_end_matches('/'))?; std::fs::copy(from, to).map_err(new_std_io_error)?; Ok(RpCopy::default()) } fn blocking_rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let from = self.core.root.join(from.trim_end_matches('/')); // try to get the metadata of the source file to ensure it exists std::fs::metadata(&from).map_err(new_std_io_error)?; let to = self .core .blocking_ensure_write_abs_path(&self.core.root, to.trim_end_matches('/'))?; std::fs::rename(from, to).map_err(new_std_io_error)?; Ok(RpRename::default()) } } #[cfg(test)] mod tests { use super::*; #[test] fn test_tmp_file_of() { let cases = vec![ ("hello.txt", "hello.txt"), ("/tmp/opendal.log", "opendal.log"), ("/abc/def/hello.parquet", "hello.parquet"), ]; for (path, expected_prefix) in cases { let tmp_file = tmp_file_of(path); assert!(tmp_file.len() > expected_prefix.len()); assert!(tmp_file.starts_with(expected_prefix)); } } } opendal-0.52.0/src/services/fs/config.rs000064400000000000000000000022151046102023000161670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// config for file system #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct FsConfig { /// root dir for backend pub root: Option, /// tmp dir for atomic write pub atomic_write_dir: Option, } opendal-0.52.0/src/services/fs/core.rs000064400000000000000000000061501046102023000156540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::path::Path; use std::path::PathBuf; use uuid::Uuid; use crate::raw::*; use crate::*; #[derive(Debug)] pub struct FsCore { pub root: PathBuf, pub atomic_write_dir: Option, pub buf_pool: oio::PooledBuf, } impl FsCore { // Synchronously build write path and ensure the parent dirs created pub fn blocking_ensure_write_abs_path(&self, parent: &Path, path: &str) -> Result { let p = parent.join(path); // Create dir before write path. // // TODO(xuanwo): There are many works to do here: // - Is it safe to create dir concurrently? // - Do we need to extract this logic as new util functions? // - Is it better to check the parent dir exists before call mkdir? let parent = PathBuf::from(&p) .parent() .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "path should have parent but not, it must be malformed", ) .with_context("input", p.to_string_lossy()) })? .to_path_buf(); std::fs::create_dir_all(parent).map_err(new_std_io_error)?; Ok(p) } // Build write path and ensure the parent dirs created pub async fn ensure_write_abs_path(&self, parent: &Path, path: &str) -> Result { let p = parent.join(path); // Create dir before write path. // // TODO(xuanwo): There are many works to do here: // - Is it safe to create dir concurrently? // - Do we need to extract this logic as new util functions? // - Is it better to check the parent dir exists before call mkdir? let parent = PathBuf::from(&p) .parent() .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "path should have parent but not, it must be malformed", ) .with_context("input", p.to_string_lossy()) })? .to_path_buf(); tokio::fs::create_dir_all(&parent) .await .map_err(new_std_io_error)?; Ok(p) } } #[inline] pub fn tmp_file_of(path: &str) -> String { let name = get_basename(path); let uuid = Uuid::new_v4().to_string(); format!("{name}.{uuid}") } opendal-0.52.0/src/services/fs/delete.rs000064400000000000000000000045551046102023000161750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use crate::raw::*; use crate::*; use std::sync::Arc; pub struct FsDeleter { core: Arc, } impl FsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for FsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = self.core.root.join(path.trim_end_matches('/')); let meta = tokio::fs::metadata(&p).await; match meta { Ok(meta) => { if meta.is_dir() { tokio::fs::remove_dir(&p).await.map_err(new_std_io_error)?; } else { tokio::fs::remove_file(&p).await.map_err(new_std_io_error)?; } Ok(()) } Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()), Err(err) => Err(new_std_io_error(err)), } } } impl oio::BlockingOneShotDelete for FsDeleter { fn blocking_delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = self.core.root.join(path.trim_end_matches('/')); let meta = std::fs::metadata(&p); match meta { Ok(meta) => { if meta.is_dir() { std::fs::remove_dir(&p).map_err(new_std_io_error)?; } else { std::fs::remove_file(&p).map_err(new_std_io_error)?; } Ok(()) } Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()), Err(err) => Err(new_std_io_error(err)), } } } opendal-0.52.0/src/services/fs/docs.md000064400000000000000000000015631046102023000156330ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] append - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [ ] ~~presign~~ - [x] blocking ## Configuration - `root`: Set the work dir for backend. - You can refer to [`FsBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Fs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create fs backend builder. let mut builder = Fs::default() // Set the root for fs, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/tmp"); // `Accessor` provides the low level APIs, we will use `Operator` normally. let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/fs/lister.rs000064400000000000000000000074721046102023000162360ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::path::Path; use std::path::PathBuf; use crate::raw::*; use crate::EntryMode; use crate::Metadata; use crate::Result; pub struct FsLister

{ root: PathBuf, current_path: Option, rd: P, } impl

FsLister

{ pub fn new(root: &Path, path: &str, rd: P) -> Self { Self { root: root.to_owned(), current_path: Some(path.to_string()), rd, } } } /// # Safety /// /// We will only take `&mut Self` reference for FsLister. unsafe impl

Sync for FsLister

{} impl oio::List for FsLister { async fn next(&mut self) -> Result> { // since list should return path itself, we return it first if let Some(path) = self.current_path.take() { let e = oio::Entry::new(path.as_str(), Metadata::new(EntryMode::DIR)); return Ok(Some(e)); } let Some(de) = self.rd.next_entry().await.map_err(new_std_io_error)? else { return Ok(None); }; let entry_path = de.path(); let rel_path = normalize_path( &entry_path .strip_prefix(&self.root) .expect("cannot fail because the prefix is iterated") .to_string_lossy() .replace('\\', "/"), ); let ft = de.file_type().await.map_err(new_std_io_error)?; let entry = if ft.is_dir() { // Make sure we are returning the correct path. oio::Entry::new(&format!("{rel_path}/"), Metadata::new(EntryMode::DIR)) } else if ft.is_file() { oio::Entry::new(&rel_path, Metadata::new(EntryMode::FILE)) } else { oio::Entry::new(&rel_path, Metadata::new(EntryMode::Unknown)) }; Ok(Some(entry)) } } impl oio::BlockingList for FsLister { fn next(&mut self) -> Result> { // since list should return path itself, we return it first if let Some(path) = self.current_path.take() { let e = oio::Entry::new(path.as_str(), Metadata::new(EntryMode::DIR)); return Ok(Some(e)); } let de = match self.rd.next() { Some(de) => de.map_err(new_std_io_error)?, None => return Ok(None), }; let entry_path = de.path(); let rel_path = normalize_path( &entry_path .strip_prefix(&self.root) .expect("cannot fail because the prefix is iterated") .to_string_lossy() .replace('\\', "/"), ); let ft = de.file_type().map_err(new_std_io_error)?; let entry = if ft.is_dir() { // Make sure we are returning the correct path. oio::Entry::new(&format!("{rel_path}/"), Metadata::new(EntryMode::DIR)) } else if ft.is_file() { oio::Entry::new(&rel_path, Metadata::new(EntryMode::FILE)) } else { oio::Entry::new(&rel_path, Metadata::new(EntryMode::Unknown)) }; Ok(Some(entry)) } } opendal-0.52.0/src/services/fs/mod.rs000064400000000000000000000022231046102023000155000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-fs")] mod core; #[cfg(feature = "services-fs")] mod delete; #[cfg(feature = "services-fs")] mod lister; #[cfg(feature = "services-fs")] mod reader; #[cfg(feature = "services-fs")] mod writer; #[cfg(feature = "services-fs")] mod backend; #[cfg(feature = "services-fs")] pub use backend::FsBuilder as Fs; mod config; pub use config::FsConfig; opendal-0.52.0/src/services/fs/reader.rs000064400000000000000000000065711046102023000161750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io::Read; use std::sync::Arc; use tokio::io::AsyncReadExt; use tokio::io::ReadBuf; use super::core::*; use crate::raw::*; use crate::*; pub struct FsReader { core: Arc, f: F, read: usize, size: usize, buf_size: usize, } impl FsReader { pub fn new(core: Arc, f: F, size: usize) -> Self { Self { core, f, read: 0, size, // Use 2 MiB as default value. buf_size: 2 * 1024 * 1024, } } } impl oio::Read for FsReader { async fn read(&mut self) -> Result { if self.read >= self.size { return Ok(Buffer::new()); } let mut bs = self.core.buf_pool.get(); bs.reserve(self.buf_size); let size = (self.size - self.read).min(self.buf_size); let buf = &mut bs.spare_capacity_mut()[..size]; let mut read_buf: ReadBuf = ReadBuf::uninit(buf); // SAFETY: Read at most `limit` bytes into `read_buf`. unsafe { read_buf.assume_init(size); } let n = self .f .read_buf(&mut read_buf) .await .map_err(new_std_io_error)?; self.read += n; // Safety: We make sure that bs contains `n` more bytes. let filled = read_buf.filled().len(); unsafe { bs.set_len(filled) } let frozen = bs.split().freeze(); // Return the buffer to the pool. self.core.buf_pool.put(bs); Ok(Buffer::from(frozen)) } } impl oio::BlockingRead for FsReader { fn read(&mut self) -> Result { if self.read >= self.size { return Ok(Buffer::new()); } let mut bs = self.core.buf_pool.get(); bs.reserve(self.buf_size); let size = (self.size - self.read).min(self.buf_size); let buf = &mut bs.spare_capacity_mut()[..size]; let mut read_buf: ReadBuf = ReadBuf::uninit(buf); // SAFETY: Read at most `limit` bytes into `read_buf`. unsafe { read_buf.assume_init(size); } let n = self .f .read(read_buf.initialize_unfilled()) .map_err(new_std_io_error)?; read_buf.advance(n); self.read += n; // Safety: We make sure that bs contains `n` more bytes. let filled = read_buf.filled().len(); unsafe { bs.set_len(filled) } let frozen = bs.split().freeze(); // Return the buffer to the pool. self.core.buf_pool.put(bs); Ok(Buffer::from(frozen)) } } opendal-0.52.0/src/services/fs/writer.rs000064400000000000000000000153371046102023000162470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fs::File; use std::io::Write; use std::path::PathBuf; use bytes::Buf; use tokio::io::AsyncWriteExt; use crate::raw::*; use crate::*; pub type FsWriters = TwoWays, oio::PositionWriter>>; pub struct FsWriter { target_path: PathBuf, tmp_path: Option, f: Option, } impl FsWriter { pub fn new(target_path: PathBuf, tmp_path: Option, f: F) -> Self { Self { target_path, tmp_path, f: Some(f), } } } /// # Safety /// /// We will only take `&mut Self` reference for FsWriter. unsafe impl Sync for FsWriter {} impl oio::Write for FsWriter { async fn write(&mut self, mut bs: Buffer) -> Result<()> { let f = self.f.as_mut().expect("FsWriter must be initialized"); while bs.has_remaining() { let n = f.write(bs.chunk()).await.map_err(new_std_io_error)?; bs.advance(n); } Ok(()) } async fn close(&mut self) -> Result { let f = self.f.as_mut().expect("FsWriter must be initialized"); f.flush().await.map_err(new_std_io_error)?; f.sync_all().await.map_err(new_std_io_error)?; if let Some(tmp_path) = &self.tmp_path { tokio::fs::rename(tmp_path, &self.target_path) .await .map_err(new_std_io_error)?; } let file_meta = f.metadata().await.map_err(new_std_io_error)?; let mode = if file_meta.is_file() { EntryMode::FILE } else if file_meta.is_dir() { EntryMode::DIR } else { EntryMode::Unknown }; let meta = Metadata::new(mode) .with_content_length(file_meta.len()) .with_last_modified(file_meta.modified().map_err(new_std_io_error)?.into()); Ok(meta) } async fn abort(&mut self) -> Result<()> { if let Some(tmp_path) = &self.tmp_path { tokio::fs::remove_file(tmp_path) .await .map_err(new_std_io_error) } else { Err(Error::new( ErrorKind::Unsupported, "Fs doesn't support abort if atomic_write_dir is not set", )) } } } impl oio::BlockingWrite for FsWriter { fn write(&mut self, mut bs: Buffer) -> Result<()> { let f = self.f.as_mut().expect("FsWriter must be initialized"); while bs.has_remaining() { let n = f.write(bs.chunk()).map_err(new_std_io_error)?; bs.advance(n); } Ok(()) } fn close(&mut self) -> Result { let f = self.f.as_mut().expect("FsWriter must be initialized"); f.sync_all().map_err(new_std_io_error)?; if let Some(tmp_path) = &self.tmp_path { std::fs::rename(tmp_path, &self.target_path).map_err(new_std_io_error)?; } let file_meta = f.metadata().map_err(new_std_io_error)?; let mode = if file_meta.is_file() { EntryMode::FILE } else if file_meta.is_dir() { EntryMode::DIR } else { EntryMode::Unknown }; let meta = Metadata::new(mode) .with_content_length(file_meta.len()) .with_last_modified(file_meta.modified().map_err(new_std_io_error)?.into()); Ok(meta) } } impl oio::PositionWrite for FsWriter { async fn write_all_at(&self, offset: u64, buf: Buffer) -> Result<()> { let f = self.f.as_ref().expect("FsWriter must be initialized"); let f = f .try_clone() .await .map_err(new_std_io_error)? .into_std() .await; tokio::task::spawn_blocking(move || { let mut buf = buf; let mut offset = offset; while !buf.is_empty() { match write_at(&f, buf.chunk(), offset) { Ok(n) => { buf.advance(n); offset += n as u64 } Err(e) => return Err(e), } } Ok(()) }) .await .map_err(new_task_join_error)? } async fn close(&self) -> Result { let f = self.f.as_ref().expect("FsWriter must be initialized"); let mut f = f .try_clone() .await .map_err(new_std_io_error)? .into_std() .await; f.flush().map_err(new_std_io_error)?; f.sync_all().map_err(new_std_io_error)?; if let Some(tmp_path) = &self.tmp_path { tokio::fs::rename(tmp_path, &self.target_path) .await .map_err(new_std_io_error)?; } let file_meta = f.metadata().map_err(new_std_io_error)?; let mode = if file_meta.is_file() { EntryMode::FILE } else if file_meta.is_dir() { EntryMode::DIR } else { EntryMode::Unknown }; let meta = Metadata::new(mode) .with_content_length(file_meta.len()) .with_last_modified(file_meta.modified().map_err(new_std_io_error)?.into()); Ok(meta) } async fn abort(&self) -> Result<()> { if let Some(tmp_path) = &self.tmp_path { tokio::fs::remove_file(tmp_path) .await .map_err(new_std_io_error) } else { Err(Error::new( ErrorKind::Unsupported, "Fs doesn't support abort if atomic_write_dir is not set", )) } } } #[cfg(windows)] fn write_at(f: &File, buf: &[u8], offset: u64) -> Result { use std::os::windows::fs::FileExt; f.seek_write(buf, offset).map_err(new_std_io_error) } #[cfg(unix)] fn write_at(f: &File, buf: &[u8], offset: u64) -> Result { use std::os::unix::fs::FileExt; f.write_at(buf, offset).map_err(new_std_io_error) } opendal-0.52.0/src/services/ftp/backend.rs000064400000000000000000000341431046102023000164770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::str; use std::str::FromStr; use std::sync::Arc; use async_tls::TlsConnector; use bb8::PooledConnection; use bb8::RunError; use http::Uri; use log::debug; use suppaftp::list::File; use suppaftp::types::FileType; use suppaftp::types::Response; use suppaftp::AsyncRustlsConnector; use suppaftp::AsyncRustlsFtpStream; use suppaftp::FtpError; use suppaftp::ImplAsyncFtpStream; use suppaftp::Status; use tokio::sync::OnceCell; use uuid::Uuid; use super::delete::FtpDeleter; use super::err::parse_error; use super::lister::FtpLister; use super::reader::FtpReader; use super::writer::FtpWriter; use crate::raw::*; use crate::services::FtpConfig; use crate::*; impl Configurator for FtpConfig { type Builder = FtpBuilder; fn into_builder(self) -> Self::Builder { FtpBuilder { config: self } } } /// FTP and FTPS services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct FtpBuilder { config: FtpConfig, } impl Debug for FtpBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("FtpBuilder") .field("config", &self.config) .finish() } } impl FtpBuilder { /// set endpoint for ftp backend. pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.to_string()) }; self } /// set root path for ftp backend. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// set user for ftp backend. pub fn user(mut self, user: &str) -> Self { self.config.user = if user.is_empty() { None } else { Some(user.to_string()) }; self } /// set password for ftp backend. pub fn password(mut self, password: &str) -> Self { self.config.password = if password.is_empty() { None } else { Some(password.to_string()) }; self } } impl Builder for FtpBuilder { const SCHEME: Scheme = Scheme::Ftp; type Config = FtpConfig; fn build(self) -> Result { debug!("ftp backend build started: {:?}", &self); let endpoint = match &self.config.endpoint { None => return Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty")), Some(v) => v, }; let endpoint_uri = match endpoint.parse::() { Err(e) => { return Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is invalid") .with_context("endpoint", endpoint) .set_source(e)); } Ok(uri) => uri, }; let host = endpoint_uri.host().unwrap_or("127.0.0.1"); let port = endpoint_uri.port_u16().unwrap_or(21); let endpoint = format!("{host}:{port}"); let enable_secure = match endpoint_uri.scheme_str() { Some("ftp") => false, // if the user forgot to add a scheme prefix // treat it as using secured scheme Some("ftps") | None => true, Some(s) => { return Err(Error::new( ErrorKind::ConfigInvalid, "endpoint is unsupported or invalid", ) .with_context("endpoint", s)); } }; let root = normalize_root(&self.config.root.unwrap_or_default()); let user = match &self.config.user { None => "".to_string(), Some(v) => v.clone(), }; let password = match &self.config.password { None => "".to_string(), Some(v) => v.clone(), }; Ok(FtpBackend { endpoint, root, user, password, enable_secure, pool: OnceCell::new(), }) } } pub struct Manager { endpoint: String, root: String, user: String, password: String, enable_secure: bool, } #[async_trait::async_trait] impl bb8::ManageConnection for Manager { type Connection = AsyncRustlsFtpStream; type Error = FtpError; async fn connect(&self) -> Result { let stream = ImplAsyncFtpStream::connect(&self.endpoint).await?; // switch to secure mode if ssl/tls is on. let mut ftp_stream = if self.enable_secure { stream .into_secure( AsyncRustlsConnector::from(TlsConnector::default()), &self.endpoint, ) .await? } else { stream }; // login if needed if !self.user.is_empty() { ftp_stream.login(&self.user, &self.password).await?; } // change to the root path match ftp_stream.cwd(&self.root).await { Err(FtpError::UnexpectedResponse(e)) if e.status == Status::FileUnavailable => { ftp_stream.mkdir(&self.root).await?; // Then change to root path ftp_stream.cwd(&self.root).await?; } // Other errors, return. Err(e) => return Err(e), // Do nothing if success. Ok(_) => (), } ftp_stream.transfer_type(FileType::Binary).await?; Ok(ftp_stream) } async fn is_valid(&self, conn: &mut Self::Connection) -> Result<(), Self::Error> { conn.noop().await } /// Don't allow reuse conn. /// /// We need to investigate why reuse conn will cause error. fn has_broken(&self, _: &mut Self::Connection) -> bool { true } } /// Backend is used to serve `Accessor` support for ftp. #[derive(Clone)] pub struct FtpBackend { endpoint: String, root: String, user: String, password: String, enable_secure: bool, pool: OnceCell>, } impl Debug for FtpBackend { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend").finish() } } impl Access for FtpBackend { type Reader = FtpReader; type Writer = FtpWriter; type Lister = FtpLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Ftp) .set_root(&self.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, write: true, write_can_multi: true, write_can_append: true, delete: true, create_dir: true, list: true, list_has_content_length: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let mut ftp_stream = self.ftp_connect(Operation::CreateDir).await?; let paths: Vec<&str> = path.split_inclusive('/').collect(); let mut curr_path = String::new(); for path in paths { curr_path.push_str(path); match ftp_stream.mkdir(&curr_path).await { // Do nothing if status is FileUnavailable or OK(()) is return. Err(FtpError::UnexpectedResponse(Response { status: Status::FileUnavailable, .. })) | Ok(()) => (), Err(e) => { return Err(parse_error(e)); } } } Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _: OpStat) -> Result { let file = self.ftp_stat(path).await?; let mode = if file.is_file() { EntryMode::FILE } else if file.is_directory() { EntryMode::DIR } else { EntryMode::Unknown }; let mut meta = Metadata::new(mode); meta.set_content_length(file.size() as u64); meta.set_last_modified(file.modified().into()); Ok(RpStat::new(meta)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let ftp_stream = self.ftp_connect(Operation::Read).await?; let reader = FtpReader::new(ftp_stream, path.to_string(), args).await?; Ok((RpRead::new(), reader)) } async fn write(&self, path: &str, op: OpWrite) -> Result<(RpWrite, Self::Writer)> { // Ensure the parent dir exists. let parent = get_parent(path); let paths: Vec<&str> = parent.split('/').collect(); // TODO: we can optimize this by checking dir existence first. let mut ftp_stream = self.ftp_connect(Operation::Write).await?; let mut curr_path = String::new(); for path in paths { if path.is_empty() { continue; } curr_path.push_str(path); curr_path.push('/'); match ftp_stream.mkdir(&curr_path).await { // Do nothing if status is FileUnavailable or OK(()) is return. Err(FtpError::UnexpectedResponse(Response { status: Status::FileUnavailable, .. })) | Ok(()) => (), Err(e) => { return Err(parse_error(e)); } } } let tmp_path = if op.append() { None } else { let uuid = Uuid::new_v4().to_string(); Some(format!("{}.{}", path, uuid)) }; let w = FtpWriter::new(ftp_stream, path.to_string(), tmp_path); Ok((RpWrite::new(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(FtpDeleter::new(Arc::new(self.clone()))), )) } async fn list(&self, path: &str, _: OpList) -> Result<(RpList, Self::Lister)> { let mut ftp_stream = self.ftp_connect(Operation::List).await?; let pathname = if path == "/" { None } else { Some(path) }; let files = ftp_stream.list(pathname).await.map_err(parse_error)?; Ok(( RpList::default(), FtpLister::new(if path == "/" { "" } else { path }, files), )) } } impl FtpBackend { pub async fn ftp_connect(&self, _: Operation) -> Result> { let pool = self .pool .get_or_try_init(|| async { bb8::Pool::builder() .max_size(64) .build(Manager { endpoint: self.endpoint.to_string(), root: self.root.to_string(), user: self.user.to_string(), password: self.password.to_string(), enable_secure: self.enable_secure, }) .await }) .await .map_err(parse_error)?; pool.get_owned().await.map_err(|err| match err { RunError::User(err) => parse_error(err), RunError::TimedOut => { Error::new(ErrorKind::Unexpected, "connection request: timeout").set_temporary() } }) } pub async fn ftp_stat(&self, path: &str) -> Result { let mut ftp_stream = self.ftp_connect(Operation::Stat).await?; let (parent, basename) = (get_parent(path), get_basename(path)); let pathname = if parent == "/" { None } else { Some(parent) }; let resp = ftp_stream.list(pathname).await.map_err(parse_error)?; // Get stat of file. let mut files = resp .into_iter() .filter_map(|file| File::from_str(file.as_str()).ok()) .filter(|f| f.name() == basename.trim_end_matches('/')) .collect::>(); if files.is_empty() { Err(Error::new( ErrorKind::NotFound, "file is not found during list", )) } else { Ok(files.remove(0)) } } } #[cfg(test)] mod build_test { use super::FtpBuilder; use crate::*; #[test] fn test_build() { // ftps scheme, should suffix with default port 21 let b = FtpBuilder::default() .endpoint("ftps://ftp_server.local") .build(); assert!(b.is_ok()); // ftp scheme let b = FtpBuilder::default() .endpoint("ftp://ftp_server.local:1234") .build(); assert!(b.is_ok()); // no scheme let b = FtpBuilder::default() .endpoint("ftp_server.local:8765") .build(); assert!(b.is_ok()); // invalid scheme let b = FtpBuilder::default() .endpoint("invalidscheme://ftp_server.local:8765") .build(); assert!(b.is_err()); let e = b.unwrap_err(); assert_eq!(e.kind(), ErrorKind::ConfigInvalid); } } opendal-0.52.0/src/services/ftp/config.rs000064400000000000000000000030411046102023000163460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Ftp services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct FtpConfig { /// endpoint of this backend pub endpoint: Option, /// root of this backend pub root: Option, /// user of this backend pub user: Option, /// password of this backend pub password: Option, } impl Debug for FtpConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("FtpConfig") .field("endpoint", &self.endpoint) .field("root", &self.root) .finish_non_exhaustive() } } opendal-0.52.0/src/services/ftp/delete.rs000064400000000000000000000034121046102023000163450ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::backend::FtpBackend; use super::err::parse_error; use crate::raw::*; use crate::*; use std::sync::Arc; use suppaftp::types::Response; use suppaftp::FtpError; use suppaftp::Status; pub struct FtpDeleter { core: Arc, } impl FtpDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for FtpDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let mut ftp_stream = self.core.ftp_connect(Operation::Delete).await?; let result = if path.ends_with('/') { ftp_stream.rmdir(&path).await } else { ftp_stream.rm(&path).await }; match result { Err(FtpError::UnexpectedResponse(Response { status: Status::FileUnavailable, .. })) | Ok(_) => (), Err(e) => { return Err(parse_error(e)); } } Ok(()) } } opendal-0.52.0/src/services/ftp/docs.md000064400000000000000000000013241046102023000160070ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [x] list - [ ] ~~presign~~ - [ ] blocking ## Configuration - `endpoint`: Set the endpoint for connection - `root`: Set the work directory for backend - `user`: Set the login user - `password`: Set the login password You can refer to [`FtpBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Ftp; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Ftp::default() .endpoint("127.0.0.1"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/ftp/err.rs000064400000000000000000000032631046102023000156770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use suppaftp::FtpError; use suppaftp::Status; use crate::Error; use crate::ErrorKind; pub(super) fn parse_error(err: FtpError) -> Error { let (kind, retryable) = match err { // Allow retry for error // // `{ status: NotAvailable, body: "421 There are too many connections from your internet address." }` FtpError::UnexpectedResponse(ref resp) if resp.status == Status::NotAvailable => { (ErrorKind::Unexpected, true) } FtpError::UnexpectedResponse(ref resp) if resp.status == Status::FileUnavailable => { (ErrorKind::NotFound, false) } // Allow retry bad response. FtpError::BadResponse => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let mut err = Error::new(kind, "ftp error").set_source(err); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/ftp/lister.rs000064400000000000000000000042771046102023000164170ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::str; use std::str::FromStr; use std::vec::IntoIter; use suppaftp::list::File; use crate::raw::*; use crate::*; pub struct FtpLister { path: String, file_iter: IntoIter, } impl FtpLister { pub fn new(path: &str, files: Vec) -> Self { Self { path: path.to_string(), file_iter: files.into_iter(), } } } impl oio::List for FtpLister { async fn next(&mut self) -> Result> { let de = match self.file_iter.next() { Some(file_str) => File::from_str(file_str.as_str()).map_err(|e| { Error::new(ErrorKind::Unexpected, "parse file from response").set_source(e) })?, None => return Ok(None), }; let path = self.path.to_string() + de.name(); let mut meta = if de.is_file() { Metadata::new(EntryMode::FILE) } else if de.is_directory() { Metadata::new(EntryMode::DIR) } else { Metadata::new(EntryMode::Unknown) }; meta.set_content_length(de.size() as u64); meta.set_last_modified(de.modified().into()); let entry = if de.is_file() { oio::Entry::new(&path, meta) } else if de.is_directory() { oio::Entry::new(&format!("{}/", &path), meta) } else { oio::Entry::new(&path, meta) }; Ok(Some(entry)) } } opendal-0.52.0/src/services/ftp/mod.rs000064400000000000000000000022341046102023000156630ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-ftp")] mod delete; #[cfg(feature = "services-ftp")] mod err; #[cfg(feature = "services-ftp")] mod lister; #[cfg(feature = "services-ftp")] mod reader; #[cfg(feature = "services-ftp")] mod writer; #[cfg(feature = "services-ftp")] mod backend; #[cfg(feature = "services-ftp")] pub use backend::FtpBuilder as Ftp; mod config; pub use config::FtpConfig; opendal-0.52.0/src/services/ftp/reader.rs000064400000000000000000000046311046102023000163510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bb8::PooledConnection; use bytes::BytesMut; use futures::AsyncRead; use futures::AsyncReadExt; use super::backend::Manager; use super::err::parse_error; use crate::raw::*; use crate::*; pub struct FtpReader { /// Keep the connection alive while data stream is alive. _ftp_stream: PooledConnection<'static, Manager>, data_stream: Box, chunk: usize, buf: BytesMut, } /// # Safety /// /// We only have `&mut self` for FtpReader. unsafe impl Sync for FtpReader {} impl FtpReader { pub async fn new( mut ftp_stream: PooledConnection<'static, Manager>, path: String, args: OpRead, ) -> Result { let (offset, size) = ( args.range().offset(), args.range().size().unwrap_or(u64::MAX), ); if offset != 0 { ftp_stream .resume_transfer(offset as usize) .await .map_err(parse_error)?; } let ds = ftp_stream .retr_as_stream(path) .await .map_err(parse_error)? .take(size as _); Ok(Self { _ftp_stream: ftp_stream, data_stream: Box::new(ds), chunk: 1024 * 1024, buf: BytesMut::new(), }) } } impl oio::Read for FtpReader { async fn read(&mut self) -> Result { self.buf.resize(self.chunk, 0); let n = self .data_stream .read(&mut self.buf) .await .map_err(new_std_io_error)?; Ok(Buffer::from(self.buf.split_to(n).freeze())) } } opendal-0.52.0/src/services/ftp/writer.rs000064400000000000000000000067321046102023000164270ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bb8::PooledConnection; use bytes::Buf; use futures::AsyncWrite; use futures::AsyncWriteExt; use super::backend::Manager; use super::err::parse_error; use crate::raw::*; use crate::*; pub struct FtpWriter { target_path: String, tmp_path: Option, ftp_stream: PooledConnection<'static, Manager>, data_stream: Option>, } /// # Safety /// /// We only have `&mut self` for FtpWrite. unsafe impl Sync for FtpWriter {} /// # TODO /// /// Writer is not implemented correctly. /// /// After we can use data stream, we should return it directly. impl FtpWriter { pub fn new( ftp_stream: PooledConnection<'static, Manager>, target_path: String, tmp_path: Option, ) -> Self { FtpWriter { target_path, tmp_path, ftp_stream, data_stream: None, } } } impl oio::Write for FtpWriter { async fn write(&mut self, mut bs: Buffer) -> Result<()> { let path = if let Some(tmp_path) = &self.tmp_path { tmp_path } else { &self.target_path }; if self.data_stream.is_none() { self.data_stream = Some(Box::new( self.ftp_stream .append_with_stream(path) .await .map_err(parse_error)?, )); } while bs.has_remaining() { let n = self .data_stream .as_mut() .unwrap() .write(bs.chunk()) .await .map_err(|err| { Error::new(ErrorKind::Unexpected, "copy from ftp stream").set_source(err) })?; bs.advance(n); } Ok(()) } async fn close(&mut self) -> Result { let data_stream = self.data_stream.take(); if let Some(mut data_stream) = data_stream { data_stream.flush().await.map_err(|err| { Error::new(ErrorKind::Unexpected, "flush data stream failed").set_source(err) })?; self.ftp_stream .finalize_put_stream(data_stream) .await .map_err(parse_error)?; if let Some(tmp_path) = &self.tmp_path { self.ftp_stream .rename(tmp_path, &self.target_path) .await .map_err(parse_error)?; } } Ok(Metadata::default()) } async fn abort(&mut self) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "FtpWriter doesn't support abort", )) } } opendal-0.52.0/src/services/gcs/backend.rs000064400000000000000000000472351046102023000164700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::Response; use http::StatusCode; use log::debug; use reqsign::GoogleCredentialLoader; use reqsign::GoogleSigner; use reqsign::GoogleTokenLoad; use reqsign::GoogleTokenLoader; use serde::Deserialize; use serde_json; use super::core::*; use super::delete::GcsDeleter; use super::error::parse_error; use super::lister::GcsLister; use super::writer::GcsWriter; use super::writer::GcsWriters; use crate::raw::oio::BatchDeleter; use crate::raw::*; use crate::services::GcsConfig; use crate::*; const DEFAULT_GCS_ENDPOINT: &str = "https://storage.googleapis.com"; const DEFAULT_GCS_SCOPE: &str = "https://www.googleapis.com/auth/devstorage.read_write"; impl Configurator for GcsConfig { type Builder = GcsBuilder; fn into_builder(self) -> Self::Builder { GcsBuilder { config: self, http_client: None, customized_token_loader: None, } } } /// [Google Cloud Storage](https://cloud.google.com/storage) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct GcsBuilder { config: GcsConfig, http_client: Option, customized_token_loader: Option>, } impl Debug for GcsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("GcsBuilder"); ds.field("config", &self.config); ds.finish_non_exhaustive() } } impl GcsBuilder { /// set the working directory root of backend pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// set the container's name pub fn bucket(mut self, bucket: &str) -> Self { self.config.bucket = bucket.to_string(); self } /// set the GCS service scope /// /// If not set, we will use `https://www.googleapis.com/auth/devstorage.read_write`. /// /// # Valid scope examples /// /// - read-only: `https://www.googleapis.com/auth/devstorage.read_only` /// - read-write: `https://www.googleapis.com/auth/devstorage.read_write` /// - full-control: `https://www.googleapis.com/auth/devstorage.full_control` /// /// Reference: [Cloud Storage authentication](https://cloud.google.com/storage/docs/authentication) pub fn scope(mut self, scope: &str) -> Self { if !scope.is_empty() { self.config.scope = Some(scope.to_string()) }; self } /// Set the GCS service account. /// /// service account will be used for fetch token from vm metadata. /// If not set, we will try to fetch with `default` service account. pub fn service_account(mut self, service_account: &str) -> Self { if !service_account.is_empty() { self.config.service_account = Some(service_account.to_string()) }; self } /// set the endpoint GCS service uses pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { self.config.endpoint = Some(endpoint.to_string()) }; self } /// set the base64 hashed credentials string used for OAuth2 authentication. /// /// this method allows to specify the credentials directly as a base64 hashed string. /// alternatively, you can use `credential_path()` to provide the local path to a credentials file. /// we will use one of `credential` and `credential_path` to complete the OAuth2 authentication. /// /// Reference: [Google Cloud Storage Authentication](https://cloud.google.com/docs/authentication). pub fn credential(mut self, credential: &str) -> Self { if !credential.is_empty() { self.config.credential = Some(credential.to_string()) }; self } /// set the local path to credentials file which is used for OAuth2 authentication. /// /// credentials file contains the original credentials that have not been base64 hashed. /// we will use one of `credential` and `credential_path` to complete the OAuth2 authentication. /// /// Reference: [Google Cloud Storage Authentication](https://cloud.google.com/docs/authentication). pub fn credential_path(mut self, path: &str) -> Self { if !path.is_empty() { self.config.credential_path = Some(path.to_string()) }; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } /// Specify the customized token loader used by this service. pub fn customized_token_loader(mut self, token_load: Box) -> Self { self.customized_token_loader = Some(token_load); self } /// Provide the OAuth2 token to use. pub fn token(mut self, token: String) -> Self { self.config.token = Some(token); self } /// Disable attempting to load credentials from the GCE metadata server. pub fn disable_vm_metadata(mut self) -> Self { self.config.disable_vm_metadata = true; self } /// Disable loading configuration from the environment. pub fn disable_config_load(mut self) -> Self { self.config.disable_config_load = true; self } /// Set the predefined acl for GCS. /// /// Available values are: /// - `authenticatedRead` /// - `bucketOwnerFullControl` /// - `bucketOwnerRead` /// - `private` /// - `projectPrivate` /// - `publicRead` pub fn predefined_acl(mut self, acl: &str) -> Self { if !acl.is_empty() { self.config.predefined_acl = Some(acl.to_string()) }; self } /// Set the default storage class for GCS. /// /// Available values are: /// - `STANDARD` /// - `NEARLINE` /// - `COLDLINE` /// - `ARCHIVE` pub fn default_storage_class(mut self, class: &str) -> Self { if !class.is_empty() { self.config.default_storage_class = Some(class.to_string()) }; self } /// Allow anonymous requests. /// /// This is typically used for buckets which are open to the public or GCS /// storage emulators. pub fn allow_anonymous(mut self) -> Self { self.config.allow_anonymous = true; self } } impl Builder for GcsBuilder { const SCHEME: Scheme = Scheme::Gcs; type Config = GcsConfig; fn build(self) -> Result { debug!("backend build started: {:?}", self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); // Handle endpoint and bucket name let bucket = match self.config.bucket.is_empty() { false => Ok(&self.config.bucket), true => Err( Error::new(ErrorKind::ConfigInvalid, "The bucket is misconfigured") .with_operation("Builder::build") .with_context("service", Scheme::Gcs), ), }?; // TODO: server side encryption let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Gcs) })? }; let endpoint = self .config .endpoint .clone() .unwrap_or_else(|| DEFAULT_GCS_ENDPOINT.to_string()); debug!("backend use endpoint: {endpoint}"); let mut cred_loader = GoogleCredentialLoader::default(); if let Some(cred) = &self.config.credential { cred_loader = cred_loader.with_content(cred); } if let Some(cred) = &self.config.credential_path { cred_loader = cred_loader.with_path(cred); } #[cfg(target_arch = "wasm32")] { cred_loader = cred_loader.with_disable_env(); cred_loader = cred_loader.with_disable_well_known_location(); } if self.config.disable_config_load { cred_loader = cred_loader .with_disable_env() .with_disable_well_known_location(); } let scope = if let Some(scope) = &self.config.scope { scope } else { DEFAULT_GCS_SCOPE }; let mut token_loader = GoogleTokenLoader::new(scope, GLOBAL_REQWEST_CLIENT.clone()); if let Some(account) = &self.config.service_account { token_loader = token_loader.with_service_account(account); } if let Ok(Some(cred)) = cred_loader.load() { token_loader = token_loader.with_credentials(cred) } if let Some(loader) = self.customized_token_loader { token_loader = token_loader.with_customized_token_loader(loader) } if self.config.disable_vm_metadata { token_loader = token_loader.with_disable_vm_metadata(true); } let signer = GoogleSigner::new("storage"); let backend = GcsBackend { core: Arc::new(GcsCore { endpoint, bucket: bucket.to_string(), root, client, signer, token_loader, token: self.config.token, scope: scope.to_string(), credential_loader: cred_loader, predefined_acl: self.config.predefined_acl.clone(), default_storage_class: self.config.default_storage_class.clone(), allow_anonymous: self.config.allow_anonymous, }), }; Ok(backend) } } /// GCS storage backend #[derive(Clone, Debug)] pub struct GcsBackend { core: Arc, } impl Access for GcsBackend { type Reader = HttpBody; type Writer = GcsWriters; type Lister = oio::PageLister; type Deleter = oio::BatchDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Gcs) .set_root(&self.core.root) .set_name(&self.core.bucket) .set_native_capability(Capability { stat: true, stat_with_if_match: true, stat_with_if_none_match: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_last_modified: true, stat_has_user_metadata: true, read: true, read_with_if_match: true, read_with_if_none_match: true, write: true, write_can_empty: true, write_can_multi: true, write_with_content_type: true, write_with_content_encoding: true, write_with_user_metadata: true, write_with_if_not_exists: true, // The min multipart size of Gcs is 5 MiB. // // ref: write_multi_min_size: Some(5 * 1024 * 1024), // The max multipart size of Gcs is 5 GiB. // // ref: write_multi_max_size: if cfg!(target_pointer_width = "64") { Some(5 * 1024 * 1024 * 1024) } else { Some(usize::MAX) }, delete: true, delete_max_size: Some(100), copy: true, list: true, list_with_limit: true, list_with_start_after: true, list_with_recursive: true, list_has_etag: true, list_has_content_md5: true, list_has_content_length: true, list_has_content_type: true, list_has_last_modified: true, presign: true, presign_stat: true, presign_read: true, presign_write: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, args: OpStat) -> Result { let resp = self.core.gcs_get_object_metadata(path, &args).await?; if !resp.status().is_success() { return Err(parse_error(resp)); } let slc = resp.into_body(); let meta: GetObjectJsonResponse = serde_json::from_reader(slc.reader()).map_err(new_json_deserialize_error)?; let mut m = Metadata::new(EntryMode::FILE); m.set_etag(&meta.etag); m.set_content_md5(&meta.md5_hash); let size = meta .size .parse::() .map_err(|e| Error::new(ErrorKind::Unexpected, "parse u64").set_source(e))?; m.set_content_length(size); if !meta.content_type.is_empty() { m.set_content_type(&meta.content_type); } if !meta.content_encoding.is_empty() { m.set_content_encoding(&meta.content_encoding); } m.set_last_modified(parse_datetime_from_rfc3339(&meta.updated)?); if !meta.metadata.is_empty() { m.with_user_metadata(meta.metadata); } Ok(RpStat::new(m)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.gcs_get_object(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let concurrent = args.concurrent(); let executor = args.executor().cloned(); let w = GcsWriter::new(self.core.clone(), path, args); let w = oio::MultipartWriter::new(w, executor, concurrent); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), BatchDeleter::new(GcsDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = GcsLister::new( self.core.clone(), path, args.recursive(), args.limit(), args.start_after(), ); Ok((RpList::default(), oio::PageLister::new(l))) } async fn copy(&self, from: &str, to: &str, _: OpCopy) -> Result { let resp = self.core.gcs_copy_object(from, to).await?; if resp.status().is_success() { Ok(RpCopy::default()) } else { Err(parse_error(resp)) } } async fn presign(&self, path: &str, args: OpPresign) -> Result { // We will not send this request out, just for signing. let mut req = match args.operation() { PresignOperation::Stat(v) => self.core.gcs_head_object_xml_request(path, v)?, PresignOperation::Read(v) => self.core.gcs_get_object_xml_request(path, v)?, PresignOperation::Write(v) => { self.core .gcs_insert_object_xml_request(path, v, Buffer::new())? } }; self.core.sign_query(&mut req, args.expire())?; // We don't need this request anymore, consume it directly. let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } } /// The raw json response returned by [`get`](https://cloud.google.com/storage/docs/json_api/v1/objects/get) #[derive(Debug, Default, Deserialize)] #[serde(default, rename_all = "camelCase")] struct GetObjectJsonResponse { /// GCS will return size in string. /// /// For example: `"size": "56535"` size: String, /// etag is not quoted. /// /// For example: `"etag": "CKWasoTgyPkCEAE="` etag: String, /// RFC3339 styled datetime string. /// /// For example: `"updated": "2022-08-15T11:33:34.866Z"` updated: String, /// Content md5 hash /// /// For example: `"md5Hash": "fHcEH1vPwA6eTPqxuasXcg=="` md5_hash: String, /// Content type of this object. /// /// For example: `"contentType": "image/png",` content_type: String, /// Content encoding of this object /// /// For example: "contentEncoding": "br" content_encoding: String, /// Custom metadata of this object. /// /// For example: `"metadata" : { "my-key": "my-value" }` metadata: HashMap, } #[cfg(test)] mod tests { use super::*; #[test] fn test_deserialize_get_object_json_response() { let content = r#"{ "kind": "storage#object", "id": "example/1.png/1660563214863653", "selfLink": "https://www.googleapis.com/storage/v1/b/example/o/1.png", "mediaLink": "https://content-storage.googleapis.com/download/storage/v1/b/example/o/1.png?generation=1660563214863653&alt=media", "name": "1.png", "bucket": "example", "generation": "1660563214863653", "metageneration": "1", "contentType": "image/png", "contentEncoding": "br", "storageClass": "STANDARD", "size": "56535", "md5Hash": "fHcEH1vPwA6eTPqxuasXcg==", "crc32c": "j/un9g==", "etag": "CKWasoTgyPkCEAE=", "timeCreated": "2022-08-15T11:33:34.866Z", "updated": "2022-08-15T11:33:34.866Z", "timeStorageClassUpdated": "2022-08-15T11:33:34.866Z", "metadata" : { "location" : "everywhere" } }"#; let meta: GetObjectJsonResponse = serde_json::from_str(content).expect("json Deserialize must succeed"); assert_eq!(meta.size, "56535"); assert_eq!(meta.updated, "2022-08-15T11:33:34.866Z"); assert_eq!(meta.md5_hash, "fHcEH1vPwA6eTPqxuasXcg=="); assert_eq!(meta.etag, "CKWasoTgyPkCEAE="); assert_eq!(meta.content_type, "image/png"); assert_eq!(meta.content_encoding, "br".to_string()); assert_eq!( meta.metadata, HashMap::from_iter([("location".to_string(), "everywhere".to_string())]) ); } } opendal-0.52.0/src/services/gcs/config.rs000064400000000000000000000052521046102023000163370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// [Google Cloud Storage](https://cloud.google.com/storage) services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct GcsConfig { /// root URI, all operations happens under `root` pub root: Option, /// bucket name pub bucket: String, /// endpoint URI of GCS service, /// default is `https://storage.googleapis.com` pub endpoint: Option, /// Scope for gcs. pub scope: Option, /// Service Account for gcs. pub service_account: Option, /// Credentials string for GCS service OAuth2 authentication. pub credential: Option, /// Local path to credentials file for GCS service OAuth2 authentication. pub credential_path: Option, /// The predefined acl for GCS. pub predefined_acl: Option, /// The default storage class used by gcs. pub default_storage_class: Option, /// Allow opendal to send requests without signing when credentials are not /// loaded. pub allow_anonymous: bool, /// Disable attempting to load credentials from the GCE metadata server when /// running within Google Cloud. pub disable_vm_metadata: bool, /// Disable loading configuration from the environment. pub disable_config_load: bool, /// A Google Cloud OAuth2 token. /// /// Takes precedence over `credential` and `credential_path`. pub token: Option, } impl Debug for GcsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("GcsConfig") .field("root", &self.root) .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .field("scope", &self.scope) .finish_non_exhaustive() } } opendal-0.52.0/src/services/gcs/core.rs000064400000000000000000000657671046102023000160430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::Write; use std::time::Duration; use backon::ExponentialBuilder; use backon::Retryable; use bytes::Bytes; use http::header::CONTENT_ENCODING; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::header::HOST; use http::header::IF_MATCH; use http::header::IF_MODIFIED_SINCE; use http::header::IF_NONE_MATCH; use http::header::IF_UNMODIFIED_SINCE; use http::Request; use http::Response; use once_cell::sync::Lazy; use reqsign::GoogleCredential; use reqsign::GoogleCredentialLoader; use reqsign::GoogleSigner; use reqsign::GoogleToken; use reqsign::GoogleTokenLoader; use serde::Deserialize; use serde::Serialize; use super::uri::percent_encode_path; use crate::raw::*; use crate::*; use constants::*; pub mod constants { pub const X_GOOG_ACL: &str = "x-goog-acl"; pub const X_GOOG_STORAGE_CLASS: &str = "x-goog-storage-class"; pub const X_GOOG_META_PREFIX: &str = "x-goog-meta-"; } pub struct GcsCore { pub endpoint: String, pub bucket: String, pub root: String, pub client: HttpClient, pub signer: GoogleSigner, pub token_loader: GoogleTokenLoader, pub token: Option, pub scope: String, pub credential_loader: GoogleCredentialLoader, pub predefined_acl: Option, pub default_storage_class: Option, pub allow_anonymous: bool, } impl Debug for GcsCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("Backend"); de.field("endpoint", &self.endpoint) .field("bucket", &self.bucket) .field("root", &self.root) .finish_non_exhaustive() } } static BACKOFF: Lazy = Lazy::new(|| ExponentialBuilder::default().with_jitter()); impl GcsCore { async fn load_token(&self) -> Result> { if let Some(token) = &self.token { return Ok(Some(GoogleToken::new(token, usize::MAX, &self.scope))); } let cred = { || self.token_loader.load() } .retry(*BACKOFF) .await .map_err(new_request_credential_error)?; if let Some(cred) = cred { return Ok(Some(cred)); } if self.allow_anonymous { return Ok(None); } Err(Error::new( ErrorKind::ConfigInvalid, "no valid credential found", )) } fn load_credential(&self) -> Result> { let cred = self .credential_loader .load() .map_err(new_request_credential_error)?; if let Some(cred) = cred { return Ok(Some(cred)); } if self.allow_anonymous { return Ok(None); } Err(Error::new( ErrorKind::ConfigInvalid, "no valid credential found", )) } pub async fn sign(&self, req: &mut Request) -> Result<()> { if let Some(cred) = self.load_token().await? { self.signer .sign(req, &cred) .map_err(new_request_sign_error)?; } else { return Ok(()); } // Always remove host header, let users' client to set it based on HTTP // version. // // As discussed in , // google server could send RST_STREAM of PROTOCOL_ERROR if our request // contains host header. req.headers_mut().remove(HOST); Ok(()) } pub fn sign_query(&self, req: &mut Request, duration: Duration) -> Result<()> { if let Some(cred) = self.load_credential()? { self.signer .sign_query(req, duration, &cred) .map_err(new_request_sign_error)?; } else { return Ok(()); } // Always remove host header, let users' client to set it based on HTTP // version. // // As discussed in , // google server could send RST_STREAM of PROTOCOL_ERROR if our request // contains host header. req.headers_mut().remove(HOST); Ok(()) } #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } } impl GcsCore { pub fn gcs_get_object_request( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/b/{}/o/{}?alt=media", self.endpoint, self.bucket, percent_encode_path(&p) ); let mut req = Request::get(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if !range.is_full() { req = req.header(http::header::RANGE, range.to_header()); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } // It's for presign operation. Gcs only supports query sign over XML API. pub fn gcs_get_object_xml_request(&self, path: &str, args: &OpRead) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}/{}", self.endpoint, self.bucket, p); let mut req = Request::get(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(if_modified_since) = args.if_modified_since() { req = req.header( IF_MODIFIED_SINCE, format_datetime_into_http_date(if_modified_since), ); } if let Some(if_unmodified_since) = args.if_unmodified_since() { req = req.header( IF_UNMODIFIED_SINCE, format_datetime_into_http_date(if_unmodified_since), ); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub async fn gcs_get_object( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let mut req = self.gcs_get_object_request(path, range, args)?; self.sign(&mut req).await?; self.client.fetch(req).await } pub fn gcs_insert_object_request( &self, path: &str, size: Option, op: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let request_metadata = InsertRequestMetadata { storage_class: self.default_storage_class.as_deref(), cache_control: op.cache_control(), content_type: op.content_type(), content_encoding: op.content_encoding(), metadata: op.user_metadata(), }; let mut url = format!( "{}/upload/storage/v1/b/{}/o?uploadType={}&name={}", self.endpoint, self.bucket, if request_metadata.is_empty() { "media" } else { "multipart" }, percent_encode_path(&p) ); if let Some(acl) = &self.predefined_acl { write!(&mut url, "&predefinedAcl={}", acl).unwrap(); } // Makes the operation conditional on whether the object's current generation // matches the given value. Setting to 0 makes the operation succeed only if // there are no live versions of the object. if op.if_not_exists() { write!(&mut url, "&ifGenerationMatch=0").unwrap(); } let mut req = Request::post(&url); req = req.header(CONTENT_LENGTH, size.unwrap_or_default()); if request_metadata.is_empty() { // If the metadata is empty, we do not set any `Content-Type` header, // since if we had it in the `op.content_type()`, it would be already set in the // `multipart` metadata body and this branch won't be executed. let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } else { let mut multipart = Multipart::new(); let metadata_part = FormDataPart::new("metadata") .header( CONTENT_TYPE, "application/json; charset=UTF-8".parse().unwrap(), ) .content( serde_json::to_vec(&request_metadata) .expect("metadata serialization should success"), ); multipart = multipart.part(metadata_part); let media_part = FormDataPart::new("media").content(body); multipart = multipart.part(media_part); let req = multipart.apply(Request::post(url))?; Ok(req) } } // It's for presign operation. Gcs only supports query sign over XML API. pub fn gcs_insert_object_xml_request( &self, path: &str, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}/{}", self.endpoint, self.bucket, p); let mut req = Request::put(&url); if let Some(user_metadata) = args.user_metadata() { for (key, value) in user_metadata { req = req.header(format!("{X_GOOG_META_PREFIX}{key}"), value) } } if let Some(content_type) = args.content_type() { req = req.header(CONTENT_TYPE, content_type); } if let Some(content_encoding) = args.content_encoding() { req = req.header(CONTENT_ENCODING, content_encoding); } if let Some(acl) = &self.predefined_acl { req = req.header(X_GOOG_ACL, acl); } if let Some(storage_class) = &self.default_storage_class { req = req.header(X_GOOG_STORAGE_CLASS, storage_class); } let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub fn gcs_head_object_request(&self, path: &str, args: &OpStat) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/b/{}/o/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); let mut req = Request::get(&url); if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } // It's for presign operation. Gcs only supports query sign over XML API. pub fn gcs_head_object_xml_request( &self, path: &str, args: &OpStat, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}/{}", self.endpoint, self.bucket, p); let mut req = Request::head(&url); if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub async fn gcs_get_object_metadata( &self, path: &str, args: &OpStat, ) -> Result> { let mut req = self.gcs_head_object_request(path, args)?; self.sign(&mut req).await?; self.send(req).await } pub async fn gcs_delete_object(&self, path: &str) -> Result> { let mut req = self.gcs_delete_object_request(path)?; self.sign(&mut req).await?; self.send(req).await } pub fn gcs_delete_object_request(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/b/{}/o/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); Request::delete(&url) .body(Buffer::new()) .map_err(new_request_build_error) } pub async fn gcs_delete_objects(&self, paths: Vec) -> Result> { let uri = format!("{}/batch/storage/v1", self.endpoint); let mut multipart = Multipart::new(); for (idx, path) in paths.iter().enumerate() { let req = self.gcs_delete_object_request(path)?; multipart = multipart.part( MixedPart::from_request(req).part_header("content-id".parse().unwrap(), idx.into()), ); } let req = Request::post(uri); let mut req = multipart.apply(req)?; self.sign(&mut req).await?; self.send(req).await } pub async fn gcs_copy_object(&self, from: &str, to: &str) -> Result> { let source = build_abs_path(&self.root, from); let dest = build_abs_path(&self.root, to); let req_uri = format!( "{}/storage/v1/b/{}/o/{}/copyTo/b/{}/o/{}", self.endpoint, self.bucket, percent_encode_path(&source), self.bucket, percent_encode_path(&dest) ); let mut req = Request::post(req_uri) .header(CONTENT_LENGTH, 0) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn gcs_list_objects( &self, path: &str, page_token: &str, delimiter: &str, limit: Option, start_after: Option, ) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/storage/v1/b/{}/o?prefix={}", self.endpoint, self.bucket, percent_encode_path(&p) ); if !delimiter.is_empty() { write!(url, "&delimiter={delimiter}").expect("write into string must succeed"); } if let Some(limit) = limit { write!(url, "&maxResults={limit}").expect("write into string must succeed"); } if let Some(start_after) = start_after { let start_after = build_abs_path(&self.root, &start_after); write!(url, "&startOffset={}", percent_encode_path(&start_after)) .expect("write into string must succeed"); } if !page_token.is_empty() { // NOTE: // // GCS uses pageToken in request and nextPageToken in response // // Don't know how will those tokens be like so this part are copied // directly from AWS S3 service. write!(url, "&pageToken={}", percent_encode_path(page_token)) .expect("write into string must succeed"); } let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn gcs_initiate_multipart_upload(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}/{}?uploads", self.endpoint, self.bucket, p); let mut req = Request::post(&url) .header(CONTENT_LENGTH, 0) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn gcs_upload_part( &self, path: &str, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}?partNumber={}&uploadId={}", self.endpoint, self.bucket, percent_encode_path(&p), part_number, percent_encode_path(upload_id) ); let mut req = Request::put(&url); req = req.header(CONTENT_LENGTH, size); let mut req = req.body(body).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn gcs_complete_multipart_upload( &self, path: &str, upload_id: &str, parts: Vec, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}?uploadId={}", self.endpoint, self.bucket, percent_encode_path(&p), percent_encode_path(upload_id) ); let req = Request::post(&url); let content = quick_xml::se::to_string(&CompleteMultipartUploadRequest { part: parts }) .map_err(new_xml_deserialize_error)?; // Make sure content length has been set to avoid post with chunked encoding. let req = req.header(CONTENT_LENGTH, content.len()); // Set content-type to `application/xml` to avoid mixed with form post. let req = req.header(CONTENT_TYPE, "application/xml"); let mut req = req .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn gcs_abort_multipart_upload( &self, path: &str, upload_id: &str, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}?uploadId={}", self.endpoint, self.bucket, percent_encode_path(&p), percent_encode_path(upload_id) ); let mut req = Request::delete(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } } #[derive(Debug, Serialize)] #[serde(default, rename_all = "camelCase")] pub struct InsertRequestMetadata<'a> { #[serde(skip_serializing_if = "Option::is_none")] content_type: Option<&'a str>, #[serde(skip_serializing_if = "Option::is_none")] content_encoding: Option<&'a str>, #[serde(skip_serializing_if = "Option::is_none")] storage_class: Option<&'a str>, #[serde(skip_serializing_if = "Option::is_none")] cache_control: Option<&'a str>, #[serde(skip_serializing_if = "Option::is_none")] metadata: Option<&'a HashMap>, } impl InsertRequestMetadata<'_> { pub fn is_empty(&self) -> bool { self.content_type.is_none() && self.content_encoding.is_none() && self.storage_class.is_none() && self.cache_control.is_none() && self.metadata.is_none() } } /// Response JSON from GCS list objects API. /// /// refer to https://cloud.google.com/storage/docs/json_api/v1/objects/list for details #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "camelCase")] pub struct ListResponse { /// The continuation token. /// /// If this is the last page of results, then no continuation token is returned. pub next_page_token: Option, /// Object name prefixes for objects that matched the listing request /// but were excluded from [items] because of a delimiter. pub prefixes: Vec, /// The list of objects, ordered lexicographically by name. pub items: Vec, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(default, rename_all = "camelCase")] pub struct ListResponseItem { pub name: String, pub size: String, // metadata pub etag: String, pub md5_hash: String, pub updated: String, pub content_type: String, } /// Result of CreateMultipartUpload #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct InitiateMultipartUploadResult { pub upload_id: String, } /// Request of CompleteMultipartUploadRequest #[derive(Default, Debug, Serialize)] #[serde(default, rename = "CompleteMultipartUpload", rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequest { pub part: Vec, } #[derive(Clone, Default, Debug, Serialize)] #[serde(default, rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequestPart { #[serde(rename = "PartNumber")] pub part_number: usize, #[serde(rename = "ETag")] pub etag: String, } #[cfg(test)] mod tests { use super::*; #[test] fn test_deserialize_list_response() { let content = r#" { "kind": "storage#objects", "prefixes": [ "dir/", "test/" ], "items": [ { "kind": "storage#object", "id": "example/1.png/1660563214863653", "selfLink": "https://www.googleapis.com/storage/v1/b/example/o/1.png", "mediaLink": "https://content-storage.googleapis.com/download/storage/v1/b/example/o/1.png?generation=1660563214863653&alt=media", "name": "1.png", "bucket": "example", "generation": "1660563214863653", "metageneration": "1", "contentType": "image/png", "storageClass": "STANDARD", "size": "56535", "md5Hash": "fHcEH1vPwA6eTPqxuasXcg==", "crc32c": "j/un9g==", "etag": "CKWasoTgyPkCEAE=", "timeCreated": "2022-08-15T11:33:34.866Z", "updated": "2022-08-15T11:33:34.866Z", "timeStorageClassUpdated": "2022-08-15T11:33:34.866Z" }, { "kind": "storage#object", "id": "example/2.png/1660563214883337", "selfLink": "https://www.googleapis.com/storage/v1/b/example/o/2.png", "mediaLink": "https://content-storage.googleapis.com/download/storage/v1/b/example/o/2.png?generation=1660563214883337&alt=media", "name": "2.png", "bucket": "example", "generation": "1660563214883337", "metageneration": "1", "contentType": "image/png", "storageClass": "STANDARD", "size": "45506", "md5Hash": "e6LsGusU7pFJZk+114NV1g==", "crc32c": "L00QAg==", "etag": "CIm0s4TgyPkCEAE=", "timeCreated": "2022-08-15T11:33:34.886Z", "updated": "2022-08-15T11:33:34.886Z", "timeStorageClassUpdated": "2022-08-15T11:33:34.886Z" } ] } "#; let output: ListResponse = serde_json::from_str(content).expect("JSON deserialize must succeed"); assert!(output.next_page_token.is_none()); assert_eq!(output.items.len(), 2); assert_eq!(output.items[0].name, "1.png"); assert_eq!(output.items[0].size, "56535"); assert_eq!(output.items[0].md5_hash, "fHcEH1vPwA6eTPqxuasXcg=="); assert_eq!(output.items[0].etag, "CKWasoTgyPkCEAE="); assert_eq!(output.items[0].updated, "2022-08-15T11:33:34.866Z"); assert_eq!(output.items[1].name, "2.png"); assert_eq!(output.items[1].size, "45506"); assert_eq!(output.items[1].md5_hash, "e6LsGusU7pFJZk+114NV1g=="); assert_eq!(output.items[1].etag, "CIm0s4TgyPkCEAE="); assert_eq!(output.items[1].updated, "2022-08-15T11:33:34.886Z"); assert_eq!(output.items[1].content_type, "image/png"); assert_eq!(output.prefixes, vec!["dir/", "test/"]) } #[test] fn test_deserialize_list_response_with_next_page_token() { let content = r#" { "kind": "storage#objects", "prefixes": [ "dir/", "test/" ], "nextPageToken": "CgYxMC5wbmc=", "items": [ { "kind": "storage#object", "id": "example/1.png/1660563214863653", "selfLink": "https://www.googleapis.com/storage/v1/b/example/o/1.png", "mediaLink": "https://content-storage.googleapis.com/download/storage/v1/b/example/o/1.png?generation=1660563214863653&alt=media", "name": "1.png", "bucket": "example", "generation": "1660563214863653", "metageneration": "1", "contentType": "image/png", "storageClass": "STANDARD", "size": "56535", "md5Hash": "fHcEH1vPwA6eTPqxuasXcg==", "crc32c": "j/un9g==", "etag": "CKWasoTgyPkCEAE=", "timeCreated": "2022-08-15T11:33:34.866Z", "updated": "2022-08-15T11:33:34.866Z", "timeStorageClassUpdated": "2022-08-15T11:33:34.866Z" }, { "kind": "storage#object", "id": "example/2.png/1660563214883337", "selfLink": "https://www.googleapis.com/storage/v1/b/example/o/2.png", "mediaLink": "https://content-storage.googleapis.com/download/storage/v1/b/example/o/2.png?generation=1660563214883337&alt=media", "name": "2.png", "bucket": "example", "generation": "1660563214883337", "metageneration": "1", "contentType": "image/png", "storageClass": "STANDARD", "size": "45506", "md5Hash": "e6LsGusU7pFJZk+114NV1g==", "crc32c": "L00QAg==", "etag": "CIm0s4TgyPkCEAE=", "timeCreated": "2022-08-15T11:33:34.886Z", "updated": "2022-08-15T11:33:34.886Z", "timeStorageClassUpdated": "2022-08-15T11:33:34.886Z" } ] } "#; let output: ListResponse = serde_json::from_str(content).expect("JSON deserialize must succeed"); assert_eq!(output.next_page_token, Some("CgYxMC5wbmc=".to_string())); assert_eq!(output.items.len(), 2); assert_eq!(output.items[0].name, "1.png"); assert_eq!(output.items[0].size, "56535"); assert_eq!(output.items[0].md5_hash, "fHcEH1vPwA6eTPqxuasXcg=="); assert_eq!(output.items[0].etag, "CKWasoTgyPkCEAE="); assert_eq!(output.items[0].updated, "2022-08-15T11:33:34.866Z"); assert_eq!(output.items[1].name, "2.png"); assert_eq!(output.items[1].size, "45506"); assert_eq!(output.items[1].md5_hash, "e6LsGusU7pFJZk+114NV1g=="); assert_eq!(output.items[1].etag, "CIm0s4TgyPkCEAE="); assert_eq!(output.items[1].updated, "2022-08-15T11:33:34.886Z"); assert_eq!(output.prefixes, vec!["dir/", "test/"]) } } opendal-0.52.0/src/services/gcs/delete.rs000064400000000000000000000066001046102023000163320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::oio::BatchDeleteResult; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct GcsDeleter { core: Arc, } impl GcsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::BatchDelete for GcsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.gcs_delete_object(&path).await?; // deleting not existing objects is ok if resp.status().is_success() || resp.status() == StatusCode::NOT_FOUND { Ok(()) } else { Err(parse_error(resp)) } } async fn delete_batch(&self, batch: Vec<(String, OpDelete)>) -> Result { let paths: Vec = batch.into_iter().map(|(p, _)| p).collect(); let resp = self.core.gcs_delete_objects(paths.clone()).await?; let status = resp.status(); // If the overall request isn't formatted correctly and Cloud Storage is unable to parse it into sub-requests, you receive a 400 error. // Otherwise, Cloud Storage returns a 200 status code, even if some or all of the sub-requests fail. if status != StatusCode::OK { return Err(parse_error(resp)); } let boundary = parse_multipart_boundary(resp.headers())?.ok_or_else(|| { Error::new( ErrorKind::Unexpected, "gcs batch delete response content type is empty", ) })?; let multipart: Multipart = Multipart::new() .with_boundary(boundary) .parse(resp.into_body().to_bytes())?; let parts = multipart.into_parts(); let mut batched_result = BatchDeleteResult::default(); for (i, part) in parts.into_iter().enumerate() { let resp = part.into_response(); // TODO: maybe we can take it directly? let path = paths[i].clone(); // deleting not existing objects is ok if resp.status().is_success() || resp.status() == StatusCode::NOT_FOUND { batched_result.succeeded.push((path, OpDelete::default())); } else { batched_result .failed .push((path, OpDelete::default(), parse_error(resp))); } } // If no object is deleted, return directly. if batched_result.succeeded.is_empty() { let err = batched_result.failed.remove(0).2; return Err(err); } Ok(batched_result) } } opendal-0.52.0/src/services/gcs/docs.md000064400000000000000000000052211046102023000157720ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [x] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `bucket`: Set the container name for backend - `endpoint`: Customizable endpoint setting - `credential`: Service Account or External Account JSON, in base64 - `credential_path`: local path to Service Account or External Account JSON file - `service_account`: name of Service Account - `predefined_acl`: Predefined ACL for GCS - `default_storage_class`: Default storage class for GCS Refer to public API docs for more information. For authentication related options, read on. ## Options to authenticate to GCS OpenDAL supports the following authentication options: 1. Provide a base64-ed JSON key string with `credential` 2. Provide a JSON key file at explicit path with `credential_path` 3. Provide a JSON key file at implicit path - `GcsBackend` will attempt to load Service Account key from [ADC well-known places](https://cloud.google.com/docs/authentication/application-default-credentials). 4. Fetch access token from [VM metadata](https://cloud.google.com/docs/authentication/rest#metadata-server) - Only works when running inside Google Cloud. - If a non-default Service Account name is required, set with `service_account`. Otherwise, nothing need to be set. 5. A custom `TokenLoader` via `GcsBuilder.customized_token_loader()` Notes: - When a Service Account key is provided, it will be used to create access tokens (VM metadata will not be used). - Explicit Service Account key, in json or path, always take precedence over ADC-defined key paths. - Due to [limitation in GCS](https://cloud.google.com/storage/docs/authentication/signatures#signing-process), a private key is required to create Pre-signed URL. Currently, OpenDAL only supports Service Account key. ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Gcs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Gcs::default() // set the storage bucket for OpenDAL .bucket("test") // set the working directory root for GCS // all operations will happen within it .root("/path/to/dir") // set the credentials with service account .credential("service account JSON in base64") // set the predefined ACL for GCS .predefined_acl("publicRead") // set the default storage class for GCS .default_storage_class("STANDARD"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/gcs/error.rs000064400000000000000000000066761046102023000162360ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use serde::Deserialize; use serde_json::de; use crate::raw::*; use crate::*; #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "camelCase")] struct GcsErrorResponse { error: GcsError, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "camelCase")] struct GcsError { code: usize, message: String, errors: Vec, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "camelCase")] struct GcsErrorDetail { domain: String, location: String, location_type: String, message: String, reason: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::NOT_MODIFIED => { (ErrorKind::ConditionNotMatch, false) } StatusCode::TOO_MANY_REQUESTS => (ErrorKind::RateLimited, true), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = match de::from_slice::(&bs) { Ok(gcs_err) => format!("{gcs_err:?}"), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod tests { use super::*; #[test] fn test_parse_error() { let bs = bytes::Bytes::from( r#" { "error": { "errors": [ { "domain": "global", "reason": "required", "message": "Login Required", "locationType": "header", "location": "Authorization" } ], "code": 401, "message": "Login Required" } } "#, ); let out: GcsErrorResponse = de::from_slice(&bs).expect("must success"); println!("{out:?}"); assert_eq!(out.error.code, 401); assert_eq!(out.error.message, "Login Required"); assert_eq!(out.error.errors[0].domain, "global"); assert_eq!(out.error.errors[0].reason, "required"); assert_eq!(out.error.errors[0].message, "Login Required"); assert_eq!(out.error.errors[0].location_type, "header"); assert_eq!(out.error.errors[0].location, "Authorization"); } } opendal-0.52.0/src/services/gcs/lister.rs000064400000000000000000000077071046102023000164030ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use serde_json; use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; /// GcsLister takes over task of listing objects and /// helps walking directory pub struct GcsLister { core: Arc, path: String, delimiter: &'static str, limit: Option, /// Filter results to objects whose names are lexicographically /// **equal to or after** startOffset start_after: Option, } impl GcsLister { /// Generate a new directory walker pub fn new( core: Arc, path: &str, recursive: bool, limit: Option, start_after: Option<&str>, ) -> Self { let delimiter = if recursive { "" } else { "/" }; Self { core, path: path.to_string(), delimiter, limit, start_after: start_after.map(String::from), } } } impl oio::PageList for GcsLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .gcs_list_objects( &self.path, &ctx.token, self.delimiter, self.limit, if ctx.token.is_empty() { self.start_after.clone() } else { None }, ) .await?; if !resp.status().is_success() { return Err(parse_error(resp)); } let bytes = resp.into_body(); let output: ListResponse = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; if let Some(token) = &output.next_page_token { ctx.token.clone_from(token); } else { ctx.done = true; } for prefix in output.prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de); } for object in output.items { // exclude the inclusive start_after itself let mut path = build_rel_path(&self.core.root, &object.name); if path.is_empty() { path = "/".to_string(); } if self.start_after.as_ref() == Some(&path) { continue; } let mut meta = Metadata::new(EntryMode::from_path(&path)); // set metadata fields meta.set_content_md5(object.md5_hash.as_str()); meta.set_etag(object.etag.as_str()); let size = object.size.parse().map_err(|e| { Error::new(ErrorKind::Unexpected, "parse u64 from list response").set_source(e) })?; meta.set_content_length(size); if !object.content_type.is_empty() { meta.set_content_type(&object.content_type); } meta.set_last_modified(parse_datetime_from_rfc3339(object.updated.as_str())?); let de = oio::Entry::with(path, meta); ctx.entries.push_back(de); } Ok(()) } } opendal-0.52.0/src/services/gcs/mod.rs000064400000000000000000000023061046102023000156460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-gcs")] mod core; #[cfg(feature = "services-gcs")] mod delete; #[cfg(feature = "services-gcs")] mod error; #[cfg(feature = "services-gcs")] mod lister; #[cfg(feature = "services-gcs")] mod uri; #[cfg(feature = "services-gcs")] mod writer; #[cfg(feature = "services-gcs")] mod backend; #[cfg(feature = "services-gcs")] pub use backend::GcsBuilder as Gcs; mod config; pub use config::GcsConfig; opendal-0.52.0/src/services/gcs/uri.rs000064400000000000000000000054041046102023000156700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use percent_encoding::utf8_percent_encode; use percent_encoding::AsciiSet; use percent_encoding::NON_ALPHANUMERIC; /// PATH_ENCODE_SET is the encode set for http url path. /// /// This set follows [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) which will encode all non-ASCII characters except `A-Z a-z 0-9 - _ . ! ~ * ' ( )` /// /// Following characters is allowed in GCS, check "https://cloud.google.com/storage/docs/request-endpoints#encoding" for details static GCS_PATH_ENCODE_SET: AsciiSet = NON_ALPHANUMERIC .remove(b'-') .remove(b'_') .remove(b'.') .remove(b'*'); /// percent_encode_path will do percent encoding for http encode path. /// /// Follows [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) /// which will encode all non-ASCII characters except `A-Z a-z 0-9 - _ . *` /// /// GCS does not allow '/'s in paths, this should also be dealt with pub(super) fn percent_encode_path(path: &str) -> String { utf8_percent_encode(path, &GCS_PATH_ENCODE_SET).to_string() } #[cfg(test)] mod tests { use super::*; #[test] fn test_percent_encode_path() { let cases = vec![ ( "Reserved Characters", ";,/?:@&=+$", "%3B%2C%2F%3F%3A%40%26%3D%2B%24", ), ("Unescaped Characters", "-_.*", "-_.*"), ("Number Sign", "#", "%23"), ( "Alphanumeric Characters + Space", "ABC abc 123", "ABC%20abc%20123", ), ( "Unicode", "你好，世界！❤", "%E4%BD%A0%E5%A5%BD%EF%BC%8C%E4%B8%96%E7%95%8C%EF%BC%81%E2%9D%A4", ), ]; for (name, input, expected) in cases { let actual = percent_encode_path(input); assert_eq!(actual, expected, "{name}"); } } } opendal-0.52.0/src/services/gcs/writer.rs000064400000000000000000000105241046102023000164040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use super::core::CompleteMultipartUploadRequestPart; use super::core::GcsCore; use super::core::InitiateMultipartUploadResult; use super::error::parse_error; use crate::raw::*; use crate::*; pub type GcsWriters = oio::MultipartWriter; pub struct GcsWriter { core: Arc, path: String, op: OpWrite, } impl GcsWriter { pub fn new(core: Arc, path: &str, op: OpWrite) -> Self { GcsWriter { core, path: path.to_string(), op, } } } impl oio::MultipartWrite for GcsWriter { async fn write_once(&self, _: u64, body: Buffer) -> Result { let size = body.len() as u64; let mut req = self.core.gcs_insert_object_request( &percent_encode_path(&self.path), Some(size), &self.op, body, )?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn initiate_part(&self) -> Result { let resp = self .core .gcs_initiate_multipart_upload(&percent_encode_path(&self.path)) .await?; if !resp.status().is_success() { return Err(parse_error(resp)); } let buf = resp.into_body(); let upload_id: InitiateMultipartUploadResult = quick_xml::de::from_reader(buf.reader()).map_err(new_xml_deserialize_error)?; Ok(upload_id.upload_id) } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result { // Gcs requires part number must between [1..=10000] let part_number = part_number + 1; let resp = self .core .gcs_upload_part(&self.path, upload_id, part_number, size, body) .await?; if !resp.status().is_success() { return Err(parse_error(resp)); } let etag = parse_etag(resp.headers())? .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "ETag not present in returning response", ) })? .to_string(); Ok(oio::MultipartPart { part_number, etag, checksum: None, }) } async fn complete_part( &self, upload_id: &str, parts: &[oio::MultipartPart], ) -> Result { let parts = parts .iter() .map(|p| CompleteMultipartUploadRequestPart { part_number: p.part_number, etag: p.etag.clone(), }) .collect(); let resp = self .core .gcs_complete_multipart_upload(&self.path, upload_id, parts) .await?; if !resp.status().is_success() { return Err(parse_error(resp)); } Ok(Metadata::default()) } async fn abort_part(&self, upload_id: &str) -> Result<()> { let resp = self .core .gcs_abort_multipart_upload(&self.path, upload_id) .await?; match resp.status() { // gcs returns code 204 if abort succeeds. StatusCode::NO_CONTENT => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/gdrive/backend.rs000064400000000000000000000203631046102023000171650ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::sync::Arc; use bytes::Buf; use bytes::Bytes; use chrono::Utc; use http::Request; use http::Response; use http::StatusCode; use serde_json::json; use super::core::GdriveCore; use super::core::GdriveFile; use super::delete::GdriveDeleter; use super::error::parse_error; use super::lister::GdriveLister; use super::writer::GdriveWriter; use crate::raw::*; use crate::*; #[derive(Clone, Debug)] pub struct GdriveBackend { pub core: Arc, } impl Access for GdriveBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut ma = AccessorInfo::default(); ma.set_scheme(Scheme::Gdrive) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_last_modified: true, read: true, list: true, list_has_content_type: true, list_has_content_length: true, list_has_etag: true, write: true, create_dir: true, delete: true, rename: true, copy: true, shared: true, ..Default::default() }); ma.into() } async fn create_dir(&self, path: &str, _args: OpCreateDir) -> Result { let path = build_abs_path(&self.core.root, path); let _ = self.core.path_cache.ensure_dir(&path).await?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _args: OpStat) -> Result { let resp = self.core.gdrive_stat(path).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let gdrive_file: GdriveFile = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let file_type = if gdrive_file.mime_type == "application/vnd.google-apps.folder" { EntryMode::DIR } else { EntryMode::FILE }; let mut meta = Metadata::new(file_type).with_content_type(gdrive_file.mime_type); if let Some(v) = gdrive_file.size { meta = meta.with_content_length(v.parse::().map_err(|e| { Error::new(ErrorKind::Unexpected, "parse content length").set_source(e) })?); } if let Some(v) = gdrive_file.modified_time { meta = meta.with_last_modified(v.parse::>().map_err(|e| { Error::new(ErrorKind::Unexpected, "parse last modified time").set_source(e) })?); } Ok(RpStat::new(meta)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.gdrive_get(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, _: OpWrite) -> Result<(RpWrite, Self::Writer)> { let path = build_abs_path(&self.core.root, path); // As Google Drive allows files have the same name, we need to check if the file exists. // If the file exists, we will keep its ID and update it. let file_id = self.core.path_cache.get(&path).await?; Ok(( RpWrite::default(), oio::OneShotWriter::new(GdriveWriter::new(self.core.clone(), path, file_id)), )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(GdriveDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, _args: OpList) -> Result<(RpList, Self::Lister)> { let path = build_abs_path(&self.core.root, path); let l = GdriveLister::new(path, self.core.clone()); Ok((RpList::default(), oio::PageLister::new(l))) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let from = build_abs_path(&self.core.root, from); let from_file_id = self.core.path_cache.get(&from).await?.ok_or(Error::new( ErrorKind::NotFound, "the file to copy does not exist", ))?; let to_name = get_basename(to); let to_path = build_abs_path(&self.core.root, to); let to_parent_id = self .core .path_cache .ensure_dir(get_parent(&to_path)) .await?; // copy will overwrite `to`, delete it if exist if let Some(id) = self.core.path_cache.get(&to_path).await? { let resp = self.core.gdrive_trash(&id).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } self.core.path_cache.remove(&to_path).await; } let url = format!( "https://www.googleapis.com/drive/v3/files/{}/copy", from_file_id ); let request_body = &json!({ "name": to_name, "parents": [to_parent_id], }); let body = Buffer::from(Bytes::from(request_body.to_string())); let mut req = Request::post(&url) .body(body) .map_err(new_request_build_error)?; self.core.sign(&mut req).await?; let resp = self.core.client.send(req).await?; match resp.status() { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let source = build_abs_path(&self.core.root, from); let target = build_abs_path(&self.core.root, to); // rename will overwrite `to`, delete it if exist if let Some(id) = self.core.path_cache.get(&target).await? { let resp = self.core.gdrive_trash(&id).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } self.core.path_cache.remove(&target).await; } let resp = self .core .gdrive_patch_metadata_request(&source, &target) .await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let meta: GdriveFile = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; let cache = &self.core.path_cache; cache.remove(&build_abs_path(&self.core.root, from)).await; cache .insert(&build_abs_path(&self.core.root, to), &meta.id) .await; Ok(RpRename::default()) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/gdrive/builder.rs000064400000000000000000000153611046102023000172260ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use chrono::DateTime; use chrono::Utc; use log::debug; use tokio::sync::Mutex; use super::backend::GdriveBackend; use super::core::GdriveCore; use super::core::GdrivePathQuery; use super::core::GdriveSigner; use crate::raw::normalize_root; use crate::raw::Access; use crate::raw::HttpClient; use crate::raw::PathCacher; use crate::services::GdriveConfig; use crate::Scheme; use crate::*; impl Configurator for GdriveConfig { type Builder = GdriveBuilder; fn into_builder(self) -> Self::Builder { GdriveBuilder { config: self, http_client: None, } } } /// [GoogleDrive](https://drive.google.com/) backend support. #[derive(Default)] #[doc = include_str!("docs.md")] pub struct GdriveBuilder { config: GdriveConfig, http_client: Option, } impl Debug for GdriveBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("config", &self.config) .finish() } } impl GdriveBuilder { /// Set root path of GoogleDrive folder. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Access token is used for temporary access to the GoogleDrive API. /// /// You can get the access token from [GoogleDrive App Console](https://console.cloud.google.com/apis/credentials) /// or [GoogleDrive OAuth2 Playground](https://developers.google.com/oauthplayground/) /// /// # Note /// /// - An access token is valid for 1 hour. /// - If you want to use the access token for a long time, /// you can use the refresh token to get a new access token. pub fn access_token(mut self, access_token: &str) -> Self { self.config.access_token = Some(access_token.to_string()); self } /// Refresh token is used for long term access to the GoogleDrive API. /// /// You can get the refresh token via OAuth 2.0 Flow of GoogleDrive API. /// /// OpenDAL will use this refresh token to get a new access token when the old one is expired. pub fn refresh_token(mut self, refresh_token: &str) -> Self { self.config.refresh_token = Some(refresh_token.to_string()); self } /// Set the client id for GoogleDrive. /// /// This is required for OAuth 2.0 Flow to refresh the access token. pub fn client_id(mut self, client_id: &str) -> Self { self.config.client_id = Some(client_id.to_string()); self } /// Set the client secret for GoogleDrive. /// /// This is required for OAuth 2.0 Flow with refresh the access token. pub fn client_secret(mut self, client_secret: &str) -> Self { self.config.client_secret = Some(client_secret.to_string()); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, http_client: HttpClient) -> Self { self.http_client = Some(http_client); self } } impl Builder for GdriveBuilder { const SCHEME: Scheme = Scheme::Gdrive; type Config = GdriveConfig; fn build(self) -> Result { let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Gdrive) })? }; let mut signer = GdriveSigner::new(client.clone()); match (self.config.access_token, self.config.refresh_token) { (Some(access_token), None) => { signer.access_token = access_token; // We will never expire user specified access token. signer.expires_in = DateTime::::MAX_UTC; } (None, Some(refresh_token)) => { let client_id = self.config.client_id.ok_or_else(|| { Error::new( ErrorKind::ConfigInvalid, "client_id must be set when refresh_token is set", ) .with_context("service", Scheme::Gdrive) })?; let client_secret = self.config.client_secret.ok_or_else(|| { Error::new( ErrorKind::ConfigInvalid, "client_secret must be set when refresh_token is set", ) .with_context("service", Scheme::Gdrive) })?; signer.refresh_token = refresh_token; signer.client = client.clone(); signer.client_id = client_id; signer.client_secret = client_secret; } (Some(_), Some(_)) => { return Err(Error::new( ErrorKind::ConfigInvalid, "access_token and refresh_token cannot be set at the same time", ) .with_context("service", Scheme::Gdrive)) } (None, None) => { return Err(Error::new( ErrorKind::ConfigInvalid, "access_token or refresh_token must be set", ) .with_context("service", Scheme::Gdrive)) } }; let signer = Arc::new(Mutex::new(signer)); Ok(GdriveBackend { core: Arc::new(GdriveCore { root, signer: signer.clone(), client: client.clone(), path_cache: PathCacher::new(GdrivePathQuery::new(client, signer)).with_lock(), }), }) } } opendal-0.52.0/src/services/gdrive/config.rs000064400000000000000000000031521046102023000170400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// [GoogleDrive](https://drive.google.com/) configuration. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct GdriveConfig { /// The root for gdrive pub root: Option, /// Access token for gdrive. pub access_token: Option, /// Refresh token for gdrive. pub refresh_token: Option, /// Client id for gdrive. pub client_id: Option, /// Client secret for gdrive. pub client_secret: Option, } impl Debug for GdriveConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("GdriveConfig") .field("root", &self.root) .finish_non_exhaustive() } } opendal-0.52.0/src/services/gdrive/core.rs000064400000000000000000000346241046102023000165330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes; use bytes::Buf; use bytes::Bytes; use chrono::DateTime; use chrono::Utc; use http::header; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use serde_json::json; use tokio::sync::Mutex; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct GdriveCore { pub root: String, pub client: HttpClient, pub signer: Arc>, /// Cache the mapping from path to file id pub path_cache: PathCacher, } impl Debug for GdriveCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("GdriveCore"); de.field("root", &self.root); de.finish() } } impl GdriveCore { pub async fn gdrive_stat(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let file_id = self.path_cache.get(&path).await?.ok_or(Error::new( ErrorKind::NotFound, format!("path not found: {}", path), ))?; // The file metadata in the Google Drive API is very complex. // For now, we only need the file id, name, mime type and modified time. let mut req = Request::get(format!( "https://www.googleapis.com/drive/v3/files/{}?fields=id,name,mimeType,size,modifiedTime", file_id )) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.send(req).await } pub async fn gdrive_get(&self, path: &str, range: BytesRange) -> Result> { let path = build_abs_path(&self.root, path); let path_id = self.path_cache.get(&path).await?.ok_or(Error::new( ErrorKind::NotFound, format!("path not found: {}", path), ))?; let url: String = format!( "https://www.googleapis.com/drive/v3/files/{}?alt=media", path_id ); let mut req = Request::get(&url) .header(header::RANGE, range.to_header()) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.fetch(req).await } pub async fn gdrive_list( &self, file_id: &str, page_size: i32, next_page_token: &str, ) -> Result> { let q = format!("'{}' in parents and trashed = false", file_id); let mut url = format!( "https://www.googleapis.com/drive/v3/files?pageSize={}&q={}", page_size, percent_encode_path(&q) ); if !next_page_token.is_empty() { url += &format!("&pageToken={next_page_token}"); }; let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.send(req).await } // Update with content and metadata pub async fn gdrive_patch_metadata_request( &self, source: &str, target: &str, ) -> Result> { let source_file_id = self.path_cache.get(source).await?.ok_or(Error::new( ErrorKind::NotFound, format!("source path not found: {}", source), ))?; let source_parent = get_parent(source); let source_parent_id = self .path_cache .get(source_parent) .await? .expect("old parent must exist"); let target_parent_id = self.path_cache.ensure_dir(get_parent(target)).await?; let target_file_name = get_basename(target); let metadata = &json!({ "name": target_file_name, "removeParents": [source_parent_id], "addParents": [target_parent_id], }); let url = format!( "https://www.googleapis.com/drive/v3/files/{}", source_file_id ); let mut req = Request::patch(url) .body(Buffer::from(Bytes::from(metadata.to_string()))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.send(req).await } pub async fn gdrive_trash(&self, file_id: &str) -> Result> { let url = format!("https://www.googleapis.com/drive/v3/files/{}", file_id); let body = serde_json::to_vec(&json!({ "trashed": true })) .map_err(new_json_serialize_error)?; let mut req = Request::patch(&url) .body(Buffer::from(Bytes::from(body))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.send(req).await } /// Create a file with the content. pub async fn gdrive_upload_simple_request( &self, path: &str, size: u64, body: Buffer, ) -> Result> { let parent = self.path_cache.ensure_dir(get_parent(path)).await?; let url = "https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart"; let file_name = get_basename(path); let metadata = serde_json::to_vec(&json!({ "name": file_name, "parents": [parent], })) .map_err(new_json_serialize_error)?; let req = Request::post(url).header("X-Upload-Content-Length", size); let multipart = Multipart::new() .part( FormDataPart::new("metadata") .header( header::CONTENT_TYPE, "application/json; charset=UTF-8".parse().unwrap(), ) .content(metadata), ) .part( FormDataPart::new("file") .header( header::CONTENT_TYPE, "application/octet-stream".parse().unwrap(), ) .content(body), ); let mut req = multipart.apply(req)?; self.sign(&mut req).await?; self.client.send(req).await } /// Overwrite the file with the content. /// /// # Notes /// /// - The file id is required. Do not use this method to create a file. pub async fn gdrive_upload_overwrite_simple_request( &self, file_id: &str, size: u64, body: Buffer, ) -> Result> { let url = format!( "https://www.googleapis.com/upload/drive/v3/files/{}?uploadType=media", file_id ); let mut req = Request::patch(url) .header(header::CONTENT_TYPE, "application/octet-stream") .header(header::CONTENT_LENGTH, size) .header("X-Upload-Content-Length", size) .body(body) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.client.send(req).await } pub async fn sign(&self, req: &mut Request) -> Result<()> { let mut signer = self.signer.lock().await; signer.sign(req).await } } #[derive(Clone)] pub struct GdriveSigner { pub client: HttpClient, pub client_id: String, pub client_secret: String, pub refresh_token: String, pub access_token: String, pub expires_in: DateTime, } impl GdriveSigner { /// Create a new signer. pub fn new(client: HttpClient) -> Self { GdriveSigner { client, client_id: "".to_string(), client_secret: "".to_string(), refresh_token: "".to_string(), access_token: "".to_string(), expires_in: DateTime::::MIN_UTC, } } /// Sign a request. pub async fn sign(&mut self, req: &mut Request) -> Result<()> { if !self.access_token.is_empty() && self.expires_in > Utc::now() { let value = format!("Bearer {}", self.access_token) .parse() .expect("access token must be valid header value"); req.headers_mut().insert(header::AUTHORIZATION, value); return Ok(()); } let url = format!( "https://oauth2.googleapis.com/token?refresh_token={}&client_id={}&client_secret={}&grant_type=refresh_token", self.refresh_token, self.client_id, self.client_secret ); { let req = Request::post(url) .header(header::CONTENT_LENGTH, 0) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let token: GdriveTokenResponse = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; self.access_token.clone_from(&token.access_token); self.expires_in = Utc::now() + chrono::TimeDelta::try_seconds(token.expires_in) .expect("expires_in must be valid seconds") - chrono::TimeDelta::try_seconds(120).expect("120 must be valid seconds"); } _ => { return Err(parse_error(resp)); } } } let auth_header_content = format!("Bearer {}", self.access_token); req.headers_mut() .insert(header::AUTHORIZATION, auth_header_content.parse().unwrap()); Ok(()) } } pub struct GdrivePathQuery { pub client: HttpClient, pub signer: Arc>, } impl GdrivePathQuery { pub fn new(client: HttpClient, signer: Arc>) -> Self { GdrivePathQuery { client, signer } } } impl PathQuery for GdrivePathQuery { async fn root(&self) -> Result { Ok("root".to_string()) } async fn query(&self, parent_id: &str, name: &str) -> Result> { let mut queries = vec![ // Make sure name has been replaced with escaped name. // // ref: format!( "name = '{}'", name.replace('\'', "\\'").trim_end_matches('/') ), format!("'{}' in parents", parent_id), "trashed = false".to_string(), ]; if name.ends_with('/') { queries.push("mimeType = 'application/vnd.google-apps.folder'".to_string()); } let query = queries.join(" and "); let url = format!( "https://www.googleapis.com/drive/v3/files?q={}", percent_encode_path(query.as_str()) ); let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.signer.lock().await.sign(&mut req).await?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let meta: GdriveFileList = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; if let Some(f) = meta.files.first() { Ok(Some(f.id.clone())) } else { Ok(None) } } _ => Err(parse_error(resp)), } } async fn create_dir(&self, parent_id: &str, name: &str) -> Result { let url = "https://www.googleapis.com/drive/v3/files"; let content = serde_json::to_vec(&json!({ "name": name.trim_end_matches('/'), "mimeType": "application/vnd.google-apps.folder", // If the parent is not provided, the folder will be created in the root folder. "parents": [parent_id], })) .map_err(new_json_serialize_error)?; let mut req = Request::post(url) .header(header::CONTENT_TYPE, "application/json") .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; self.signer.lock().await.sign(&mut req).await?; let resp = self.client.send(req).await?; if !resp.status().is_success() { return Err(parse_error(resp)); } let body = resp.into_body(); let file: GdriveFile = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; Ok(file.id) } } #[derive(Deserialize)] pub struct GdriveTokenResponse { access_token: String, expires_in: i64, } /// This is the file struct returned by the Google Drive API. /// This is a complex struct, but we only add the fields we need. /// refer to https://developers.google.com/drive/api/reference/rest/v3/files#File #[derive(Deserialize, Debug)] #[serde(rename_all = "camelCase")] pub struct GdriveFile { pub mime_type: String, pub id: String, pub name: String, pub size: Option, // The modified time is not returned unless the `fields` // query parameter contains `modifiedTime`. // As we only need the modified time when we do `stat` operation, // if other operations(such as search) do not specify the `fields` query parameter, // try to access this field, it will be `None`. pub modified_time: Option, } /// refer to https://developers.google.com/drive/api/reference/rest/v3/files/list #[derive(Deserialize)] #[serde(rename_all = "camelCase")] pub(crate) struct GdriveFileList { pub(crate) files: Vec, pub(crate) next_page_token: Option, } opendal-0.52.0/src/services/gdrive/delete.rs000064400000000000000000000032551046102023000170410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct GdriveDeleter { core: Arc, } impl GdriveDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for GdriveDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let path = build_abs_path(&self.core.root, &path); let file_id = self.core.path_cache.get(&path).await?; let file_id = if let Some(id) = file_id { id } else { return Ok(()); }; let resp = self.core.gdrive_trash(&file_id).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } self.core.path_cache.remove(&path).await; Ok(()) } } opendal-0.52.0/src/services/gdrive/docs.md000064400000000000000000000030111046102023000164710ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] delete - [x] create_dir - [x] list - [x] copy - [x] rename - [ ] batch # Configuration - `root`: Set the work directory for backend ### Credentials related #### Just provide Access Token (Temporary) - `access_token`: set the access_token for google drive api Please notice its expiration. #### Or provide Client ID and Client Secret and refresh token (Long Term) If you want to let OpenDAL to refresh the access token automatically, please provide the following fields: - `refresh_token`: set the refresh_token for google drive api - `client_id`: set the client_id for google drive api - `client_secret`: set the client_secret for google drive api OpenDAL is a library, it cannot do the first step of OAuth2 for you. You need to get authorization code from user by calling GoogleDrive's authorize url and exchange it for refresh token. Make sure you have enabled Google Drive API in your Google Cloud Console. And your OAuth scope contains `https://www.googleapis.com/auth/drive`. Please refer to [GoogleDrive OAuth2 Flow](https://developers.google.com/identity/protocols/oauth2/) for more information. You can refer to [`GdriveBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Gdrive; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Gdrive::default() .root("/test") .access_token(""); Ok(()) } opendal-0.52.0/src/services/gdrive/error.rs000064400000000000000000000052701046102023000167270ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use serde::Deserialize; use crate::raw::*; use crate::*; #[derive(Default, Debug, Deserialize)] struct GdriveError { error: GdriveInnerError, } #[derive(Default, Debug, Deserialize)] struct GdriveInnerError { message: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (mut kind, mut retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT // Gdrive sometimes return METHOD_NOT_ALLOWED for our requests for abuse detection. | StatusCode::METHOD_NOT_ALLOWED => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, gdrive_err) = serde_json::from_slice::(bs.as_ref()) .map(|gdrive_err| (format!("{gdrive_err:?}"), Some(gdrive_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); if let Some(gdrive_err) = gdrive_err { (kind, retryable) = parse_gdrive_error_code(gdrive_err.error.message.as_str()).unwrap_or((kind, retryable)); } let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } pub fn parse_gdrive_error_code(message: &str) -> Option<(ErrorKind, bool)> { match message { // > Please reduce your request rate. // // It's Ok to retry since later on the request rate may get reduced. "User rate limit exceeded." => Some((ErrorKind::RateLimited, true)), _ => None, } } opendal-0.52.0/src/services/gdrive/lister.rs000064400000000000000000000071521046102023000171010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::GdriveCore; use super::core::GdriveFileList; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct GdriveLister { path: String, core: Arc, } impl GdriveLister { pub fn new(path: String, core: Arc) -> Self { Self { path, core } } } impl oio::PageList for GdriveLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let file_id = self.core.path_cache.get(&self.path).await?; let file_id = match file_id { Some(file_id) => file_id, None => { ctx.done = true; return Ok(()); } }; let resp = self .core .gdrive_list(file_id.as_str(), 100, &ctx.token) .await?; let bytes = match resp.status() { StatusCode::OK => resp.into_body().to_bytes(), _ => return Err(parse_error(resp)), }; // Google Drive returns an empty response when attempting to list a non-existent directory. if bytes.is_empty() { ctx.done = true; return Ok(()); } // Include the current directory itself when handling the first page of the listing. if ctx.token.is_empty() && !ctx.done { let path = build_rel_path(&self.core.root, &self.path); let e = oio::Entry::new(&path, Metadata::new(EntryMode::DIR)); ctx.entries.push_back(e); } let decoded_response = serde_json::from_slice::(&bytes).map_err(new_json_deserialize_error)?; if let Some(next_page_token) = decoded_response.next_page_token { ctx.token = next_page_token; } else { ctx.done = true; } for mut file in decoded_response.files { let file_type = if file.mime_type.as_str() == "application/vnd.google-apps.folder" { if !file.name.ends_with('/') { file.name += "/"; } EntryMode::DIR } else { EntryMode::FILE }; let root = &self.core.root; let path = format!("{}{}", &self.path, file.name); let normalized_path = build_rel_path(root, &path); // Update path cache when path doesn't exist. // When Google Drive converts a format, for example, Microsoft PowerPoint, // Google Drive keeps two entries with the same ID. if let Ok(None) = self.core.path_cache.get(&path).await { self.core.path_cache.insert(&path, &file.id).await; } let entry = oio::Entry::new(&normalized_path, Metadata::new(file_type)); ctx.entries.push_back(entry); } Ok(()) } } opendal-0.52.0/src/services/gdrive/mod.rs000064400000000000000000000023531046102023000163540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-gdrive")] mod backend; #[cfg(feature = "services-gdrive")] mod core; #[cfg(feature = "services-gdrive")] mod delete; #[cfg(feature = "services-gdrive")] mod error; #[cfg(feature = "services-gdrive")] mod lister; #[cfg(feature = "services-gdrive")] mod writer; #[cfg(feature = "services-gdrive")] mod builder; #[cfg(feature = "services-gdrive")] pub use builder::GdriveBuilder as Gdrive; mod config; pub use config::GdriveConfig; opendal-0.52.0/src/services/gdrive/writer.rs000064400000000000000000000045701046102023000171140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use super::core::GdriveCore; use super::core::GdriveFile; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct GdriveWriter { core: Arc, path: String, file_id: Option, } impl GdriveWriter { pub fn new(core: Arc, path: String, file_id: Option) -> Self { GdriveWriter { core, path, file_id, } } } impl oio::OneShotWrite for GdriveWriter { async fn write_once(&self, bs: Buffer) -> Result { let size = bs.len(); let resp = if let Some(file_id) = &self.file_id { self.core .gdrive_upload_overwrite_simple_request(file_id, size as u64, bs) .await } else { self.core .gdrive_upload_simple_request(&self.path, size as u64, bs) .await }?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => { // If we don't have the file id before, let's update the cache to avoid re-fetching. if self.file_id.is_none() { let bs = resp.into_body(); let file: GdriveFile = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; self.core.path_cache.insert(&self.path, &file.id).await; } Ok(Metadata::default()) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/ghac/backend.rs000064400000000000000000000227431046102023000166130ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::env; use std::sync::Arc; use super::core::*; use super::error::parse_error; use super::writer::GhacWriter; use crate::raw::*; use crate::services::ghac::core::GhacCore; use crate::services::GhacConfig; use crate::*; use http::header; use http::Request; use http::Response; use http::StatusCode; use log::debug; use sha2::Digest; fn value_or_env( explicit_value: Option, env_var_name: &str, operation: &'static str, ) -> Result { if let Some(value) = explicit_value { return Ok(value); } env::var(env_var_name).map_err(|err| { let text = format!( "{} not found, maybe not in github action environment?", env_var_name ); Error::new(ErrorKind::ConfigInvalid, text) .with_operation(operation) .set_source(err) }) } impl Configurator for GhacConfig { type Builder = GhacBuilder; fn into_builder(self) -> Self::Builder { GhacBuilder { config: self, http_client: None, } } } /// GitHub Action Cache Services support. #[doc = include_str!("docs.md")] #[derive(Debug, Default)] pub struct GhacBuilder { config: GhacConfig, http_client: Option, } impl GhacBuilder { /// set the working directory root of backend pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// set the version that used by cache. /// /// The version is the unique value that provides namespacing. /// It's better to make sure this value is only used by this backend. /// /// If not set, we will use `opendal` as default. pub fn version(mut self, version: &str) -> Self { if !version.is_empty() { self.config.version = Some(version.to_string()) } self } /// Set the endpoint for ghac service. /// /// For example, this is provided as the `ACTIONS_CACHE_URL` environment variable by the GHA runner. /// /// Default: the value of the `ACTIONS_CACHE_URL` environment variable. pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { self.config.endpoint = Some(endpoint.to_string()) } self } /// Set the runtime token for ghac service. /// /// For example, this is provided as the `ACTIONS_RUNTIME_TOKEN` environment variable by the GHA /// runner. /// /// Default: the value of the `ACTIONS_RUNTIME_TOKEN` environment variable. pub fn runtime_token(mut self, runtime_token: &str) -> Self { if !runtime_token.is_empty() { self.config.runtime_token = Some(runtime_token.to_string()) } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for GhacBuilder { const SCHEME: Scheme = Scheme::Ghac; type Config = GhacConfig; fn build(self) -> Result { debug!("backend build started: {:?}", self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let service_version = get_cache_service_version(); debug!("backend use service version {:?}", service_version); let mut version = self .config .version .clone() .unwrap_or_else(|| "opendal".to_string()); debug!("backend use version {version}"); // ghac requires to use hex digest of Sha256 as version. if matches!(service_version, GhacVersion::V2) { let hash = sha2::Sha256::digest(&version); version = format!("{:x}", hash); } let cache_url = self .config .endpoint .unwrap_or_else(|| get_cache_service_url(service_version)); if cache_url.is_empty() { return Err(Error::new( ErrorKind::ConfigInvalid, "cache url for ghac not found, maybe not in github action environment?".to_string(), )); } let http_client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Ghac) })? }; let core = GhacCore { root, cache_url, catch_token: value_or_env( self.config.runtime_token, ACTIONS_RUNTIME_TOKEN, "Builder::build", )?, version, service_version, http_client, }; Ok(GhacBackend { core: Arc::new(core), }) } } /// Backend for github action cache services. #[derive(Debug, Clone)] pub struct GhacBackend { core: Arc, } impl Access for GhacBackend { type Reader = HttpBody; type Writer = GhacWriter; type Lister = (); type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Ghac) .set_root(&self.core.root) .set_name(&self.core.version) .set_native_capability(Capability { stat: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, read: true, write: true, write_can_multi: true, shared: true, ..Default::default() }); am.into() } /// Some self-hosted GHES instances are backed by AWS S3 services which only returns /// signed url with `GET` method. So we will use `GET` with empty range to simulate /// `HEAD` instead. /// /// In this way, we can support both self-hosted GHES and `github.com`. async fn stat(&self, path: &str, _: OpStat) -> Result { let location = self.core.ghac_get_download_url(path).await?; let req = Request::get(location) .header(header::RANGE, "bytes=0-0") .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.core.http_client.send(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT | StatusCode::RANGE_NOT_SATISFIABLE => { let mut meta = parse_into_metadata(path, resp.headers())?; // Correct content length via returning content range. meta.set_content_length( meta.content_range() .expect("content range must be valid") .size() .expect("content range must contains size"), ); Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let location = self.core.ghac_get_download_url(path).await?; let mut req = Request::get(location); if !args.range().is_full() { req = req.header(header::RANGE, args.range().to_header()); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.core.http_client.fetch(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, _: OpWrite) -> Result<(RpWrite, Self::Writer)> { let url = self.core.ghac_get_upload_url(path).await?; Ok(( RpWrite::default(), GhacWriter::new(self.core.clone(), path.to_string(), url)?, )) } } opendal-0.52.0/src/services/ghac/config.rs000064400000000000000000000025051046102023000164630ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// Config for GitHub Action Cache Services support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct GhacConfig { /// The root path for ghac. pub root: Option, /// The version that used by cache. pub version: Option, /// The endpoint for ghac service. pub endpoint: Option, /// The runtime token for ghac service. pub runtime_token: Option, } opendal-0.52.0/src/services/ghac/core.rs000064400000000000000000000347201046102023000161520ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::error::parse_error; use crate::raw::{ build_abs_path, new_json_deserialize_error, new_json_serialize_error, new_request_build_error, percent_encode_path, HttpClient, }; use crate::*; use ::ghac::v1 as ghac_types; use bytes::{Buf, Bytes}; use http::header::{ACCEPT, AUTHORIZATION, CONTENT_LENGTH, CONTENT_TYPE}; use http::{Request, StatusCode, Uri}; use prost::Message; use serde::{Deserialize, Serialize}; use std::env; use std::fmt::{Debug, Formatter}; use std::str::FromStr; /// The base url for cache url. pub const CACHE_URL_BASE: &str = "_apis/artifactcache"; /// The base url for cache service v2. pub const CACHE_URL_BASE_V2: &str = "twirp/github.actions.results.api.v1.CacheService"; /// Cache API requires to provide an accept header. pub const CACHE_HEADER_ACCEPT: &str = "application/json;api-version=6.0-preview.1"; /// The cache url env for ghac. /// /// The url will be like `https://artifactcache.actions.githubusercontent.com//` pub const ACTIONS_CACHE_URL: &str = "ACTIONS_CACHE_URL"; /// The runtime token env for ghac. /// /// This token will be valid for 6h and github action will running for 6 /// hours at most. So we don't need to refetch it again. pub const ACTIONS_RUNTIME_TOKEN: &str = "ACTIONS_RUNTIME_TOKEN"; /// The cache service version env for ghac. pub const ACTIONS_CACHE_SERVICE_V2: &str = "ACTIONS_CACHE_SERVICE_V2"; /// The results url env for ghac. pub const ACTIONS_RESULTS_URL: &str = "ACTIONS_RESULTS_URL"; /// The content type for protobuf. pub const CONTENT_TYPE_JSON: &str = "application/json"; /// The content type for protobuf. pub const CONTENT_TYPE_PROTOBUF: &str = "application/protobuf"; /// The version of github action cache. #[derive(Clone, Copy, Debug)] pub enum GhacVersion { V1, V2, } /// Core for github action cache services. #[derive(Clone)] pub struct GhacCore { // root should end with "/" pub root: String, pub cache_url: String, pub catch_token: String, pub version: String, pub service_version: GhacVersion, pub http_client: HttpClient, } impl Debug for GhacCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("GhacCore") .field("root", &self.root) .field("cache_url", &self.cache_url) .field("version", &self.version) .field("service_version", &self.service_version) .finish_non_exhaustive() } } impl GhacCore { pub async fn ghac_get_download_url(&self, path: &str) -> Result { let p = build_abs_path(&self.root, path); match self.service_version { GhacVersion::V1 => { let url = format!( "{}{CACHE_URL_BASE}/cache?keys={}&version={}", self.cache_url, percent_encode_path(&p), self.version ); let mut req = Request::get(&url); req = req.header(AUTHORIZATION, format!("Bearer {}", self.catch_token)); req = req.header(ACCEPT, CACHE_HEADER_ACCEPT); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.http_client.send(req).await?; let location = if resp.status() == StatusCode::OK { let slc = resp.into_body(); let query_resp: GhacQueryResponse = serde_json::from_reader(slc.reader()) .map_err(new_json_deserialize_error)?; query_resp.archive_location } else { return Err(parse_error(resp)); }; Ok(location) } GhacVersion::V2 => { let url = format!( "{}{CACHE_URL_BASE_V2}/GetCacheEntryDownloadURL", self.cache_url, ); let req = ghac_types::GetCacheEntryDownloadUrlRequest { key: p, version: self.version.clone(), metadata: None, restore_keys: vec![], }; let body = Buffer::from(req.encode_to_vec()); let req = Request::post(&url) .header(AUTHORIZATION, format!("Bearer {}", self.catch_token)) .header(CONTENT_TYPE, CONTENT_TYPE_PROTOBUF) .header(CONTENT_LENGTH, body.len()) .body(body) .map_err(new_request_build_error)?; let resp = self.http_client.send(req).await?; let location = if resp.status() == StatusCode::OK { let slc = resp.into_body(); let query_resp = ghac_types::GetCacheEntryDownloadUrlResponse::decode(slc) .map_err(new_prost_decode_error)?; if !query_resp.ok { let mut err = Error::new( ErrorKind::NotFound, "GetCacheEntryDownloadURL returns non-ok, the key doesn't exist", ); // GHAC is a cache service, so it's acceptable for it to occasionally not contain // data that users have just written. However, we don't want users to always have // to retry reading it, nor do we want our CI to fail constantly. // // Here's the trick: we check if the environment variable `OPENDAL_TEST` is set to `ghac`. // If it is, we mark the error as temporary to allow retries in the test CI. if env::var("OPENDAL_TEST") == Ok("ghac".to_string()) { err = err.set_temporary(); } return Err(err); } query_resp.signed_download_url } else { return Err(parse_error(resp)); }; Ok(location) } } } pub async fn ghac_get_upload_url(&self, path: &str) -> Result { let p = build_abs_path(&self.root, path); match self.service_version { GhacVersion::V1 => { let url = format!("{}{CACHE_URL_BASE}/caches", self.cache_url); let bs = serde_json::to_vec(&GhacReserveRequest { key: p, version: self.version.to_string(), }) .map_err(new_json_serialize_error)?; let mut req = Request::post(&url); req = req.header(AUTHORIZATION, format!("Bearer {}", self.catch_token)); req = req.header(ACCEPT, CACHE_HEADER_ACCEPT); req = req.header(CONTENT_TYPE, CONTENT_TYPE_JSON); req = req.header(CONTENT_LENGTH, bs.len()); let req = req .body(Buffer::from(Bytes::from(bs))) .map_err(new_request_build_error)?; let resp = self.http_client.send(req).await?; let cache_id = if resp.status().is_success() { let slc = resp.into_body(); let reserve_resp: GhacReserveResponse = serde_json::from_reader(slc.reader()) .map_err(new_json_deserialize_error)?; reserve_resp.cache_id } else { return Err( parse_error(resp).map(|err| err.with_operation("Backend::ghac_reserve")) ); }; let url = format!("{}{CACHE_URL_BASE}/caches/{cache_id}", self.cache_url); Ok(url) } GhacVersion::V2 => { let url = format!("{}{CACHE_URL_BASE_V2}/CreateCacheEntry", self.cache_url,); let req = ghac_types::CreateCacheEntryRequest { key: p, version: self.version.clone(), metadata: None, }; let body = Buffer::from(req.encode_to_vec()); let req = Request::post(&url) .header(AUTHORIZATION, format!("Bearer {}", self.catch_token)) .header(CONTENT_TYPE, CONTENT_TYPE_PROTOBUF) .header(CONTENT_LENGTH, body.len()) .body(body) .map_err(new_request_build_error)?; let resp = self.http_client.send(req).await?; let location = if resp.status() == StatusCode::OK { let (parts, slc) = resp.into_parts(); let query_resp = ghac_types::CreateCacheEntryResponse::decode(slc) .map_err(new_prost_decode_error)?; if !query_resp.ok { return Err(Error::new( ErrorKind::Unexpected, "create cache entry returns non-ok", ) .with_context("parts", format!("{:?}", parts))); } query_resp.signed_upload_url } else { return Err(parse_error(resp)); }; Ok(location) } } } pub async fn ghac_finalize_upload(&self, path: &str, url: &str, size: u64) -> Result<()> { let p = build_abs_path(&self.root, path); match self.service_version { GhacVersion::V1 => { let bs = serde_json::to_vec(&GhacCommitRequest { size }) .map_err(new_json_serialize_error)?; let req = Request::post(url) .header(AUTHORIZATION, format!("Bearer {}", self.catch_token)) .header(ACCEPT, CACHE_HEADER_ACCEPT) .header(CONTENT_TYPE, CONTENT_TYPE_JSON) .header(CONTENT_LENGTH, bs.len()) .body(Buffer::from(bs)) .map_err(new_request_build_error)?; let resp = self.http_client.send(req).await?; if resp.status().is_success() { Ok(()) } else { Err(parse_error(resp)) } } GhacVersion::V2 => { let url = format!( "{}{CACHE_URL_BASE_V2}/FinalizeCacheEntryUpload", self.cache_url, ); let req = ghac_types::FinalizeCacheEntryUploadRequest { key: p, version: self.version.clone(), size_bytes: size as i64, metadata: None, }; let body = Buffer::from(req.encode_to_vec()); let req = Request::post(&url) .header(AUTHORIZATION, format!("Bearer {}", self.catch_token)) .header(CONTENT_TYPE, CONTENT_TYPE_PROTOBUF) .header(CONTENT_LENGTH, body.len()) .body(body) .map_err(new_request_build_error)?; let resp = self.http_client.send(req).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); }; Ok(()) } } } } /// Determines if the current environment is GitHub Enterprise Server (GHES) /// /// We need to know this since GHES doesn't support ghac v2 yet. pub fn is_ghes() -> bool { // Fetch GitHub Server URL with fallback to "https://github.com" let server_url = env::var("GITHUB_SERVER_URL").unwrap_or_else(|_| "https://github.com".to_string()); let Ok(url) = Uri::from_str(&server_url) else { // We just return false if the URL is invalid return false; }; // Check against known non-GHES host patterns let hostname = url.host().unwrap_or("").trim_end().to_lowercase(); let is_github_host = hostname == "github.com"; let is_ghe_host = hostname.ends_with(".ghe.com"); let is_localhost = hostname.ends_with(".localhost"); !is_github_host && !is_ghe_host && !is_localhost } /// Determines the cache service version based on environment pub fn get_cache_service_version() -> GhacVersion { if is_ghes() { // GHES only supports v1 regardless of feature flags GhacVersion::V1 } else { // Check for presence of non-empty ACTIONS_CACHE_SERVICE_V2 let value = env::var(ACTIONS_CACHE_SERVICE_V2).unwrap_or_default(); if value.is_empty() { GhacVersion::V1 } else { GhacVersion::V2 } } } /// Returns the appropriate cache service URL based on version pub fn get_cache_service_url(version: GhacVersion) -> String { match version { GhacVersion::V1 => { // Priority order for v1: CACHE_URL -> RESULTS_URL env::var(ACTIONS_CACHE_URL) .or_else(|_| env::var(ACTIONS_RESULTS_URL)) .unwrap_or_default() } GhacVersion::V2 => { // Only RESULTS_URL is used for v2 env::var(ACTIONS_RESULTS_URL).unwrap_or_default() } } } /// Parse prost decode error into opendal::Error. pub fn new_prost_decode_error(e: prost::DecodeError) -> Error { Error::new(ErrorKind::Unexpected, "deserialize protobuf").set_source(e) } #[derive(Deserialize)] #[serde(rename_all = "camelCase")] pub struct GhacQueryResponse { // Not used fields. // cache_key: String, // scope: String, pub archive_location: String, } #[derive(Serialize)] pub struct GhacReserveRequest { pub key: String, pub version: String, } #[derive(Deserialize)] #[serde(rename_all = "camelCase")] pub struct GhacReserveResponse { pub cache_id: i64, } #[derive(Serialize)] pub struct GhacCommitRequest { pub size: u64, } opendal-0.52.0/src/services/ghac/docs.md000064400000000000000000000037551046102023000161320ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [ ] rename - [ ] list - [ ] presign - [ ] blocking ## Notes This service is mainly provided by GitHub actions. Refer to [Caching dependencies to speed up workflows](https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows) for more information. To make this service work as expected, please make sure to either call `endpoint` and `token` to configure the URL and credentials, or that the following environment has been setup correctly: - `ACTIONS_CACHE_URL` - `ACTIONS_RUNTIME_TOKEN` They can be exposed by following action: ```yaml - name: Configure Cache Env uses: actions/github-script@v6 with: script: | core.exportVariable('ACTIONS_CACHE_URL', process.env.ACTIONS_CACHE_URL || ''); core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || ''); ``` To make `delete` work as expected, `GITHUB_TOKEN` should also be set via: ```yaml env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` ## Limitations Unlike other services, ghac doesn't support create empty files. We provide a `enable_create_simulation()` to support this operation but may result unexpected side effects. Also, `ghac` is a cache service which means the data store inside could be automatically evicted at any time. ## Configuration - `root`: Set the work dir for backend. Refer to [`GhacBuilder`]'s public API docs for more information. ## Example ### Via Builder ```no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Ghac; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create ghac backend builder. let mut builder = Ghac::default() // Set the root for ghac, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/path/to/dir"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/ghac/error.rs000064400000000000000000000035031046102023000163460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; use http::Response; use http::StatusCode; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND | StatusCode::NO_CONTENT => (ErrorKind::NotFound, false), StatusCode::CONFLICT => (ErrorKind::AlreadyExists, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::TOO_MANY_REQUESTS => (ErrorKind::RateLimited, true), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let bs = body.to_bytes(); let message = String::from_utf8_lossy(&bs); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/ghac/mod.rs000064400000000000000000000021131046102023000157700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-ghac")] mod error; #[cfg(feature = "services-ghac")] mod writer; #[cfg(feature = "services-ghac")] mod backend; #[cfg(feature = "services-ghac")] pub use backend::GhacBuilder as Ghac; #[cfg(feature = "services-ghac")] mod core; mod config; pub use config::GhacConfig; opendal-0.52.0/src/services/ghac/writer.rs000064400000000000000000000140621046102023000165330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::services::core::AzblobCore; use crate::services::writer::AzblobWriter; use crate::*; use http::header::{ACCEPT, AUTHORIZATION, CONTENT_LENGTH, CONTENT_RANGE, CONTENT_TYPE}; use http::Request; use std::str::FromStr; use std::sync::Arc; pub type GhacWriter = TwoWays; impl GhacWriter { /// TODO: maybe we can move the signed url logic to azblob service instead. pub fn new(core: Arc, write_path: String, url: String) -> Result { match core.service_version { GhacVersion::V1 => Ok(TwoWays::One(GhacWriterV1 { core, path: write_path, url, size: 0, })), GhacVersion::V2 => { let uri = http::Uri::from_str(&url) .map_err(new_http_uri_invalid_error)? .into_parts(); let (Some(scheme), Some(authority), Some(pq)) = (uri.scheme, uri.authority, uri.path_and_query) else { return Err(Error::new( ErrorKind::Unexpected, "ghac returns invalid signed url", ) .with_context("url", &url)); }; let endpoint = format!("{scheme}://{authority}"); let Some((container, path)) = pq.path().trim_matches('/').split_once("/") else { return Err(Error::new( ErrorKind::Unexpected, "ghac returns invalid signed url that bucket or path is missing", ) .with_context("url", &url)); }; let Some(query) = pq.query() else { return Err(Error::new( ErrorKind::Unexpected, "ghac returns invalid signed url that sas is missing", ) .with_context("url", &url)); }; let azure_core = Arc::new(AzblobCore { container: container.to_string(), root: "/".to_string(), endpoint, encryption_key: None, encryption_key_sha256: None, encryption_algorithm: None, client: core.http_client.clone(), loader: { let config = reqsign::AzureStorageConfig { sas_token: Some(query.to_string()), ..Default::default() }; reqsign::AzureStorageLoader::new(config) }, signer: { reqsign::AzureStorageSigner::new() }, }); let w = AzblobWriter::new(azure_core, OpWrite::default(), path.to_string()); let writer = oio::BlockWriter::new(w, None, 4); Ok(TwoWays::Two(GhacWriterV2 { core, writer, path: write_path, url, size: 0, })) } } } } pub struct GhacWriterV1 { core: Arc, path: String, url: String, size: u64, } impl oio::Write for GhacWriterV1 { async fn write(&mut self, bs: Buffer) -> Result<()> { let size = bs.len() as u64; let offset = self.size; let mut req = Request::patch(&self.url); req = req.header(AUTHORIZATION, format!("Bearer {}", self.core.catch_token)); req = req.header(ACCEPT, CACHE_HEADER_ACCEPT); req = req.header(CONTENT_LENGTH, size); req = req.header(CONTENT_TYPE, "application/octet-stream"); req = req.header( CONTENT_RANGE, BytesContentRange::default() .with_range(offset, offset + size - 1) .to_header(), ); let req = req.body(bs).map_err(new_request_build_error)?; let resp = self.core.http_client.send(req).await?; if !resp.status().is_success() { return Err(parse_error(resp).map(|err| err.with_operation("Backend::ghac_upload"))); } self.size += size; Ok(()) } async fn abort(&mut self) -> Result<()> { Ok(()) } async fn close(&mut self) -> Result { self.core .ghac_finalize_upload(&self.path, &self.url, self.size) .await?; Ok(Metadata::default().with_content_length(self.size)) } } pub struct GhacWriterV2 { core: Arc, writer: oio::BlockWriter, path: String, url: String, size: u64, } impl oio::Write for GhacWriterV2 { async fn write(&mut self, bs: Buffer) -> Result<()> { let size = bs.len() as u64; self.writer.write(bs).await?; self.size += size; Ok(()) } async fn close(&mut self) -> Result { self.writer.close().await?; let _ = self .core .ghac_finalize_upload(&self.path, &self.url, self.size) .await; Ok(Metadata::default().with_content_length(self.size)) } async fn abort(&mut self) -> Result<()> { Ok(()) } } opendal-0.52.0/src/services/github/backend.rs000064400000000000000000000204111046102023000171610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::Response; use http::StatusCode; use log::debug; use super::core::Entry; use super::core::GithubCore; use super::delete::GithubDeleter; use super::error::parse_error; use super::lister::GithubLister; use super::writer::GithubWriter; use super::writer::GithubWriters; use crate::raw::*; use crate::services::GithubConfig; use crate::*; impl Configurator for GithubConfig { type Builder = GithubBuilder; fn into_builder(self) -> Self::Builder { GithubBuilder { config: self, http_client: None, } } } /// [github contents](https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28#create-or-update-file-contents) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct GithubBuilder { config: GithubConfig, http_client: Option, } impl Debug for GithubBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("GithubBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl GithubBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Github access_token. /// /// required. pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()); } self } /// Set Github repo owner. pub fn owner(mut self, owner: &str) -> Self { self.config.owner = owner.to_string(); self } /// Set Github repo name. pub fn repo(mut self, repo: &str) -> Self { self.config.repo = repo.to_string(); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for GithubBuilder { const SCHEME: Scheme = Scheme::Github; type Config = GithubConfig; /// Builds the backend and returns the result of GithubBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle owner. if self.config.owner.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "owner is empty") .with_operation("Builder::build") .with_context("service", Scheme::Github)); } debug!("backend use owner {}", &self.config.owner); // Handle repo. if self.config.repo.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "repo is empty") .with_operation("Builder::build") .with_context("service", Scheme::Github)); } debug!("backend use repo {}", &self.config.repo); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Github) })? }; Ok(GithubBackend { core: Arc::new(GithubCore { root, token: self.config.token.clone(), owner: self.config.owner.clone(), repo: self.config.repo.clone(), client, }), }) } } /// Backend for Github services. #[derive(Debug, Clone)] pub struct GithubBackend { core: Arc, } impl Access for GithubBackend { type Reader = HttpBody; type Writer = GithubWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Github) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_etag: true, read: true, create_dir: true, write: true, write_can_empty: true, delete: true, list: true, list_with_recursive: true, list_has_content_length: true, list_has_etag: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let empty_bytes = Buffer::new(); let resp = self .core .upload(&format!("{}.gitkeep", path), empty_bytes) .await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => Ok(RpCreateDir::default()), _ => Err(parse_error(resp)), } } async fn stat(&self, path: &str, _args: OpStat) -> Result { let resp = self.core.stat(path).await?; let status = resp.status(); match status { StatusCode::OK => { let body = resp.into_body(); let resp: Entry = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; let m = if resp.type_field == "dir" { Metadata::new(EntryMode::DIR) } else { Metadata::new(EntryMode::FILE) .with_content_length(resp.size) .with_etag(resp.sha) }; Ok(RpStat::new(m)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.get(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, _args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = GithubWriter::new(self.core.clone(), path.to_string()); let w = oio::OneShotWriter::new(writer); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(GithubDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = GithubLister::new(self.core.clone(), path, args.recursive()); Ok((RpList::default(), oio::PageLister::new(l))) } } opendal-0.52.0/src/services/github/config.rs000064400000000000000000000035401046102023000170430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for GitHub services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct GithubConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, /// GitHub access_token. /// /// optional. /// If not provided, the backend will only support read operations for public repositories. /// And rate limit will be limited to 60 requests per hour. pub token: Option, /// GitHub repo owner. /// /// required. pub owner: String, /// GitHub repo name. /// /// required. pub repo: String, } impl Debug for GithubConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("GithubConfig"); d.field("root", &self.root) .field("owner", &self.owner) .field("repo", &self.repo); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/github/core.rs000064400000000000000000000231161046102023000165270ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use base64::Engine; use bytes::Buf; use bytes::Bytes; use http::header; use http::request; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use serde::Serialize; use super::error::parse_error; use crate::raw::*; use crate::*; /// Core of [github contents](https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28#create-or-update-file-contents) services support. #[derive(Clone)] pub struct GithubCore { /// The root of this core. pub root: String, /// Github access_token. pub token: Option, /// Github repo owner. pub owner: String, /// Github repo name. pub repo: String, pub client: HttpClient, } impl Debug for GithubCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("owner", &self.owner) .field("repo", &self.repo) .finish_non_exhaustive() } } impl GithubCore { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } pub fn sign(&self, req: request::Builder) -> Result { let mut req = req .header(header::USER_AGENT, format!("opendal-{}", VERSION)) .header("X-GitHub-Api-Version", "2022-11-28"); // Github access_token is optional. if let Some(token) = &self.token { req = req.header( header::AUTHORIZATION, format_authorization_by_bearer(token)?, ) } Ok(req) } } impl GithubCore { pub async fn get_file_sha(&self, path: &str) -> Result> { // if the token is not set, we should not try to get the sha of the file. if self.token.is_none() { return Err(Error::new( ErrorKind::PermissionDenied, "Github access_token is not set", )); } let resp = self.stat(path).await?; match resp.status() { StatusCode::OK => { let body = resp.into_body(); let resp: Entry = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; Ok(Some(resp.sha)) } StatusCode::NOT_FOUND => Ok(None), _ => Err(parse_error(resp)), } } pub async fn stat(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://api.github.com/repos/{}/{}/contents/{}", self.owner, self.repo, percent_encode_path(&path) ); let req = Request::get(url); let req = self.sign(req)?; let req = req .header("Accept", "application/vnd.github.object+json") .body(Buffer::new()) .map_err(new_request_build_error)?; self.send(req).await } pub async fn get(&self, path: &str, range: BytesRange) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://api.github.com/repos/{}/{}/contents/{}", self.owner, self.repo, percent_encode_path(&path) ); let req = Request::get(url); let req = self.sign(req)?; let req = req .header(header::ACCEPT, "application/vnd.github.raw+json") .header(header::RANGE, range.to_header()) .body(Buffer::new()) .map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn upload(&self, path: &str, bs: Buffer) -> Result> { let sha = self.get_file_sha(path).await?; let path = build_abs_path(&self.root, path); let url = format!( "https://api.github.com/repos/{}/{}/contents/{}", self.owner, self.repo, percent_encode_path(&path) ); let req = Request::put(url); let req = self.sign(req)?; let mut req_body = CreateOrUpdateContentsRequest { message: format!("Write {} at {} via opendal", path, chrono::Local::now()), content: base64::engine::general_purpose::STANDARD.encode(bs.to_bytes()), sha: None, }; if let Some(sha) = sha { req_body.sha = Some(sha); } let req_body = serde_json::to_vec(&req_body).map_err(new_json_serialize_error)?; let req = req .header("Accept", "application/vnd.github+json") .body(Buffer::from(req_body)) .map_err(new_request_build_error)?; self.send(req).await } pub async fn delete(&self, path: &str) -> Result<()> { // If path is a directory, we should delete path/.gitkeep let formatted_path = format!("{}.gitkeep", path); let p = if path.ends_with('/') { formatted_path.as_str() } else { path }; let Some(sha) = self.get_file_sha(p).await? else { return Ok(()); }; let path = build_abs_path(&self.root, p); let url = format!( "https://api.github.com/repos/{}/{}/contents/{}", self.owner, self.repo, percent_encode_path(&path) ); let req = Request::delete(url); let req = self.sign(req)?; let req_body = DeleteContentsRequest { message: format!("Delete {} at {} via opendal", path, chrono::Local::now()), sha, }; let req_body = serde_json::to_vec(&req_body).map_err(new_json_serialize_error)?; let req = req .header("Accept", "application/vnd.github.object+json") .body(Buffer::from(Bytes::from(req_body))) .map_err(new_request_build_error)?; let resp = self.send(req).await?; match resp.status() { StatusCode::OK => Ok(()), StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } pub async fn list(&self, path: &str) -> Result { let path = build_abs_path(&self.root, path); let url = format!( "https://api.github.com/repos/{}/{}/contents/{}", self.owner, self.repo, percent_encode_path(&path) ); let req = Request::get(url); let req = self.sign(req)?; let req = req .header("Accept", "application/vnd.github.object+json") .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.send(req).await?; match resp.status() { StatusCode::OK => { let body = resp.into_body(); let resp: ListResponse = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; Ok(resp) } StatusCode::NOT_FOUND => Ok(ListResponse::default()), _ => Err(parse_error(resp)), } } /// We use git_url to call github's Tree based API. pub async fn list_with_recursive(&self, git_url: &str) -> Result> { let url = format!("{}?recursive=true", git_url); let req = Request::get(url); let req = self.sign(req)?; let req = req .header("Accept", "application/vnd.github.object+json") .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.send(req).await?; match resp.status() { StatusCode::OK => { let body = resp.into_body(); let resp: ListTreeResponse = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; Ok(resp.tree) } _ => Err(parse_error(resp)), } } } #[derive(Default, Debug, Clone, Serialize)] pub struct CreateOrUpdateContentsRequest { pub message: String, pub content: String, pub sha: Option, } #[derive(Default, Debug, Clone, Serialize)] pub struct DeleteContentsRequest { pub message: String, pub sha: String, } #[derive(Default, Debug, Clone, Deserialize)] pub struct ListTreeResponse { pub tree: Vec, } #[derive(Default, Debug, Clone, Deserialize)] pub struct Tree { pub path: String, #[serde(rename = "type")] pub type_field: String, pub size: Option, pub sha: String, } #[derive(Default, Debug, Clone, Deserialize)] pub struct ListResponse { pub git_url: String, pub entries: Vec, } #[derive(Default, Debug, Clone, Deserialize)] pub struct Entry { pub path: String, pub sha: String, pub size: u64, #[serde(rename = "type")] pub type_field: String, } opendal-0.52.0/src/services/github/delete.rs000064400000000000000000000023771046102023000170470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use crate::raw::*; use crate::*; use std::sync::Arc; pub struct GithubDeleter { core: Arc, } impl GithubDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for GithubDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { match self.core.delete(&path).await { Ok(_) => Ok(()), Err(err) => Err(err), } } } opendal-0.52.0/src/services/github/docs.md000064400000000000000000000017241046102023000165040ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [ ] create_dir - [x] delete - [ ] copy - [ ] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `token`: Github access token - `owner`: Github owner - `repo`: Github repository You can refer to [`GithubBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Github; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Github::default() // set the storage root for OpenDAL .root("/") // set the access token for Github API .token("your_access_token") // set the owner for Github .owner("your_owner") // set the repository for Github .repo("your_repo"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/github/error.rs000064400000000000000000000062441046102023000167330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use serde::Deserialize; use crate::raw::*; use crate::*; #[derive(Default, Debug, Deserialize)] #[allow(dead_code)] struct GithubError { error: GithubSubError, } #[derive(Default, Debug, Deserialize)] #[allow(dead_code)] struct GithubSubError { message: String, documentation_url: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status.as_u16() { 401 | 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), 304 | 412 => (ErrorKind::ConditionNotMatch, false), // https://github.com/apache/opendal/issues/4146 // https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/423 // We should retry it when we get 423 error. 423 => (ErrorKind::RateLimited, true), // Service like Upyun could return 499 error with a message like: // Client Disconnect, we should retry it. 499 => (ErrorKind::Unexpected, true), 500 | 502 | 503 | 504 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, _github_content_err) = serde_json::from_reader::<_, GithubError>(bs.clone().reader()) .map(|github_content_err| (format!("{github_content_err:?}"), Some(github_content_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[tokio::test] async fn test_parse_error() { let err_res = vec![( r#"{ "message": "Not Found", "documentation_url": "https://docs.github.com/rest/repos/contents#get-repository-content" }"#, ErrorKind::NotFound, StatusCode::NOT_FOUND, )]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/github/lister.rs000064400000000000000000000072401046102023000171010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use super::core::GithubCore; use crate::raw::oio::Entry; use crate::raw::*; use crate::*; pub struct GithubLister { core: Arc, path: String, recursive: bool, } impl GithubLister { pub fn new(core: Arc, path: &str, recursive: bool) -> Self { Self { core, path: path.to_string(), recursive, } } } impl oio::PageList for GithubLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self.core.list(&self.path).await?; // Record whether there is a dir in the list so that we need recursive later. let has_dir = resp.entries.iter().any(|e| e.type_field == "dir"); ctx.done = true; if !self.recursive || !has_dir { for entry in resp.entries { let path = build_rel_path(&self.core.root, &entry.path); let entry = if entry.type_field == "dir" { let path = format!("{}/", path); Entry::new(&path, Metadata::new(EntryMode::DIR)) } else { if path.ends_with(".gitkeep") { continue; } let m = Metadata::new(EntryMode::FILE) .with_content_length(entry.size) .with_etag(entry.sha); Entry::new(&path, m) }; ctx.entries.push_back(entry); } if !self.path.ends_with('/') { ctx.entries.push_back(Entry::new( &format!("{}/", self.path), Metadata::new(EntryMode::DIR), )); } return Ok(()); } // if recursive is true and there is a dir in the list, we need to list it recursively. let tree = self.core.list_with_recursive(&resp.git_url).await?; for t in tree { let path = if self.path == "/" { t.path } else { format!("{}/{}", self.path, t.path) }; let entry = if t.type_field == "tree" { let path = format!("{}/", path); Entry::new(&path, Metadata::new(EntryMode::DIR)) } else { if path.ends_with(".gitkeep") { continue; } let mut m = Metadata::new(EntryMode::FILE).with_etag(t.sha); if let Some(size) = t.size { m = m.with_content_length(size); } Entry::new(&path, m) }; ctx.entries.push_back(entry); } if !self.path.ends_with('/') { ctx.entries.push_back(Entry::new( &format!("{}/", self.path), Metadata::new(EntryMode::DIR), )); } Ok(()) } } opendal-0.52.0/src/services/github/mod.rs000064400000000000000000000022721046102023000163560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-github")] mod core; #[cfg(feature = "services-github")] mod delete; #[cfg(feature = "services-github")] mod error; #[cfg(feature = "services-github")] mod lister; #[cfg(feature = "services-github")] mod writer; #[cfg(feature = "services-github")] mod backend; #[cfg(feature = "services-github")] pub use backend::GithubBuilder as Github; mod config; pub use config::GithubConfig; opendal-0.52.0/src/services/github/writer.rs000064400000000000000000000030351046102023000171110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::GithubCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type GithubWriters = oio::OneShotWriter; pub struct GithubWriter { core: Arc, path: String, } impl GithubWriter { pub fn new(core: Arc, path: String) -> Self { GithubWriter { core, path } } } impl oio::OneShotWrite for GithubWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self.core.upload(&self.path, bs).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/gridfs/backend.rs000064400000000000000000000220571046102023000171650ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use futures::AsyncReadExt; use futures::AsyncWriteExt; use mongodb::bson::doc; use mongodb::gridfs::GridFsBucket; use mongodb::options::ClientOptions; use mongodb::options::GridFsBucketOptions; use tokio::sync::OnceCell; use super::config::GridfsConfig; use crate::raw::adapters::kv; use crate::raw::*; use crate::*; impl Configurator for GridfsConfig { type Builder = GridfsBuilder; fn into_builder(self) -> Self::Builder { GridfsBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct GridfsBuilder { config: GridfsConfig, } impl Debug for GridfsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("GridFsBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl GridfsBuilder { /// Set the connection_string of the MongoDB service. /// /// This connection string is used to connect to the MongoDB service. It typically follows the format: /// /// ## Format /// /// `mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]]` /// /// Examples: /// /// - Connecting to a local MongoDB instance: `mongodb://localhost:27017` /// - Using authentication: `mongodb://myUser:myPassword@localhost:27017/myAuthDB` /// - Specifying authentication mechanism: `mongodb://myUser:myPassword@localhost:27017/myAuthDB?authMechanism=SCRAM-SHA-256` /// /// ## Options /// /// - `authMechanism`: Specifies the authentication method to use. Examples include `SCRAM-SHA-1`, `SCRAM-SHA-256`, and `MONGODB-AWS`. /// - ... (any other options you wish to highlight) /// /// For more information, please refer to [MongoDB Connection String URI Format](https://docs.mongodb.com/manual/reference/connection-string/). pub fn connection_string(mut self, v: &str) -> Self { if !v.is_empty() { self.config.connection_string = Some(v.to_string()); } self } /// Set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the database name of the MongoDB GridFs service to read/write. pub fn database(mut self, database: &str) -> Self { if !database.is_empty() { self.config.database = Some(database.to_string()); } self } /// Set the bucket name of the MongoDB GridFs service to read/write. /// /// Default to `fs` if not specified. pub fn bucket(mut self, bucket: &str) -> Self { if !bucket.is_empty() { self.config.bucket = Some(bucket.to_string()); } self } /// Set the chunk size of the MongoDB GridFs service used to break the user file into chunks. /// /// Default to `255 KiB` if not specified. pub fn chunk_size(mut self, chunk_size: u32) -> Self { if chunk_size > 0 { self.config.chunk_size = Some(chunk_size); } self } } impl Builder for GridfsBuilder { const SCHEME: Scheme = Scheme::Gridfs; type Config = GridfsConfig; fn build(self) -> Result { let conn = match &self.config.connection_string.clone() { Some(v) => v.clone(), None => { return Err( Error::new(ErrorKind::ConfigInvalid, "connection_string is required") .with_context("service", Scheme::Gridfs), ) } }; let database = match &self.config.database.clone() { Some(v) => v.clone(), None => { return Err(Error::new(ErrorKind::ConfigInvalid, "database is required") .with_context("service", Scheme::Gridfs)) } }; let bucket = match &self.config.bucket.clone() { Some(v) => v.clone(), None => "fs".to_string(), }; let chunk_size = self.config.chunk_size.unwrap_or(255); let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); Ok(GridFsBackend::new(Adapter { connection_string: conn, database, bucket, chunk_size, bucket_instance: OnceCell::new(), }) .with_normalized_root(root)) } } pub type GridFsBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { connection_string: String, database: String, bucket: String, chunk_size: u32, bucket_instance: OnceCell, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Adapter") .field("database", &self.database) .field("bucket", &self.bucket) .field("chunk_size", &self.chunk_size) .finish() } } impl Adapter { async fn get_bucket(&self) -> Result<&GridFsBucket> { self.bucket_instance .get_or_try_init(|| async { let client_options = ClientOptions::parse(&self.connection_string) .await .map_err(parse_mongodb_error)?; let client = mongodb::Client::with_options(client_options).map_err(parse_mongodb_error)?; let bucket_options = GridFsBucketOptions::builder() .bucket_name(Some(self.bucket.clone())) .chunk_size_bytes(Some(self.chunk_size)) .build(); let bucket = client .database(&self.database) .gridfs_bucket(bucket_options); Ok(bucket) }) .await } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Gridfs, &format!("{}/{}", self.database, self.bucket), Capability { read: true, write: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let bucket = self.get_bucket().await?; let filter = doc! { "filename": path }; let Some(doc) = bucket.find_one(filter).await.map_err(parse_mongodb_error)? else { return Ok(None); }; let mut destination = Vec::new(); let file_id = doc.id; let mut stream = bucket .open_download_stream(file_id) .await .map_err(parse_mongodb_error)?; stream .read_to_end(&mut destination) .await .map_err(new_std_io_error)?; Ok(Some(Buffer::from(destination))) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let bucket = self.get_bucket().await?; // delete old file if exists let filter = doc! { "filename": path }; if let Some(doc) = bucket.find_one(filter).await.map_err(parse_mongodb_error)? { let file_id = doc.id; bucket.delete(file_id).await.map_err(parse_mongodb_error)?; }; // set new file let mut upload_stream = bucket .open_upload_stream(path) .await .map_err(parse_mongodb_error)?; upload_stream .write_all(&value.to_vec()) .await .map_err(new_std_io_error)?; upload_stream.close().await.map_err(new_std_io_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let bucket = self.get_bucket().await?; let filter = doc! { "filename": path }; let Some(doc) = bucket.find_one(filter).await.map_err(parse_mongodb_error)? else { return Ok(()); }; let file_id = doc.id; bucket.delete(file_id).await.map_err(parse_mongodb_error)?; Ok(()) } } fn parse_mongodb_error(err: mongodb::error::Error) -> Error { Error::new(ErrorKind::Unexpected, "mongodb error").set_source(err) } opendal-0.52.0/src/services/gridfs/config.rs000064400000000000000000000036271046102023000170450ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Grid file system support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct GridfsConfig { /// The connection string of the MongoDB service. pub connection_string: Option, /// The database name of the MongoDB GridFs service to read/write. pub database: Option, /// The bucket name of the MongoDB GridFs service to read/write. pub bucket: Option, /// The chunk size of the MongoDB GridFs service used to break the user file into chunks. pub chunk_size: Option, /// The working directory, all operations will be performed under it. pub root: Option, } impl Debug for GridfsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("GridFsConfig") .field("database", &self.database) .field("bucket", &self.bucket) .field("chunk_size", &self.chunk_size) .field("root", &self.root) .finish() } } opendal-0.52.0/src/services/gridfs/docs.md000064400000000000000000000017501046102023000164770ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `connection_string`: Set the connection string of mongodb server - `database`: Set the database of mongodb - `bucket`: Set the bucket of mongodb gridfs - `chunk_size`: Set the chunk size of mongodb gridfs ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Gridfs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Gridfs::default() .root("/") .connection_string("mongodb://myUser:myPassword@localhost:27017/myAuthDB") .database("your_database") .bucket("your_bucket") // The chunk size in bytes used to break the user file into chunks. .chunk_size(255); let op = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/gridfs/mod.rs000064400000000000000000000017141046102023000163520ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-gridfs")] mod backend; #[cfg(feature = "services-gridfs")] pub use backend::GridfsBuilder as Gridfs; mod config; pub use config::GridfsConfig; opendal-0.52.0/src/services/hdfs/backend.rs000064400000000000000000000451211046102023000166300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::io; use std::io::SeekFrom; use std::path::PathBuf; use std::sync::Arc; use log::debug; use uuid::Uuid; use super::delete::HdfsDeleter; use super::lister::HdfsLister; use super::reader::HdfsReader; use super::writer::HdfsWriter; use crate::raw::*; use crate::services::HdfsConfig; use crate::*; impl Configurator for HdfsConfig { type Builder = HdfsBuilder; fn into_builder(self) -> Self::Builder { HdfsBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct HdfsBuilder { config: HdfsConfig, } impl Debug for HdfsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("HdfsBuilder") .field("config", &self.config) .finish() } } impl HdfsBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set name_node of this backend. /// /// Valid format including: /// /// - `default`: using the default setting based on hadoop config. /// - `hdfs://127.0.0.1:9000`: connect to hdfs cluster. pub fn name_node(mut self, name_node: &str) -> Self { if !name_node.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.name_node = Some(name_node.trim_end_matches('/').to_string()) } self } /// Set kerberos_ticket_cache_path of this backend /// /// This should be configured when kerberos is enabled. pub fn kerberos_ticket_cache_path(mut self, kerberos_ticket_cache_path: &str) -> Self { if !kerberos_ticket_cache_path.is_empty() { self.config.kerberos_ticket_cache_path = Some(kerberos_ticket_cache_path.to_string()) } self } /// Set user of this backend pub fn user(mut self, user: &str) -> Self { if !user.is_empty() { self.config.user = Some(user.to_string()) } self } /// Enable append capacity of this backend. /// /// This should be disabled when HDFS runs in non-distributed mode. pub fn enable_append(mut self, enable_append: bool) -> Self { self.config.enable_append = enable_append; self } /// Set temp dir for atomic write. /// /// # Notes /// /// - When append is enabled, we will not use atomic write /// to avoid data loss and performance issue. pub fn atomic_write_dir(mut self, dir: &str) -> Self { self.config.atomic_write_dir = if dir.is_empty() { None } else { Some(String::from(dir)) }; self } } impl Builder for HdfsBuilder { const SCHEME: Scheme = Scheme::Hdfs; type Config = HdfsConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let name_node = match &self.config.name_node { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "name node is empty") .with_context("service", Scheme::Hdfs)) } }; let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let mut builder = hdrs::ClientBuilder::new(name_node); if let Some(ticket_cache_path) = &self.config.kerberos_ticket_cache_path { builder = builder.with_kerberos_ticket_cache_path(ticket_cache_path.as_str()); } if let Some(user) = &self.config.user { builder = builder.with_user(user.as_str()); } let client = builder.connect().map_err(new_std_io_error)?; // Create root dir if not exist. if let Err(e) = client.metadata(&root) { if e.kind() == io::ErrorKind::NotFound { debug!("root {} is not exist, creating now", root); client.create_dir(&root).map_err(new_std_io_error)? } } let atomic_write_dir = self.config.atomic_write_dir; // If atomic write dir is not exist, we must create it. if let Some(d) = &atomic_write_dir { if let Err(e) = client.metadata(d) { if e.kind() == io::ErrorKind::NotFound { client.create_dir(d).map_err(new_std_io_error)? } } } Ok(HdfsBackend { root, atomic_write_dir, client: Arc::new(client), enable_append: self.config.enable_append, }) } } #[inline] fn tmp_file_of(path: &str) -> String { let name = get_basename(path); let uuid = Uuid::new_v4().to_string(); format!("{name}.{uuid}") } /// Backend for hdfs services. #[derive(Debug, Clone)] pub struct HdfsBackend { pub root: String, atomic_write_dir: Option, pub client: Arc, enable_append: bool, } /// hdrs::Client is thread-safe. unsafe impl Send for HdfsBackend {} unsafe impl Sync for HdfsBackend {} impl Access for HdfsBackend { type Reader = HdfsReader; type Writer = HdfsWriter; type Lister = Option; type Deleter = oio::OneShotDeleter; type BlockingReader = HdfsReader; type BlockingWriter = HdfsWriter; type BlockingLister = Option; type BlockingDeleter = oio::OneShotDeleter; fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Hdfs) .set_root(&self.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, write: true, write_can_append: self.enable_append, create_dir: true, delete: true, list: true, list_has_content_length: true, list_has_last_modified: true, rename: true, blocking: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let p = build_rooted_abs_path(&self.root, path); self.client.create_dir(&p).map_err(new_std_io_error)?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _: OpStat) -> Result { let p = build_rooted_abs_path(&self.root, path); let meta = self.client.metadata(&p).map_err(new_std_io_error)?; let mode = if meta.is_dir() { EntryMode::DIR } else if meta.is_file() { EntryMode::FILE } else { EntryMode::Unknown }; let mut m = Metadata::new(mode); m.set_content_length(meta.len()); m.set_last_modified(meta.modified().into()); Ok(RpStat::new(m)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let p = build_rooted_abs_path(&self.root, path); let client = self.client.clone(); let mut f = client .open_file() .read(true) .async_open(&p) .await .map_err(new_std_io_error)?; if args.range().offset() != 0 { use futures::AsyncSeekExt; f.seek(SeekFrom::Start(args.range().offset())) .await .map_err(new_std_io_error)?; } Ok(( RpRead::new(), HdfsReader::new(f, args.range().size().unwrap_or(u64::MAX) as _), )) } async fn write(&self, path: &str, op: OpWrite) -> Result<(RpWrite, Self::Writer)> { let target_path = build_rooted_abs_path(&self.root, path); let mut initial_size = 0; let target_exists = match self.client.metadata(&target_path) { Ok(meta) => { initial_size = meta.len(); true } Err(err) => { if err.kind() != io::ErrorKind::NotFound { return Err(new_std_io_error(err)); } false } }; let should_append = op.append() && target_exists; let tmp_path = self.atomic_write_dir.as_ref().and_then(|atomic_write_dir| { // If the target file exists, we should append to the end of it directly. if should_append { None } else { Some(build_rooted_abs_path(atomic_write_dir, &tmp_file_of(path))) } }); if !target_exists { let parent = get_parent(&target_path); self.client.create_dir(parent).map_err(new_std_io_error)?; } if !should_append { initial_size = 0; } let mut open_options = self.client.open_file(); open_options.create(true); if should_append { open_options.append(true); } else { open_options.write(true); } let f = open_options .async_open(tmp_path.as_ref().unwrap_or(&target_path)) .await .map_err(new_std_io_error)?; Ok(( RpWrite::new(), HdfsWriter::new( target_path, tmp_path, f, Arc::clone(&self.client), target_exists, initial_size, ), )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(HdfsDeleter::new(Arc::new(self.clone()))), )) } async fn list(&self, path: &str, _: OpList) -> Result<(RpList, Self::Lister)> { let p = build_rooted_abs_path(&self.root, path); let f = match self.client.read_dir(&p) { Ok(f) => f, Err(e) => { return if e.kind() == io::ErrorKind::NotFound { Ok((RpList::default(), None)) } else { Err(new_std_io_error(e)) } } }; let rd = HdfsLister::new(&self.root, f, path); Ok((RpList::default(), Some(rd))) } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let from_path = build_rooted_abs_path(&self.root, from); self.client.metadata(&from_path).map_err(new_std_io_error)?; let to_path = build_rooted_abs_path(&self.root, to); let result = self.client.metadata(&to_path); match result { Err(err) => { // Early return if other error happened. if err.kind() != io::ErrorKind::NotFound { return Err(new_std_io_error(err)); } let parent = PathBuf::from(&to_path) .parent() .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "path should have parent but not, it must be malformed", ) .with_context("input", &to_path) })? .to_path_buf(); self.client .create_dir(&parent.to_string_lossy()) .map_err(new_std_io_error)?; } Ok(metadata) => { if metadata.is_file() { self.client .remove_file(&to_path) .map_err(new_std_io_error)?; } else { return Err(Error::new(ErrorKind::IsADirectory, "path should be a file") .with_context("input", &to_path)); } } } self.client .rename_file(&from_path, &to_path) .map_err(new_std_io_error)?; Ok(RpRename::new()) } fn blocking_create_dir(&self, path: &str, _: OpCreateDir) -> Result { let p = build_rooted_abs_path(&self.root, path); self.client.create_dir(&p).map_err(new_std_io_error)?; Ok(RpCreateDir::default()) } fn blocking_stat(&self, path: &str, _: OpStat) -> Result { let p = build_rooted_abs_path(&self.root, path); let meta = self.client.metadata(&p).map_err(new_std_io_error)?; let mode = if meta.is_dir() { EntryMode::DIR } else if meta.is_file() { EntryMode::FILE } else { EntryMode::Unknown }; let mut m = Metadata::new(mode); m.set_content_length(meta.len()); m.set_last_modified(meta.modified().into()); Ok(RpStat::new(m)) } fn blocking_read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::BlockingReader)> { let p = build_rooted_abs_path(&self.root, path); let mut f = self .client .open_file() .read(true) .open(&p) .map_err(new_std_io_error)?; if args.range().offset() != 0 { use std::io::Seek; f.seek(SeekFrom::Start(args.range().offset())) .map_err(new_std_io_error)?; } Ok(( RpRead::new(), HdfsReader::new(f, args.range().size().unwrap_or(u64::MAX) as _), )) } fn blocking_write(&self, path: &str, op: OpWrite) -> Result<(RpWrite, Self::BlockingWriter)> { let target_path = build_rooted_abs_path(&self.root, path); let mut initial_size = 0; let target_exists = match self.client.metadata(&target_path) { Ok(meta) => { initial_size = meta.len(); true } Err(err) => { if err.kind() != io::ErrorKind::NotFound { return Err(new_std_io_error(err)); } false } }; let should_append = op.append() && target_exists; let tmp_path = self.atomic_write_dir.as_ref().and_then(|atomic_write_dir| { // If the target file exists, we should append to the end of it directly. if should_append { None } else { Some(build_rooted_abs_path(atomic_write_dir, &tmp_file_of(path))) } }); if !target_exists { let parent = get_parent(&target_path); self.client.create_dir(parent).map_err(new_std_io_error)?; } if !should_append { initial_size = 0; } let mut open_options = self.client.open_file(); open_options.create(true); if should_append { open_options.append(true); } else { open_options.write(true); } let f = open_options .open(tmp_path.as_ref().unwrap_or(&target_path)) .map_err(new_std_io_error)?; Ok(( RpWrite::new(), HdfsWriter::new( target_path, tmp_path, f, Arc::clone(&self.client), target_exists, initial_size, ), )) } fn blocking_delete(&self) -> Result<(RpDelete, Self::BlockingDeleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(HdfsDeleter::new(Arc::new(self.clone()))), )) } fn blocking_list(&self, path: &str, _: OpList) -> Result<(RpList, Self::BlockingLister)> { let p = build_rooted_abs_path(&self.root, path); let f = match self.client.read_dir(&p) { Ok(f) => f, Err(e) => { return if e.kind() == io::ErrorKind::NotFound { Ok((RpList::default(), None)) } else { Err(new_std_io_error(e)) } } }; let rd = HdfsLister::new(&self.root, f, path); Ok((RpList::default(), Some(rd))) } fn blocking_rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let from_path = build_rooted_abs_path(&self.root, from); self.client.metadata(&from_path).map_err(new_std_io_error)?; let to_path = build_rooted_abs_path(&self.root, to); let result = self.client.metadata(&to_path); match result { Err(err) => { // Early return if other error happened. if err.kind() != io::ErrorKind::NotFound { return Err(new_std_io_error(err)); } let parent = PathBuf::from(&to_path) .parent() .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "path should have parent but not, it must be malformed", ) .with_context("input", &to_path) })? .to_path_buf(); self.client .create_dir(&parent.to_string_lossy()) .map_err(new_std_io_error)?; } Ok(metadata) => { if metadata.is_file() { self.client .remove_file(&to_path) .map_err(new_std_io_error)?; } else { return Err(Error::new(ErrorKind::IsADirectory, "path should be a file") .with_context("input", &to_path)); } } } self.client .rename_file(&from_path, &to_path) .map_err(new_std_io_error)?; Ok(RpRename::new()) } } opendal-0.52.0/src/services/hdfs/config.rs000064400000000000000000000041351046102023000165060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// [Hadoop Distributed File System (HDFS™)](https://hadoop.apache.org/) support. /// /// Config for Hdfs services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct HdfsConfig { /// work dir of this backend pub root: Option, /// name node of this backend pub name_node: Option, /// kerberos_ticket_cache_path of this backend pub kerberos_ticket_cache_path: Option, /// user of this backend pub user: Option, /// enable the append capacity pub enable_append: bool, /// atomic_write_dir of this backend pub atomic_write_dir: Option, } impl Debug for HdfsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("HdfsConfig") .field("root", &self.root) .field("name_node", &self.name_node) .field( "kerberos_ticket_cache_path", &self.kerberos_ticket_cache_path, ) .field("user", &self.user) .field("enable_append", &self.enable_append) .field("atomic_write_dir", &self.atomic_write_dir) .finish_non_exhaustive() } } opendal-0.52.0/src/services/hdfs/delete.rs000064400000000000000000000050651046102023000165060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::backend::HdfsBackend; use crate::raw::*; use crate::*; use std::io; use std::sync::Arc; pub struct HdfsDeleter { core: Arc, } impl HdfsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for HdfsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = build_rooted_abs_path(&self.core.root, &path); let meta = self.core.client.metadata(&p); if let Err(err) = meta { return if err.kind() == io::ErrorKind::NotFound { Ok(()) } else { Err(new_std_io_error(err)) }; } // Safety: Err branch has been checked, it's OK to unwrap. let meta = meta.ok().unwrap(); let result = if meta.is_dir() { self.core.client.remove_dir(&p) } else { self.core.client.remove_file(&p) }; result.map_err(new_std_io_error)?; Ok(()) } } impl oio::BlockingOneShotDelete for HdfsDeleter { fn blocking_delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = build_rooted_abs_path(&self.core.root, &path); let meta = self.core.client.metadata(&p); if let Err(err) = meta { return if err.kind() == io::ErrorKind::NotFound { Ok(()) } else { Err(new_std_io_error(err)) }; } // Safety: Err branch has been checked, it's OK to unwrap. let meta = meta.ok().unwrap(); let result = if meta.is_dir() { self.core.client.remove_dir(&p) } else { self.core.client.remove_file(&p) }; result.map_err(new_std_io_error)?; Ok(()) } } opendal-0.52.0/src/services/hdfs/docs.md000064400000000000000000000105221046102023000161420ustar 00000000000000A distributed file system that provides high-throughput access to application data. ## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [x] rename - [x] list - [ ] ~~presign~~ - [x] blocking - [x] append ## Differences with webhdfs [Webhdfs][crate::services::Webhdfs] is powered by hdfs's RESTful HTTP API. ## Features HDFS support needs to enable feature `services-hdfs`. ## Configuration - `root`: Set the work dir for backend. - `name_node`: Set the name node for backend. - `kerberos_ticket_cache_path`: Set the kerberos ticket cache path for backend, this should be gotten by `klist` after `kinit` - `user`: Set the user for backend - `enable_append`: enable the append capacity. Default is false. Refer to [`HdfsBuilder`]'s public API docs for more information. ## Environment HDFS needs some environment set correctly. - `JAVA_HOME`: the path to java home, could be found via `java -XshowSettings:properties -version` - `HADOOP_HOME`: the path to hadoop home, opendal relays on this env to discover hadoop jars and set `CLASSPATH` automatically. Most of the time, setting `JAVA_HOME` and `HADOOP_HOME` is enough. But there are some edge cases: - If meeting errors like the following: ```shell error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory ``` Java's lib are not including in pkg-config find path, please set `LD_LIBRARY_PATH`: ```shell export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH} ``` The path of `libjvm.so` could be different, please keep an eye on it. - If meeting errors like the following: ```shell (unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.) ``` `CLASSPATH` is not set correctly or your hadoop installation is incorrect. To set `CLASSPATH`: ```shell export CLASSPATH=$(find $HADOOP_HOME -iname "*.jar" | xargs echo | tr ' ' ':'):${CLASSPATH} ``` - If HDFS has High Availability (HA) enabled with multiple available NameNodes, some configuration is required: 1. Obtain the entire HDFS config folder (usually located at HADOOP_HOME/etc/hadoop). 2. Set the environment variable HADOOP_CONF_DIR to the path of this folder. ```shell export HADOOP_CONF_DIR= ``` 3. Append the HADOOP_CONF_DIR to the `CLASSPATH` ```shell export CLASSPATH=$HADOOP_CONF_DIR:$HADOOP_CLASSPATH:$CLASSPATH ``` 4. Use the `cluster_name` specified in the `core-site.xml` file (located in the HADOOP_CONF_DIR folder) to replace namenode:port. ```ignore builder.name_node("hdfs://cluster_name"); ``` ### macOS Specific Note If you encounter an issue during the build process on macOS with an error message similar to: ```shell ld: unknown file type in $HADOOP_HOME/lib/native/libhdfs.so.0.0.0 clang: error: linker command failed with exit code 1 (use -v to see invocation) ``` This error is likely due to the fact that the official Hadoop build includes the libhdfs.so file for the x86-64 architecture, which is not compatible with aarch64 architecture required for MacOS. To resolve this issue, you can add hdrs as a dependency in your Rust application's Cargo.toml file, and enable the vendored feature: ```toml [dependencies] hdrs = { version = "", features = ["vendored"] } ``` Enabling the vendored feature ensures that hdrs includes the necessary libhdfs.so library built for the correct architecture. ## Example ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Hdfs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create fs backend builder. let mut builder = Hdfs::default() // Set the name node for hdfs. // If the string starts with a protocol type such as file://, hdfs://, or gs://, this protocol type will be used. .name_node("hdfs://127.0.0.1:9000") // Set the root for hdfs, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/tmp") // Enable the append capacity for hdfs. // // Note: HDFS run in non-distributed mode doesn't support append. .enable_append(true); // `Accessor` provides the low level APIs, we will use `Operator` normally. let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/hdfs/lister.rs000064400000000000000000000061011046102023000165360ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::EntryMode; use crate::Metadata; use crate::Result; pub struct HdfsLister { root: String, rd: hdrs::Readdir, current_path: Option, } impl HdfsLister { pub fn new(root: &str, rd: hdrs::Readdir, path: &str) -> Self { Self { root: root.to_string(), rd, current_path: Some(path.to_string()), } } } impl oio::List for HdfsLister { async fn next(&mut self) -> Result> { if let Some(path) = self.current_path.take() { return Ok(Some(oio::Entry::new(&path, Metadata::new(EntryMode::DIR)))); } let de = match self.rd.next() { Some(de) => de, None => return Ok(None), }; let path = build_rel_path(&self.root, de.path()); let entry = if de.is_file() { let meta = Metadata::new(EntryMode::FILE) .with_content_length(de.len()) .with_last_modified(de.modified().into()); oio::Entry::new(&path, meta) } else if de.is_dir() { // Make sure we are returning the correct path. oio::Entry::new(&format!("{path}/"), Metadata::new(EntryMode::DIR)) } else { oio::Entry::new(&path, Metadata::new(EntryMode::Unknown)) }; Ok(Some(entry)) } } impl oio::BlockingList for HdfsLister { fn next(&mut self) -> Result> { if let Some(path) = self.current_path.take() { return Ok(Some(oio::Entry::new(&path, Metadata::new(EntryMode::DIR)))); } let de = match self.rd.next() { Some(de) => de, None => return Ok(None), }; let path = build_rel_path(&self.root, de.path()); let entry = if de.is_file() { let meta = Metadata::new(EntryMode::FILE) .with_content_length(de.len()) .with_last_modified(de.modified().into()); oio::Entry::new(&path, meta) } else if de.is_dir() { // Make sure we are returning the correct path. oio::Entry::new(&format!("{path}/"), Metadata::new(EntryMode::DIR)) } else { oio::Entry::new(&path, Metadata::new(EntryMode::Unknown)) }; Ok(Some(entry)) } } opendal-0.52.0/src/services/hdfs/mod.rs000064400000000000000000000021731046102023000160200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-hdfs")] mod delete; #[cfg(feature = "services-hdfs")] mod lister; #[cfg(feature = "services-hdfs")] mod reader; #[cfg(feature = "services-hdfs")] mod writer; #[cfg(feature = "services-hdfs")] mod backend; #[cfg(feature = "services-hdfs")] pub use backend::HdfsBuilder as Hdfs; mod config; pub use config::HdfsConfig; opendal-0.52.0/src/services/hdfs/reader.rs000064400000000000000000000063071046102023000165060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io::Read; use bytes::BytesMut; use futures::AsyncReadExt; use hdrs::AsyncFile; use hdrs::File; use tokio::io::ReadBuf; use crate::raw::*; use crate::*; pub struct HdfsReader { f: F, read: usize, size: usize, buf_size: usize, buf: BytesMut, } impl HdfsReader { pub fn new(f: F, size: usize) -> Self { Self { f, read: 0, size, // Use 2 MiB as default value. buf_size: 2 * 1024 * 1024, buf: BytesMut::new(), } } } impl oio::Read for HdfsReader { async fn read(&mut self) -> Result { if self.read >= self.size { return Ok(Buffer::new()); } let size = (self.size - self.read).min(self.buf_size); self.buf.reserve(size); let buf = &mut self.buf.spare_capacity_mut()[..size]; let mut read_buf: ReadBuf = ReadBuf::uninit(buf); // SAFETY: Read at most `limit` bytes into `read_buf`. unsafe { read_buf.assume_init(size); } let n = self .f .read(read_buf.initialize_unfilled()) .await .map_err(new_std_io_error)?; read_buf.advance(n); self.read += n; // Safety: We make sure that bs contains `n` more bytes. let filled = read_buf.filled().len(); unsafe { self.buf.set_len(filled) } let frozen = self.buf.split().freeze(); Ok(Buffer::from(frozen)) } } impl oio::BlockingRead for HdfsReader { fn read(&mut self) -> Result { if self.read >= self.size { return Ok(Buffer::new()); } let size = (self.size - self.read).min(self.buf_size); self.buf.reserve(size); let buf = &mut self.buf.spare_capacity_mut()[..size]; let mut read_buf: ReadBuf = ReadBuf::uninit(buf); // SAFETY: Read at most `limit` bytes into `read_buf`. unsafe { read_buf.assume_init(size); } let n = self .f .read(read_buf.initialize_unfilled()) .map_err(new_std_io_error)?; read_buf.advance(n); self.read += n; // Safety: We make sure that bs contains `n` more bytes. let filled = read_buf.filled().len(); unsafe { self.buf.set_len(filled) } let frozen = self.buf.split().freeze(); Ok(Buffer::from(frozen)) } } opendal-0.52.0/src/services/hdfs/writer.rs000064400000000000000000000100021046102023000165430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io::Write; use std::sync::Arc; use bytes::Buf; use futures::AsyncWriteExt; use crate::raw::*; use crate::*; pub struct HdfsWriter { target_path: String, tmp_path: Option, f: Option, client: Arc, target_path_exists: bool, size: u64, } /// # Safety /// /// We will only take `&mut Self` reference for HdfsWriter. unsafe impl Sync for HdfsWriter {} impl HdfsWriter { pub fn new( target_path: String, tmp_path: Option, f: F, client: Arc, target_path_exists: bool, initial_size: u64, ) -> Self { Self { target_path, tmp_path, f: Some(f), client, target_path_exists, size: initial_size, } } } impl oio::Write for HdfsWriter { async fn write(&mut self, mut bs: Buffer) -> Result<()> { let len = bs.len() as u64; let f = self.f.as_mut().expect("HdfsWriter must be initialized"); while bs.has_remaining() { let n = f.write(bs.chunk()).await.map_err(new_std_io_error)?; bs.advance(n); } self.size += len; Ok(()) } async fn close(&mut self) -> Result { let f = self.f.as_mut().expect("HdfsWriter must be initialized"); f.close().await.map_err(new_std_io_error)?; // TODO: we need to make rename async. if let Some(tmp_path) = &self.tmp_path { // we must delete the target_path, otherwise the rename_file operation will fail if self.target_path_exists { self.client .remove_file(&self.target_path) .map_err(new_std_io_error)?; } self.client .rename_file(tmp_path, &self.target_path) .map_err(new_std_io_error)? } Ok(Metadata::default().with_content_length(self.size)) } async fn abort(&mut self) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "HdfsWriter doesn't support abort", )) } } impl oio::BlockingWrite for HdfsWriter { fn write(&mut self, mut bs: Buffer) -> Result<()> { let len = bs.len() as u64; let f = self.f.as_mut().expect("HdfsWriter must be initialized"); while bs.has_remaining() { let n = f.write(bs.chunk()).map_err(new_std_io_error)?; bs.advance(n); } self.size += len; Ok(()) } fn close(&mut self) -> Result { let f = self.f.as_mut().expect("HdfsWriter must be initialized"); f.flush().map_err(new_std_io_error)?; if let Some(tmp_path) = &self.tmp_path { // we must delete the target_path, otherwise the rename_file operation will fail if self.target_path_exists { self.client .remove_file(&self.target_path) .map_err(new_std_io_error)?; } self.client .rename_file(tmp_path, &self.target_path) .map_err(new_std_io_error)?; } Ok(Metadata::default().with_content_length(self.size)) } } opendal-0.52.0/src/services/hdfs_native/backend.rs000064400000000000000000000170271046102023000202020ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use hdfs_native::WriteOptions; use log::debug; use super::delete::HdfsNativeDeleter; use super::error::parse_hdfs_error; use super::lister::HdfsNativeLister; use super::reader::HdfsNativeReader; use super::writer::HdfsNativeWriter; use crate::raw::*; use crate::services::HdfsNativeConfig; use crate::*; /// [Hadoop Distributed File System (HDFS™)](https://hadoop.apache.org/) support. /// Using [Native Rust HDFS client](https://github.com/Kimahriman/hdfs-native). impl Configurator for HdfsNativeConfig { type Builder = HdfsNativeBuilder; fn into_builder(self) -> Self::Builder { HdfsNativeBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct HdfsNativeBuilder { config: HdfsNativeConfig, } impl Debug for HdfsNativeBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("HdfsNativeBuilder") .field("config", &self.config) .finish() } } impl HdfsNativeBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set url of this backend. /// /// Valid format including: /// /// - `default`: using the default setting based on hadoop config. /// - `hdfs://127.0.0.1:9000`: connect to hdfs cluster. pub fn url(mut self, url: &str) -> Self { if !url.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.url = Some(url.trim_end_matches('/').to_string()) } self } /// Enable append capacity of this backend. /// /// This should be disabled when HDFS runs in non-distributed mode. pub fn enable_append(mut self, enable_append: bool) -> Self { self.config.enable_append = enable_append; self } } impl Builder for HdfsNativeBuilder { const SCHEME: Scheme = Scheme::HdfsNative; type Config = HdfsNativeConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let url = match &self.config.url { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "url is empty") .with_context("service", Scheme::HdfsNative)); } }; let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let client = hdfs_native::Client::new(url).map_err(parse_hdfs_error)?; // need to check if root dir exists, create if not Ok(HdfsNativeBackend { root, client: Arc::new(client), _enable_append: self.config.enable_append, }) } } // #[inline] // fn tmp_file_of(path: &str) -> String { // let name = get_basename(path); // let uuid = Uuid::new_v4().to_string(); // // format!("{name}.{uuid}") // } /// Backend for hdfs-native services. #[derive(Debug, Clone)] pub struct HdfsNativeBackend { pub root: String, pub client: Arc, _enable_append: bool, } /// hdfs_native::Client is thread-safe. unsafe impl Send for HdfsNativeBackend {} unsafe impl Sync for HdfsNativeBackend {} impl Access for HdfsNativeBackend { type Reader = HdfsNativeReader; type Writer = HdfsNativeWriter; type Lister = Option; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::HdfsNative) .set_root(&self.root) .set_native_capability(Capability { stat: true, stat_has_last_modified: true, stat_has_content_length: true, delete: true, rename: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _args: OpCreateDir) -> Result { let p = build_rooted_abs_path(&self.root, path); self.client .mkdirs(&p, 0o777, true) .await .map_err(parse_hdfs_error)?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _args: OpStat) -> Result { let p = build_rooted_abs_path(&self.root, path); let status: hdfs_native::client::FileStatus = self .client .get_file_info(&p) .await .map_err(parse_hdfs_error)?; let mode = if status.isdir { EntryMode::DIR } else { EntryMode::FILE }; let mut metadata = Metadata::new(mode); metadata .set_last_modified(parse_datetime_from_from_timestamp_millis( status.modification_time as i64, )?) .set_content_length(status.length as u64); Ok(RpStat::new(metadata)) } async fn read(&self, path: &str, _args: OpRead) -> Result<(RpRead, Self::Reader)> { let p = build_rooted_abs_path(&self.root, path); let f = self.client.read(&p).await.map_err(parse_hdfs_error)?; let r = HdfsNativeReader::new(f); Ok((RpRead::new(), r)) } async fn write(&self, path: &str, _args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let p = build_rooted_abs_path(&self.root, path); let f = self .client .create(&p, WriteOptions::default()) .await .map_err(parse_hdfs_error)?; let w = HdfsNativeWriter::new(f); Ok((RpWrite::new(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(HdfsNativeDeleter::new(Arc::new(self.clone()))), )) } async fn list(&self, path: &str, _args: OpList) -> Result<(RpList, Self::Lister)> { let p = build_rooted_abs_path(&self.root, path); let l = HdfsNativeLister::new(p, self.client.clone()); Ok((RpList::default(), Some(l))) } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let from_path = build_rooted_abs_path(&self.root, from); let to_path = build_rooted_abs_path(&self.root, to); self.client .rename(&from_path, &to_path, false) .await .map_err(parse_hdfs_error)?; Ok(RpRename::default()) } } opendal-0.52.0/src/services/hdfs_native/config.rs000064400000000000000000000030501046102023000200470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for HdfsNative services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct HdfsNativeConfig { /// work dir of this backend pub root: Option, /// url of this backend pub url: Option, /// enable the append capacity pub enable_append: bool, } impl Debug for HdfsNativeConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("HdfsNativeConfig") .field("root", &self.root) .field("url", &self.url) .field("enable_append", &self.enable_append) .finish_non_exhaustive() } } opendal-0.52.0/src/services/hdfs_native/delete.rs000064400000000000000000000026511046102023000200520ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::backend::HdfsNativeBackend; use super::error::parse_hdfs_error; use crate::raw::*; use crate::*; use std::sync::Arc; pub struct HdfsNativeDeleter { core: Arc, } impl HdfsNativeDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for HdfsNativeDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let p = build_rooted_abs_path(&self.core.root, &path); self.core .client .delete(&p, true) .await .map_err(parse_hdfs_error)?; Ok(()) } } opendal-0.52.0/src/services/hdfs_native/docs.md000064400000000000000000000014201046102023000175050ustar 00000000000000A distributed file system that provides high-throughput access to application data. Using [Native Rust HDFS client](https://github.com/Kimahriman/hdfs-native). ## Capabilities This service can be used to: - [x] stat - [ ] read - [ ] write - [ ] create_dir - [x] delete - [x] rename - [ ] list - [x] blocking - [ ] append ## Differences with webhdfs [Webhdfs][crate::services::Webhdfs] is powered by hdfs's RESTful HTTP API. ## Differences with hdfs [hdfs][crate::services::Hdfs] is powered by libhdfs and require the Java dependencies ## Features HDFS-native support needs to enable feature `services-hdfs-native`. ## Configuration - `root`: Set the work dir for backend. - `url`: Set the url for backend. - `enable_append`: enable the append capacity. Default is false. opendal-0.52.0/src/services/hdfs_native/error.rs000064400000000000000000000041321046102023000177350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use hdfs_native::HdfsError; use crate::*; /// Parse hdfs-native error into opendal::Error. pub fn parse_hdfs_error(hdfs_error: HdfsError) -> Error { let (kind, retryable, msg) = match &hdfs_error { HdfsError::IOError(err) => (ErrorKind::Unexpected, false, err.to_string()), HdfsError::DataTransferError(msg) => (ErrorKind::Unexpected, false, msg.clone()), HdfsError::ChecksumError => ( ErrorKind::Unexpected, false, "checksums didn't match".to_string(), ), HdfsError::UrlParseError(err) => (ErrorKind::Unexpected, false, err.to_string()), HdfsError::AlreadyExists(msg) => (ErrorKind::AlreadyExists, false, msg.clone()), HdfsError::OperationFailed(msg) => (ErrorKind::Unexpected, false, msg.clone()), HdfsError::FileNotFound(msg) => (ErrorKind::NotFound, false, msg.clone()), HdfsError::BlocksNotFound(msg) => (ErrorKind::NotFound, false, msg.clone()), HdfsError::IsADirectoryError(msg) => (ErrorKind::IsADirectory, false, msg.clone()), _ => ( ErrorKind::Unexpected, false, "unexpected error from hdfs".to_string(), ), }; let mut err = Error::new(kind, msg).set_source(hdfs_error); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/hdfs_native/lister.rs000064400000000000000000000024151046102023000201100ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use crate::raw::oio; use crate::raw::oio::Entry; use crate::*; pub struct HdfsNativeLister { _path: String, _client: Arc, } impl HdfsNativeLister { pub fn new(path: String, client: Arc) -> Self { HdfsNativeLister { _path: path, _client: client, } } } impl oio::List for HdfsNativeLister { async fn next(&mut self) -> Result> { todo!() } } opendal-0.52.0/src/services/hdfs_native/mod.rs000064400000000000000000000023541046102023000173670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-hdfs-native")] mod delete; #[cfg(feature = "services-hdfs-native")] mod error; #[cfg(feature = "services-hdfs-native")] mod lister; #[cfg(feature = "services-hdfs-native")] mod reader; #[cfg(feature = "services-hdfs-native")] mod writer; #[cfg(feature = "services-hdfs-native")] mod backend; #[cfg(feature = "services-hdfs-native")] pub use backend::HdfsNativeBuilder as HdfsNative; mod config; pub use config::HdfsNativeConfig; opendal-0.52.0/src/services/hdfs_native/reader.rs000064400000000000000000000021701046102023000200460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use hdfs_native::file::FileReader; use crate::raw::*; use crate::*; pub struct HdfsNativeReader { _f: FileReader, } impl HdfsNativeReader { pub fn new(f: FileReader) -> Self { HdfsNativeReader { _f: f } } } impl oio::Read for HdfsNativeReader { async fn read(&mut self) -> Result { todo!() } } opendal-0.52.0/src/services/hdfs_native/writer.rs000064400000000000000000000026021046102023000201200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use hdfs_native::file::FileWriter; use crate::raw::oio; use crate::*; pub struct HdfsNativeWriter { _f: FileWriter, } impl HdfsNativeWriter { pub fn new(f: FileWriter) -> Self { HdfsNativeWriter { _f: f } } } impl oio::Write for HdfsNativeWriter { async fn write(&mut self, _bs: Buffer) -> Result<()> { todo!() } async fn close(&mut self) -> Result { todo!() } async fn abort(&mut self) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "HdfsNativeWriter doesn't support abort", )) } } opendal-0.52.0/src/services/http/backend.rs000064400000000000000000000255771046102023000167000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::header; use http::header::IF_MATCH; use http::header::IF_NONE_MATCH; use http::Request; use http::Response; use http::StatusCode; use log::debug; use super::error::parse_error; use crate::raw::*; use crate::services::HttpConfig; use crate::*; impl Configurator for HttpConfig { type Builder = HttpBuilder; fn into_builder(self) -> Self::Builder { HttpBuilder { config: self, http_client: None, } } } /// HTTP Read-only service support like [Nginx](https://www.nginx.com/) and [Caddy](https://caddyserver.com/). #[doc = include_str!("docs.md")] #[derive(Default)] pub struct HttpBuilder { config: HttpConfig, http_client: Option, } impl Debug for HttpBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("HttpBuilder"); de.field("config", &self.config).finish() } } impl HttpBuilder { /// Set endpoint for http backend. /// /// For example: `https://example.com` pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.to_string()) }; self } /// set username for http backend /// /// default: no username pub fn username(mut self, username: &str) -> Self { if !username.is_empty() { self.config.username = Some(username.to_owned()); } self } /// set password for http backend /// /// default: no password pub fn password(mut self, password: &str) -> Self { if !password.is_empty() { self.config.password = Some(password.to_owned()); } self } /// set bearer token for http backend /// /// default: no access token pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()); } self } /// Set root path of http backend. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for HttpBuilder { const SCHEME: Scheme = Scheme::Http; type Config = HttpConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let endpoint = match &self.config.endpoint { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_context("service", Scheme::Http)) } }; let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Http) })? }; let mut auth = None; if let Some(username) = &self.config.username { auth = Some(format_authorization_by_basic( username, self.config.password.as_deref().unwrap_or_default(), )?); } if let Some(token) = &self.config.token { auth = Some(format_authorization_by_bearer(token)?) } Ok(HttpBackend { endpoint: endpoint.to_string(), authorization: auth, root, client, }) } } /// Backend is used to serve `Accessor` support for http. #[derive(Clone)] pub struct HttpBackend { endpoint: String, root: String, client: HttpClient, authorization: Option, } impl Debug for HttpBackend { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("endpoint", &self.endpoint) .field("root", &self.root) .field("client", &self.client) .finish() } } impl Access for HttpBackend { type Reader = HttpBody; type Writer = (); type Lister = (); type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut ma = AccessorInfo::default(); ma.set_scheme(Scheme::Http) .set_root(&self.root) .set_native_capability(Capability { stat: true, stat_with_if_match: true, stat_with_if_none_match: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, read: true, read_with_if_match: true, read_with_if_none_match: true, presign: !self.has_authorization(), presign_read: !self.has_authorization(), presign_stat: !self.has_authorization(), shared: true, ..Default::default() }); ma.into() } async fn stat(&self, path: &str, args: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let resp = self.http_head(path, &args).await?; let status = resp.status(); match status { StatusCode::OK => parse_into_metadata(path, resp.headers()).map(RpStat::new), // HTTP Server like nginx could return FORBIDDEN if auto-index // is not enabled, we should ignore them. StatusCode::NOT_FOUND | StatusCode::FORBIDDEN if path.ends_with('/') => { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.http_get(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn presign(&self, path: &str, args: OpPresign) -> Result { if self.has_authorization() { return Err(Error::new( ErrorKind::Unsupported, "Http doesn't support presigned request on backend with authorization", )); } let req = match args.operation() { PresignOperation::Stat(v) => self.http_head_request(path, v)?, PresignOperation::Read(v) => self.http_get_request(path, BytesRange::default(), v)?, _ => { return Err(Error::new( ErrorKind::Unsupported, "Http doesn't support presigned write", )) } }; let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } } impl HttpBackend { pub fn has_authorization(&self) -> bool { self.authorization.is_some() } pub fn http_get_request( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!("{}{}", self.endpoint, percent_encode_path(&p)); let mut req = Request::get(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth.clone()) } if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } req.body(Buffer::new()).map_err(new_request_build_error) } pub async fn http_get( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let req = self.http_get_request(path, range, args)?; self.client.fetch(req).await } pub fn http_head_request(&self, path: &str, args: &OpStat) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!("{}{}", self.endpoint, percent_encode_path(&p)); let mut req = Request::head(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth.clone()) } req.body(Buffer::new()).map_err(new_request_build_error) } async fn http_head(&self, path: &str, args: &OpStat) -> Result> { let req = self.http_head_request(path, args)?; self.client.send(req).await } } opendal-0.52.0/src/services/http/config.rs000064400000000000000000000031641046102023000165420ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Http service support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct HttpConfig { /// endpoint of this backend pub endpoint: Option, /// username of this backend pub username: Option, /// password of this backend pub password: Option, /// token of this backend pub token: Option, /// root of this backend pub root: Option, } impl Debug for HttpConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("HttpConfig"); de.field("endpoint", &self.endpoint); de.field("root", &self.root); de.finish_non_exhaustive() } } opendal-0.52.0/src/services/http/docs.md000064400000000000000000000014661046102023000162040ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [ ] ~~write~~ - [ ] ~~create_dir~~ - [ ] ~~delete~~ - [ ] ~~copy~~ - [ ] ~~rename~~ - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Notes Only `read` and `stat` are supported. We can use this service to visit any HTTP Server like nginx, caddy. ## Configuration - `endpoint`: set the endpoint for http - `root`: Set the work directory for backend You can refer to [`HttpBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Http; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create http backend builder let mut builder = Http::default().endpoint("127.0.0.1"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/http/error.rs000064400000000000000000000034431046102023000164260ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use crate::raw::*; use crate::*; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::NOT_MODIFIED => { (ErrorKind::ConditionNotMatch, false) } StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = String::from_utf8_lossy(&bs); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/http/mod.rs000064400000000000000000000017601046102023000160540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-http")] mod error; #[cfg(feature = "services-http")] mod backend; #[cfg(feature = "services-http")] pub use backend::HttpBuilder as Http; mod config; pub use config::HttpConfig; opendal-0.52.0/src/services/huggingface/backend.rs000064400000000000000000000223231046102023000201520ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::Response; use http::StatusCode; use log::debug; use super::core::HuggingfaceCore; use super::core::HuggingfaceStatus; use super::error::parse_error; use super::lister::HuggingfaceLister; use crate::raw::*; use crate::services::HuggingfaceConfig; use crate::*; impl Configurator for HuggingfaceConfig { type Builder = HuggingfaceBuilder; fn into_builder(self) -> Self::Builder { HuggingfaceBuilder { config: self } } } /// [Huggingface](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api)'s API support. #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct HuggingfaceBuilder { config: HuggingfaceConfig, } impl Debug for HuggingfaceBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Builder"); ds.field("config", &self.config); ds.finish() } } impl HuggingfaceBuilder { /// Set repo type of this backend. Default is model. /// /// Available values: /// - model /// - dataset /// /// Currently, only models and datasets are supported. /// [Reference](https://huggingface.co/docs/hub/repositories) pub fn repo_type(mut self, repo_type: &str) -> Self { if !repo_type.is_empty() { self.config.repo_type = Some(repo_type.to_string()); } self } /// Set repo id of this backend. This is required. /// /// Repo id consists of the account name and the repository name. /// /// For example, model's repo id looks like: /// - meta-llama/Llama-2-7b /// /// Dataset's repo id looks like: /// - databricks/databricks-dolly-15k pub fn repo_id(mut self, repo_id: &str) -> Self { if !repo_id.is_empty() { self.config.repo_id = Some(repo_id.to_string()); } self } /// Set revision of this backend. Default is main. /// /// Revision can be a branch name or a commit hash. /// /// For example, revision can be: /// - main /// - 1d0c4eb pub fn revision(mut self, revision: &str) -> Self { if !revision.is_empty() { self.config.revision = Some(revision.to_string()); } self } /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the token of this backend. /// /// This is optional. pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()); } self } } impl Builder for HuggingfaceBuilder { const SCHEME: Scheme = Scheme::Huggingface; type Config = HuggingfaceConfig; /// Build a HuggingfaceBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let repo_type = match self.config.repo_type.as_deref() { Some("model") => Ok(RepoType::Model), Some("dataset") => Ok(RepoType::Dataset), Some("space") => Err(Error::new( ErrorKind::ConfigInvalid, "repo type \"space\" is unsupported", )), Some(repo_type) => Err(Error::new( ErrorKind::ConfigInvalid, format!("unknown repo_type: {}", repo_type).as_str(), ) .with_operation("Builder::build") .with_context("service", Scheme::Huggingface)), None => Ok(RepoType::Model), }?; debug!("backend use repo_type: {:?}", &repo_type); let repo_id = match &self.config.repo_id { Some(repo_id) => Ok(repo_id.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "repo_id is empty") .with_operation("Builder::build") .with_context("service", Scheme::Huggingface)), }?; debug!("backend use repo_id: {}", &repo_id); let revision = match &self.config.revision { Some(revision) => revision.clone(), None => "main".to_string(), }; debug!("backend use revision: {}", &revision); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root: {}", &root); let token = self.config.token.as_ref().cloned(); let client = HttpClient::new()?; Ok(HuggingfaceBackend { core: Arc::new(HuggingfaceCore { repo_type, repo_id, revision, root, token, client, }), }) } } /// Backend for Huggingface service #[derive(Debug, Clone)] pub struct HuggingfaceBackend { core: Arc, } impl Access for HuggingfaceBackend { type Reader = HttpBody; type Writer = (); type Lister = oio::PageLister; type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Huggingface) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, list: true, list_with_recursive: true, list_has_content_length: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, _: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let resp = self.core.hf_path_info(path).await?; let status = resp.status(); match status { StatusCode::OK => { let mut meta = parse_into_metadata(path, resp.headers())?; let bs = resp.into_body(); let decoded_response: Vec = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; // NOTE: if the file is not found, the server will return 200 with an empty array if let Some(status) = decoded_response.first() { if let Some(commit_info) = status.last_commit.as_ref() { meta.set_last_modified(parse_datetime_from_rfc3339( commit_info.date.as_str(), )?); } meta.set_content_length(status.size); match status.type_.as_str() { "directory" => meta.set_mode(EntryMode::DIR), "file" => meta.set_mode(EntryMode::FILE), _ => return Err(Error::new(ErrorKind::Unexpected, "unknown status type")), }; } else { return Err(Error::new(ErrorKind::NotFound, "path not found")); } Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.hf_resolve(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = HuggingfaceLister::new(self.core.clone(), path.to_string(), args.recursive()); Ok((RpList::default(), oio::PageLister::new(l))) } } /// Repository type of Huggingface. Currently, we only support `model` and `dataset`. /// [Reference](https://huggingface.co/docs/hub/repositories) #[derive(Debug, Clone, Copy)] pub enum RepoType { Model, Dataset, } opendal-0.52.0/src/services/huggingface/config.rs000064400000000000000000000044261046102023000200340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Configuration for Huggingface service support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct HuggingfaceConfig { /// Repo type of this backend. Default is model. /// /// Available values: /// - model /// - dataset pub repo_type: Option, /// Repo id of this backend. /// /// This is required. pub repo_id: Option, /// Revision of this backend. /// /// Default is main. pub revision: Option, /// Root of this backend. Can be "/path/to/dir". /// /// Default is "/". pub root: Option, /// Token of this backend. /// /// This is optional. pub token: Option, } impl Debug for HuggingfaceConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("HuggingfaceConfig"); if let Some(repo_type) = &self.repo_type { ds.field("repo_type", &repo_type); } if let Some(repo_id) = &self.repo_id { ds.field("repo_id", &repo_id); } if let Some(revision) = &self.revision { ds.field("revision", &revision); } if let Some(root) = &self.root { ds.field("root", &root); } if self.token.is_some() { ds.field("token", &""); } ds.finish() } } opendal-0.52.0/src/services/huggingface/core.rs000064400000000000000000000325021046102023000175130ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use bytes::Bytes; use http::header; use http::Request; use http::Response; use serde::Deserialize; use super::backend::RepoType; use crate::raw::*; use crate::*; pub struct HuggingfaceCore { pub repo_type: RepoType, pub repo_id: String, pub revision: String, pub root: String, pub token: Option, pub client: HttpClient, } impl Debug for HuggingfaceCore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("HuggingfaceCore") .field("repo_type", &self.repo_type) .field("repo_id", &self.repo_id) .field("revision", &self.revision) .field("root", &self.root) .finish_non_exhaustive() } } impl HuggingfaceCore { pub async fn hf_path_info(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = match self.repo_type { RepoType::Model => format!( "https://huggingface.co/api/models/{}/paths-info/{}", &self.repo_id, &self.revision ), RepoType::Dataset => format!( "https://huggingface.co/api/datasets/{}/paths-info/{}", &self.repo_id, &self.revision ), }; let mut req = Request::post(&url); if let Some(token) = &self.token { let auth_header_content = format_authorization_by_bearer(token)?; req = req.header(header::AUTHORIZATION, auth_header_content); } req = req.header(header::CONTENT_TYPE, "application/x-www-form-urlencoded"); let req_body = format!("paths={}&expand=True", percent_encode_path(&p)); let req = req .body(Buffer::from(Bytes::from(req_body))) .map_err(new_request_build_error)?; self.client.send(req).await } pub async fn hf_list(&self, path: &str, recursive: bool) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let mut url = match self.repo_type { RepoType::Model => format!( "https://huggingface.co/api/models/{}/tree/{}/{}?expand=True", &self.repo_id, &self.revision, percent_encode_path(&p) ), RepoType::Dataset => format!( "https://huggingface.co/api/datasets/{}/tree/{}/{}?expand=True", &self.repo_id, &self.revision, percent_encode_path(&p) ), }; if recursive { url.push_str("&recursive=True"); } let mut req = Request::get(&url); if let Some(token) = &self.token { let auth_header_content = format_authorization_by_bearer(token)?; req = req.header(header::AUTHORIZATION, auth_header_content); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn hf_resolve( &self, path: &str, range: BytesRange, _args: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = match self.repo_type { RepoType::Model => format!( "https://huggingface.co/{}/resolve/{}/{}", &self.repo_id, &self.revision, percent_encode_path(&p) ), RepoType::Dataset => format!( "https://huggingface.co/datasets/{}/resolve/{}/{}", &self.repo_id, &self.revision, percent_encode_path(&p) ), }; let mut req = Request::get(&url); if let Some(token) = &self.token { let auth_header_content = format_authorization_by_bearer(token)?; req = req.header(header::AUTHORIZATION, auth_header_content); } if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } } #[derive(Deserialize, Eq, PartialEq, Debug)] #[serde(rename_all = "camelCase")] #[allow(dead_code)] pub(super) struct HuggingfaceStatus { #[serde(rename = "type")] pub type_: String, pub oid: String, pub size: u64, pub lfs: Option, pub path: String, pub last_commit: Option, pub security: Option, } #[derive(Deserialize, Eq, PartialEq, Debug)] #[serde(rename_all = "camelCase")] #[allow(dead_code)] pub(super) struct HuggingfaceLfs { pub oid: String, pub size: u64, pub pointer_size: u64, } #[derive(Deserialize, Eq, PartialEq, Debug)] #[serde(rename_all = "camelCase")] #[allow(dead_code)] pub(super) struct HuggingfaceLastCommit { pub id: String, pub title: String, pub date: String, } #[derive(Deserialize, Eq, PartialEq, Debug)] #[serde(rename_all = "camelCase")] #[allow(dead_code)] pub(super) struct HuggingfaceSecurity { pub blob_id: String, pub safe: bool, pub av_scan: Option, pub pickle_import_scan: Option, } #[derive(Deserialize, Eq, PartialEq, Debug)] #[allow(dead_code)] #[serde(rename_all = "camelCase")] pub(super) struct HuggingfaceAvScan { pub virus_found: bool, pub virus_names: Option>, } #[derive(Deserialize, Eq, PartialEq, Debug)] #[serde(rename_all = "camelCase")] #[allow(dead_code)] pub(super) struct HuggingfacePickleImportScan { pub highest_safety_level: String, pub imports: Vec, } #[derive(Deserialize, Eq, PartialEq, Debug)] #[allow(dead_code)] pub(super) struct HuggingfaceImport { pub module: String, pub name: String, pub safety: String, } #[cfg(test)] mod tests { use bytes::Bytes; use super::*; use crate::raw::new_json_deserialize_error; use crate::types::Result; #[test] fn parse_list_response_test() -> Result<()> { let resp = Bytes::from( r#" [ { "type": "file", "oid": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2", "size": 69512435, "lfs": { "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c", "size": 69512435, "pointerSize": 133 }, "path": "maelstrom/lib/maelstrom.jar" }, { "type": "directory", "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c", "size": 69512435, "path": "maelstrom/lib/plugins" } ] "#, ); let decoded_response = serde_json::from_slice::>(&resp) .map_err(new_json_deserialize_error)?; assert_eq!(decoded_response.len(), 2); let file_entry = HuggingfaceStatus { type_: "file".to_string(), oid: "45fa7c3d85ee7dd4139adbc056da25ae136a65f2".to_string(), size: 69512435, lfs: Some(HuggingfaceLfs { oid: "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c".to_string(), size: 69512435, pointer_size: 133, }), path: "maelstrom/lib/maelstrom.jar".to_string(), last_commit: None, security: None, }; assert_eq!(decoded_response[0], file_entry); let dir_entry = HuggingfaceStatus { type_: "directory".to_string(), oid: "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c".to_string(), size: 69512435, lfs: None, path: "maelstrom/lib/plugins".to_string(), last_commit: None, security: None, }; assert_eq!(decoded_response[1], dir_entry); Ok(()) } #[test] fn parse_files_info_test() -> Result<()> { let resp = Bytes::from( r#" [ { "type": "file", "oid": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2", "size": 69512435, "lfs": { "oid": "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c", "size": 69512435, "pointerSize": 133 }, "path": "maelstrom/lib/maelstrom.jar", "lastCommit": { "id": "bc1ef030bf3743290d5e190695ab94582e51ae2f", "title": "Upload 141 files", "date": "2023-11-17T23:50:28.000Z" }, "security": { "blobId": "45fa7c3d85ee7dd4139adbc056da25ae136a65f2", "name": "maelstrom/lib/maelstrom.jar", "safe": true, "avScan": { "virusFound": false, "virusNames": null }, "pickleImportScan": { "highestSafetyLevel": "innocuous", "imports": [ {"module": "torch", "name": "FloatStorage", "safety": "innocuous"}, {"module": "collections", "name": "OrderedDict", "safety": "innocuous"}, {"module": "torch", "name": "LongStorage", "safety": "innocuous"}, {"module": "torch._utils", "name": "_rebuild_tensor_v2", "safety": "innocuous"} ] } } } ] "#, ); let decoded_response = serde_json::from_slice::>(&resp) .map_err(new_json_deserialize_error)?; assert_eq!(decoded_response.len(), 1); let file_info = HuggingfaceStatus { type_: "file".to_string(), oid: "45fa7c3d85ee7dd4139adbc056da25ae136a65f2".to_string(), size: 69512435, lfs: Some(HuggingfaceLfs { oid: "b43f4c2ea569da1d66ca74e26ca8ea4430dfc29195e97144b2d0b4f3f6cafa1c".to_string(), size: 69512435, pointer_size: 133, }), path: "maelstrom/lib/maelstrom.jar".to_string(), last_commit: Some(HuggingfaceLastCommit { id: "bc1ef030bf3743290d5e190695ab94582e51ae2f".to_string(), title: "Upload 141 files".to_string(), date: "2023-11-17T23:50:28.000Z".to_string(), }), security: Some(HuggingfaceSecurity { blob_id: "45fa7c3d85ee7dd4139adbc056da25ae136a65f2".to_string(), safe: true, av_scan: Some(HuggingfaceAvScan { virus_found: false, virus_names: None, }), pickle_import_scan: Some(HuggingfacePickleImportScan { highest_safety_level: "innocuous".to_string(), imports: vec![ HuggingfaceImport { module: "torch".to_string(), name: "FloatStorage".to_string(), safety: "innocuous".to_string(), }, HuggingfaceImport { module: "collections".to_string(), name: "OrderedDict".to_string(), safety: "innocuous".to_string(), }, HuggingfaceImport { module: "torch".to_string(), name: "LongStorage".to_string(), safety: "innocuous".to_string(), }, HuggingfaceImport { module: "torch._utils".to_string(), name: "_rebuild_tensor_v2".to_string(), safety: "innocuous".to_string(), }, ], }), }), }; assert_eq!(decoded_response[0], file_info); Ok(()) } } opendal-0.52.0/src/services/huggingface/docs.md000064400000000000000000000034021046102023000174640ustar 00000000000000This service will visit the [Huggingface API](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) to access the Huggingface File System. Currently, we only support the `model` and `dataset` types of repositories, and operations are limited to reading and listing/stating. Huggingface doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the [`huggingface_hub` Source Code](https://github.com/huggingface/huggingface_hub). ## Capabilities This service can be used to: - [x] stat - [x] read - [ ] write - [ ] create_dir - [ ] delete - [ ] copy - [ ] rename - [x] list - [ ] ~~presign~~ - [ ] blocking ## Configurations - `repo_type`: The type of the repository. - `repo_id`: The id of the repository. - `revision`: The revision of the repository. - `root`: Set the work directory for backend. - `token`: The token for accessing the repository. Refer to [`HuggingfaceBuilder`]'s public API docs for more information. ## Examples ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Huggingface; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create Huggingface backend builder let mut builder = Huggingface::default() // set the type of Huggingface repository .repo_type("dataset") // set the id of Huggingface repository .repo_id("databricks/databricks-dolly-15k") // set the revision of Huggingface repository .revision("main") // set the root for Huggingface, all operations will happen under this root .root("/path/to/dir") // set the token for accessing the repository .token("access_token"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/huggingface/error.rs000064400000000000000000000055421046102023000177200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use http::Response; use http::StatusCode; use serde::Deserialize; use crate::raw::*; use crate::*; /// HuggingfaceError is the error returned by Huggingface File System. #[derive(Default, Deserialize)] struct HuggingfaceError { error: String, } impl Debug for HuggingfaceError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("HuggingfaceError"); de.field("message", &self.error.replace('\n', " ")); de.finish() } } pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED => (ErrorKind::ConditionNotMatch, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = match serde_json::from_slice::(&bs) { Ok(hf_error) => format!("{:?}", hf_error.error), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod test { use super::*; use crate::raw::new_json_deserialize_error; use crate::types::Result; #[test] fn test_parse_error() -> Result<()> { let resp = r#" { "error": "Invalid username or password." } "#; let decoded_response = serde_json::from_slice::(resp.as_bytes()) .map_err(new_json_deserialize_error)?; assert_eq!(decoded_response.error, "Invalid username or password."); Ok(()) } } opendal-0.52.0/src/services/huggingface/lister.rs000064400000000000000000000054021046102023000200640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use super::core::HuggingfaceCore; use super::core::HuggingfaceStatus; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct HuggingfaceLister { core: Arc, path: String, recursive: bool, } impl HuggingfaceLister { pub fn new(core: Arc, path: String, recursive: bool) -> Self { Self { core, path, recursive, } } } impl oio::PageList for HuggingfaceLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let response = self.core.hf_list(&self.path, self.recursive).await?; let status_code = response.status(); if !status_code.is_success() { let error = parse_error(response); return Err(error); } let bytes = response.into_body(); let decoded_response: Vec = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; ctx.done = true; for status in decoded_response { let entry_type = match status.type_.as_str() { "directory" => EntryMode::DIR, "file" => EntryMode::FILE, _ => EntryMode::Unknown, }; let mut meta = Metadata::new(entry_type); if let Some(commit_info) = status.last_commit.as_ref() { meta.set_last_modified(parse_datetime_from_rfc3339(commit_info.date.as_str())?); } if entry_type == EntryMode::FILE { meta.set_content_length(status.size); } let path = if entry_type == EntryMode::DIR { format!("{}/", &status.path) } else { status.path.clone() }; ctx.entries.push_back(oio::Entry::new( &build_rel_path(&self.core.root, &path), meta, )); } Ok(()) } } opendal-0.52.0/src/services/huggingface/mod.rs000064400000000000000000000022021046102023000173340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-huggingface")] mod core; #[cfg(feature = "services-huggingface")] mod error; #[cfg(feature = "services-huggingface")] mod lister; #[cfg(feature = "services-huggingface")] mod backend; #[cfg(feature = "services-huggingface")] pub use backend::HuggingfaceBuilder as Huggingface; mod config; pub use config::HuggingfaceConfig; opendal-0.52.0/src/services/icloud/backend.rs000064400000000000000000000215031046102023000171610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use tokio::sync::Mutex; use super::core::*; use crate::raw::*; use crate::services::IcloudConfig; use crate::*; impl Configurator for IcloudConfig { type Builder = IcloudBuilder; fn into_builder(self) -> Self::Builder { IcloudBuilder { config: self, http_client: None, } } } /// [IcloudDrive](https://www.icloud.com/iclouddrive/) service support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct IcloudBuilder { /// icloud config for web session request pub config: IcloudConfig, /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub http_client: Option, } impl Debug for IcloudBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("IcloudBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl IcloudBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Your Apple id /// /// It is required. your Apple login email, e.g. `example@gmail.com` pub fn apple_id(mut self, apple_id: &str) -> Self { self.config.apple_id = if apple_id.is_empty() { None } else { Some(apple_id.to_string()) }; self } /// Your Apple id password /// /// It is required. your icloud login password, e.g. `password` pub fn password(mut self, password: &str) -> Self { self.config.password = if password.is_empty() { None } else { Some(password.to_string()) }; self } /// Trust token and ds_web_auth_token is used for temporary access to the icloudDrive API. /// /// Authenticate using session token pub fn trust_token(mut self, trust_token: &str) -> Self { self.config.trust_token = if trust_token.is_empty() { None } else { Some(trust_token.to_string()) }; self } /// ds_web_auth_token must be set in Session /// /// Avoid Two Factor Authentication pub fn ds_web_auth_token(mut self, ds_web_auth_token: &str) -> Self { self.config.ds_web_auth_token = if ds_web_auth_token.is_empty() { None } else { Some(ds_web_auth_token.to_string()) }; self } /// Set if your apple id in China mainland. /// /// If in china mainland, we will connect to `https://www.icloud.com.cn`. /// Otherwise, we will connect to `https://www.icloud.com`. pub fn is_china_mainland(mut self, is_china_mainland: bool) -> Self { self.config.is_china_mainland = is_china_mainland; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for IcloudBuilder { const SCHEME: Scheme = Scheme::Icloud; type Config = IcloudConfig; fn build(self) -> Result { let root = normalize_root(&self.config.root.unwrap_or_default()); let apple_id = match &self.config.apple_id { Some(apple_id) => Ok(apple_id.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "apple_id is empty") .with_operation("Builder::build") .with_context("service", Scheme::Icloud)), }?; let password = match &self.config.password { Some(password) => Ok(password.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty") .with_operation("Builder::build") .with_context("service", Scheme::Icloud)), }?; let ds_web_auth_token = match &self.config.ds_web_auth_token { Some(ds_web_auth_token) => Ok(ds_web_auth_token.clone()), None => Err( Error::new(ErrorKind::ConfigInvalid, "ds_web_auth_token is empty") .with_operation("Builder::build") .with_context("service", Scheme::Icloud), ), }?; let trust_token = match &self.config.trust_token { Some(trust_token) => Ok(trust_token.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "trust_token is empty") .with_operation("Builder::build") .with_context("service", Scheme::Icloud)), }?; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Icloud) })? }; let session_data = SessionData::new(); let signer = IcloudSigner { client: client.clone(), data: session_data, apple_id, password, trust_token: Some(trust_token), ds_web_auth_token: Some(ds_web_auth_token), is_china_mainland: self.config.is_china_mainland, initiated: false, }; let signer = Arc::new(Mutex::new(signer)); Ok(IcloudBackend { core: Arc::new(IcloudCore { signer: signer.clone(), root, path_cache: PathCacher::new(IcloudPathQuery::new(signer.clone())), }), }) } } #[derive(Debug, Clone)] pub struct IcloudBackend { core: Arc, } impl Access for IcloudBackend { type Reader = HttpBody; type Writer = (); type Lister = (); type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut ma = AccessorInfo::default(); ma.set_scheme(Scheme::Icloud) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, shared: true, ..Default::default() }); ma.into() } async fn stat(&self, path: &str, _: OpStat) -> Result { // icloud get the filename by id, instead obtain the metadata by filename if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let node = self.core.stat(path).await?; let mut meta = Metadata::new(match node.type_field.as_str() { "FOLDER" => EntryMode::DIR, _ => EntryMode::FILE, }); if meta.mode() == EntryMode::DIR || path.ends_with('/') { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } meta = meta.with_content_length(node.size); let last_modified = parse_datetime_from_rfc3339(&node.date_modified)?; meta = meta.with_last_modified(last_modified); Ok(RpStat::new(meta)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.read(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } } opendal-0.52.0/src/services/icloud/config.rs000064400000000000000000000041511046102023000170370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for icloud services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct IcloudConfig { /// root of this backend. /// /// All operations will happen under this root. /// /// default to `/` if not set. pub root: Option, /// apple_id of this backend. /// /// apple_id must be full, mostly like `example@gmail.com`. pub apple_id: Option, /// password of this backend. /// /// password must be full. pub password: Option, /// Session /// /// token must be valid. pub trust_token: Option, /// ds_web_auth_token must be set in Session pub ds_web_auth_token: Option, /// enable the china origin /// China region `origin` Header needs to be set to "https://www.icloud.com.cn". /// /// otherwise Apple server will return 302. pub is_china_mainland: bool, } impl Debug for IcloudConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("IcloudBuilder"); d.field("root", &self.root); d.field("is_china_mainland", &self.is_china_mainland); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/icloud/core.rs000064400000000000000000000662001046102023000165250ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::BTreeMap; use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use bytes::Bytes; use http::header; use http::header::IF_MATCH; use http::header::IF_NONE_MATCH; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use serde::Serialize; use serde_json::json; use tokio::sync::Mutex; use crate::raw::*; use crate::*; static ACCOUNT_COUNTRY_HEADER: &str = "X-Apple-ID-Account-Country"; static OAUTH_STATE_HEADER: &str = "X-Apple-OAuth-State"; static SESSION_ID_HEADER: &str = "X-Apple-ID-Session-Id"; static SCNT_HEADER: &str = "scnt"; static SESSION_TOKEN_HEADER: &str = "X-Apple-Session-Token"; static APPLE_RESPONSE_HEADER: &str = "X-Apple-I-Rscd"; static AUTH_ENDPOINT: &str = "https://idmsa.apple.com/appleauth/auth"; static SETUP_ENDPOINT: &str = "https://setup.icloud.com/setup/ws/1"; const AUTH_HEADERS: [(&str, &str); 7] = [ ( // This code inspire from // https://github.com/picklepete/pyicloud/blob/master/pyicloud/base.py#L392 "X-Apple-OAuth-Client-Id", "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d", ), ("X-Apple-OAuth-Client-Type", "firstPartyAuth"), ("X-Apple-OAuth-Redirect-URI", "https://www.icloud.com"), ("X-Apple-OAuth-Require-Grant-Code", "true"), ("X-Apple-OAuth-Response-Mode", "web_message"), ("X-Apple-OAuth-Response-Type", "code"), ( "X-Apple-Widget-Key", "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d", ), ]; #[derive(Clone)] pub struct SessionData { oauth_state: String, session_id: Option, session_token: Option, scnt: Option, account_country: Option, cookies: BTreeMap, drivews_url: String, docws_url: String, } impl Default for SessionData { fn default() -> Self { Self::new() } } impl SessionData { pub fn new() -> SessionData { Self { oauth_state: format!("auth-{}", uuid::Uuid::new_v4()).to_string(), session_id: None, session_token: None, scnt: None, account_country: None, cookies: BTreeMap::default(), drivews_url: String::new(), docws_url: String::new(), } } } #[derive(Clone)] pub struct IcloudSigner { pub client: HttpClient, pub apple_id: String, pub password: String, pub is_china_mainland: bool, pub trust_token: Option, pub ds_web_auth_token: Option, pub data: SessionData, pub initiated: bool, } impl Debug for IcloudSigner { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("icloud signer"); de.field("is_china_mainland", &self.is_china_mainland); de.finish() } } impl IcloudSigner { /// Get the drivews_url from signer session data. /// Async await init finish. pub async fn drivews_url(&mut self) -> Result<&str> { self.init().await?; Ok(&self.data.drivews_url) } /// Get the docws_url from signer session data. /// Async await init finish. pub async fn docws_url(&mut self) -> Result<&str> { self.init().await?; Ok(&self.data.docws_url) } /// iCloud will use our oauth state as client id. pub fn client_id(&self) -> &str { &self.data.oauth_state } async fn init(&mut self) -> Result<()> { if self.initiated { return Ok(()); } // Sign the auth endpoint first. let uri = format!("{}/signin?isRememberMeEnable=true", AUTH_ENDPOINT); let body = serde_json::to_vec(&json!({ "accountName" : self.apple_id, "password" : self.password, "rememberMe": true, "trustTokens": [self.trust_token.clone().unwrap()], })) .map_err(new_json_serialize_error)?; let mut req = Request::post(uri) .header(header::CONTENT_TYPE, "application/json") .body(Buffer::from(Bytes::from(body))) .map_err(new_request_build_error)?; self.sign(&mut req)?; let resp = self.client.send(req).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } if let Some(rscd) = resp.headers().get(APPLE_RESPONSE_HEADER) { let status_code = StatusCode::from_bytes(rscd.as_bytes()).unwrap(); if status_code != StatusCode::CONFLICT { return Err(parse_error(resp)); } } // Setup to get the session id. let uri = format!("{}/accountLogin", SETUP_ENDPOINT); let body = serde_json::to_vec(&json!({ "accountCountryCode": self.data.account_country.clone().unwrap_or_default(), "dsWebAuthToken":self.ds_web_auth_token.clone().unwrap_or_default(), "extended_login": true, "trustToken": self.trust_token.clone().unwrap_or_default(),})) .map_err(new_json_serialize_error)?; let mut req = Request::post(uri) .header(header::CONTENT_TYPE, "application/json") .body(Buffer::from(Bytes::from(body))) .map_err(new_request_build_error)?; self.sign(&mut req)?; let resp = self.client.send(req).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } // Update SessionData cookies.We need obtain `X-APPLE-WEBAUTH-USER` cookie to get file. self.update(&resp)?; let bs = resp.into_body(); let auth_info: IcloudWebservicesResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; // Check if we have extra challenge to take. if auth_info.hsa_challenge_required && !auth_info.hsa_trusted_browser { return Err(Error::new(ErrorKind::Unexpected, "Apple icloud AuthenticationFailed:Unauthorized request:Needs two-factor authentication")); } if let Some(v) = &auth_info.webservices.drivews.url { self.data.drivews_url = v.to_string(); } if let Some(v) = &auth_info.webservices.docws.url { self.data.docws_url = v.to_string(); } self.initiated = true; Ok(()) } fn sign(&mut self, req: &mut Request) -> Result<()> { let headers = req.headers_mut(); headers.insert( OAUTH_STATE_HEADER, build_header_value(&self.data.oauth_state)?, ); if let Some(session_id) = &self.data.session_id { headers.insert(SESSION_ID_HEADER, build_header_value(session_id)?); } if let Some(scnt) = &self.data.scnt { headers.insert(SCNT_HEADER, build_header_value(scnt)?); } // You can get more information from [apple.com](https://support.apple.com/en-us/111754) if self.is_china_mainland { headers.insert( header::ORIGIN, build_header_value("https://www.icloud.com.cn")?, ); headers.insert( header::REFERER, build_header_value("https://www.icloud.com.cn/")?, ); } else { headers.insert( header::ORIGIN, build_header_value("https://www.icloud.com")?, ); headers.insert( header::REFERER, build_header_value("https://www.icloud.com/")?, ); } if !self.data.cookies.is_empty() { let cookies: Vec = self .data .cookies .iter() .map(|(k, v)| format!("{}={}", k, v)) .collect(); headers.insert( header::COOKIE, build_header_value(&cookies.as_slice().join("; "))?, ); } for (key, value) in AUTH_HEADERS { headers.insert(key, build_header_value(value)?); } Ok(()) } /// Update signer's data after request sent out. fn update(&mut self, resp: &Response) -> Result<()> { if let Some(account_country) = parse_header_to_str(resp.headers(), ACCOUNT_COUNTRY_HEADER)? { self.data.account_country = Some(account_country.to_string()); } if let Some(session_id) = parse_header_to_str(resp.headers(), SESSION_ID_HEADER)? { self.data.session_id = Some(session_id.to_string()); } if let Some(session_token) = parse_header_to_str(resp.headers(), SESSION_TOKEN_HEADER)? { self.data.session_token = Some(session_token.to_string()); } if let Some(scnt) = parse_header_to_str(resp.headers(), SCNT_HEADER)? { self.data.scnt = Some(scnt.to_string()); } let cookies: Vec = resp .headers() .get_all(header::SET_COOKIE) .iter() .map(|v| v.to_str().unwrap().to_string()) .collect(); for cookie in cookies { if let Some((key, value)) = cookie.split_once('=') { self.data.cookies.insert(key.into(), value.into()); } } Ok(()) } /// Send will make sure the following things: /// /// - Init the signer if it's not initiated. /// - Sign the request. /// - Update the session data if needed. pub async fn send(&mut self, mut req: Request) -> Result> { self.sign(&mut req)?; let resp = self.client.send(req).await?; Ok(resp) } } pub struct IcloudCore { pub signer: Arc>, pub root: String, pub path_cache: PathCacher, } impl Debug for IcloudCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("IcloudCore"); de.field("root", &self.root); de.finish() } } impl IcloudCore { // Retrieves a root within the icloud Drive. // "FOLDER::com.apple.CloudDocs::root" pub async fn get_root(&self, id: &str) -> Result { let mut signer = self.signer.lock().await; let uri = format!( "{}/retrieveItemDetailsInFolders", signer.drivews_url().await? ); let body = serde_json::to_vec(&json!([ { "drivewsid": id, "partialData": false } ])) .map_err(new_json_serialize_error)?; let req = Request::post(uri) .body(Buffer::from(Bytes::from(body))) .map_err(new_request_build_error)?; let resp = signer.send(req).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } let body = resp.into_body(); let drive_node: Vec = serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?; Ok(drive_node[0].clone()) } pub async fn get_file( &self, id: &str, zone: &str, range: BytesRange, args: OpRead, ) -> Result> { let mut signer = self.signer.lock().await; let uri = format!( "{}/ws/{}/download/by_id?document_id={}", signer.docws_url().await?, zone, id ); let req = Request::get(uri) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = signer.send(req).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } let body = resp.into_body(); let object: IcloudObject = serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?; let url = object.data_token.url.to_string(); let mut req = Request::get(url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if range.is_full() { req = req.header(header::RANGE, range.to_header()) } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = signer.client.fetch(req).await?; Ok(resp) } pub async fn read( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let path = build_rooted_abs_path(&self.root, path); let base = get_basename(&path); let path_id = self.path_cache.get(base).await?.ok_or(Error::new( ErrorKind::NotFound, format!("read path not found: {}", base), ))?; if let Some(docwsid) = path_id.strip_prefix("FILE::com.apple.CloudDocs::") { Ok(self .get_file(docwsid, "com.apple.CloudDocs", range, args.clone()) .await?) } else { Err(Error::new( ErrorKind::NotFound, "icloud DriveService read error", )) } } pub async fn stat(&self, path: &str) -> Result { let path = build_rooted_abs_path(&self.root, path); let mut base = get_basename(&path); let parent = get_parent(&path); if base.ends_with('/') { base = base.trim_end_matches('/'); } let file_id = self.path_cache.get(base).await?.ok_or(Error::new( ErrorKind::NotFound, format!("stat path not found: {}", base), ))?; let folder_id = self.path_cache.get(parent).await?.ok_or(Error::new( ErrorKind::NotFound, format!("stat path not found: {}", parent), ))?; let node = self.get_root(&folder_id).await?; match node.items.iter().find(|it| it.drivewsid == file_id.clone()) { Some(it) => Ok(it.clone()), None => Err(Error::new( ErrorKind::NotFound, "icloud DriveService stat get parent items error", )), } } } pub struct IcloudPathQuery { pub signer: Arc>, } impl IcloudPathQuery { pub fn new(signer: Arc>) -> Self { IcloudPathQuery { signer } } } impl PathQuery for IcloudPathQuery { async fn root(&self) -> Result { Ok("FOLDER::com.apple.CloudDocs::root".to_string()) } /// Retrieves the root directory within the icloud Drive. /// /// FIXME: we are reading the entire dir to find the file, this is not efficient. /// Maybe we should build a new path cache for this kind of services instead. async fn query(&self, parent_id: &str, name: &str) -> Result> { let mut signer = self.signer.lock().await; let uri = format!( "{}/retrieveItemDetailsInFolders", signer.drivews_url().await? ); let body = serde_json::to_vec(&json!([ { "drivewsid": parent_id, "partialData": false } ])) .map_err(new_json_serialize_error)?; let req = Request::post(uri) .body(Buffer::from(Bytes::from(body))) .map_err(new_request_build_error)?; let resp = signer.send(req).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } let body = resp.into_body(); let root: Vec = serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?; let node = &root[0]; Ok(node .items .iter() .find(|it| it.name == name) .map(|it| it.drivewsid.clone())) } async fn create_dir(&self, parent_id: &str, name: &str) -> Result { let mut signer = self.signer.lock().await; let client_id = signer.client_id().to_string(); let uri = format!("{}/createFolders", signer.drivews_url().await?); let body = serde_json::to_vec(&json!( { "destinationDrivewsId": parent_id, "folders": [ { "clientId": client_id, "name": name, } ], } )) .map_err(new_json_serialize_error)?; let req = Request::post(uri) .body(Buffer::from(Bytes::from(body))) .map_err(new_request_build_error)?; let resp = signer.send(req).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } let body = resp.into_body(); let create_folder: IcloudCreateFolder = serde_json::from_slice(body.chunk()).map_err(new_json_deserialize_error)?; Ok(create_folder.destination_drivews_id) } } pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let mut kind = match parts.status.as_u16() { 421 | 450 | 500 => ErrorKind::NotFound, 401 => ErrorKind::Unexpected, _ => ErrorKind::Unexpected, }; let (message, icloud_err) = serde_json::from_reader::<_, IcloudError>(bs.clone().reader()) .map(|icloud_err| (format!("{icloud_err:?}"), Some(icloud_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); if let Some(icloud_err) = &icloud_err { kind = match icloud_err.status_code.as_str() { "NOT_FOUND" => ErrorKind::NotFound, "PERMISSION_DENIED" => ErrorKind::PermissionDenied, _ => ErrorKind::Unexpected, } } let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); err } #[derive(Default, Debug, Deserialize)] #[allow(dead_code)] struct IcloudError { status_code: String, message: String, } #[derive(Default, Deserialize, Clone)] #[serde(rename_all = "camelCase")] pub struct IcloudWebservicesResponse { #[serde(default)] pub hsa_challenge_required: bool, #[serde(default)] pub hsa_trusted_browser: bool, pub webservices: Webservices, } #[derive(Deserialize, Default, Clone, Debug)] pub struct Webservices { pub drivews: Drivews, pub docws: Docws, } #[derive(Deserialize, Default, Clone, Debug)] #[serde(rename_all = "camelCase")] pub struct Drivews { pub url: Option, } #[derive(Deserialize, Default, Clone, Debug)] #[serde(rename_all = "camelCase")] pub struct Docws { pub url: Option, } #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct IcloudRoot { #[serde(default)] pub asset_quota: i64, #[serde(default)] pub date_created: String, #[serde(default)] pub direct_children_count: i64, pub docwsid: String, pub drivewsid: String, pub etag: String, #[serde(default)] pub file_count: i64, pub items: Vec, pub name: String, pub number_of_items: i64, pub status: String, #[serde(rename = "type")] pub type_field: String, pub zone: String, } #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] #[serde(rename_all = "camelCase")] pub struct IcloudItem { #[serde(default)] pub asset_quota: Option, #[serde(default)] pub date_created: String, #[serde(default)] pub date_modified: String, #[serde(default)] pub direct_children_count: Option, pub docwsid: String, pub drivewsid: String, pub etag: String, #[serde(default)] pub file_count: Option, pub item_id: Option, pub name: String, pub parent_id: String, #[serde(default)] pub size: u64, #[serde(rename = "type")] pub type_field: String, pub zone: String, #[serde(default)] pub max_depth: Option, #[serde(default)] pub is_chained_to_parent: Option, } #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct IcloudObject { pub document_id: String, pub item_id: String, pub owner_dsid: i64, pub data_token: DataToken, pub double_etag: String, } #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct DataToken { pub url: String, pub token: String, pub signature: String, pub wrapping_key: String, pub reference_signature: String, } #[derive(Default, Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct IcloudCreateFolder { pub destination_drivews_id: String, pub folders: Vec, } #[cfg(test)] mod tests { use pretty_assertions::assert_eq; use super::IcloudRoot; use super::IcloudWebservicesResponse; #[test] fn test_parse_icloud_drive_root_json() { let data = r#"{ "assetQuota": 19603579, "dateCreated": "2019-06-10T14:17:49Z", "directChildrenCount": 3, "docwsid": "root", "drivewsid": "FOLDER::com.apple.CloudDocs::root", "etag": "w7", "fileCount": 22, "items": [ { "assetQuota": 19603579, "dateCreated": "2021-02-05T08:30:58Z", "directChildrenCount": 22, "docwsid": "1E013608-C669-43DB-AC14-3D7A4E0A0500", "drivewsid": "FOLDER::com.apple.CloudDocs::1E013608-C669-43DB-AC14-3D7A4E0A0500", "etag": "sn", "fileCount": 22, "item_id": "CJWdk48eEAAiEB4BNgjGaUPbrBQ9ek4KBQAoAQ", "name": "Downloads", "parentId": "FOLDER::com.apple.CloudDocs::root", "shareAliasCount": 0, "shareCount": 0, "type": "FOLDER", "zone": "com.apple.CloudDocs" }, { "dateCreated": "2019-06-10T14:17:54Z", "docwsid": "documents", "drivewsid": "FOLDER::com.apple.Keynote::documents", "etag": "1v", "maxDepth": "ANY", "name": "Keynote", "parentId": "FOLDER::com.apple.CloudDocs::root", "type": "APP_LIBRARY", "zone": "com.apple.Keynote" }, { "assetQuota": 0, "dateCreated": "2024-01-06T02:35:08Z", "directChildrenCount": 0, "docwsid": "21E4A15E-DA77-472A-BAC8-B0C35A91F237", "drivewsid": "FOLDER::com.apple.CloudDocs::21E4A15E-DA77-472A-BAC8-B0C35A91F237", "etag": "w8", "fileCount": 0, "isChainedToParent": true, "item_id": "CJWdk48eEAAiECHkoV7ad0cqusiww1qR8jcoAQ", "name": "opendal", "parentId": "FOLDER::com.apple.CloudDocs::root", "shareAliasCount": 0, "shareCount": 0, "type": "FOLDER", "zone": "com.apple.CloudDocs" } ], "name": "", "numberOfItems": 16, "shareAliasCount": 0, "shareCount": 0, "status": "OK", "type": "FOLDER", "zone": "com.apple.CloudDocs" }"#; let response: IcloudRoot = serde_json::from_str(data).unwrap(); assert_eq!(response.name, ""); assert_eq!(response.type_field, "FOLDER"); assert_eq!(response.zone, "com.apple.CloudDocs"); assert_eq!(response.docwsid, "root"); assert_eq!(response.drivewsid, "FOLDER::com.apple.CloudDocs::root"); assert_eq!(response.etag, "w7"); assert_eq!(response.file_count, 22); } #[test] fn test_parse_icloud_drive_folder_file() { let data = r#"{ "assetQuota": 19603579, "dateCreated": "2021-02-05T08:34:21Z", "directChildrenCount": 22, "docwsid": "1E013608-C669-43DB-AC14-3D7A4E0A0500", "drivewsid": "FOLDER::com.apple.CloudDocs::1E013608-C669-43DB-AC14-3D7A4E0A0500", "etag": "w9", "fileCount": 22, "items": [ { "dateChanged": "2021-02-18T14:10:46Z", "dateCreated": "2021-02-10T07:01:34Z", "dateModified": "2021-02-10T07:01:34Z", "docwsid": "9605331E-7BF3-41A0-A128-A68FFA377C50", "drivewsid": "FILE::com.apple.CloudDocs::9605331E-7BF3-41A0-A128-A68FFA377C50", "etag": "5b::5a", "extension": "pdf", "item_id": "CJWdk48eEAAiEJYFMx5780GgoSimj_o3fFA", "lastOpenTime": "2021-02-10T10:28:42Z", "name": "1-11-ARP-notes", "parentId": "FOLDER::com.apple.CloudDocs::1E013608-C669-43DB-AC14-3D7A4E0A0500", "size": 639483, "type": "FILE", "zone": "com.apple.CloudDocs" } ], "name": "Downloads", "numberOfItems": 22, "parentId": "FOLDER::com.apple.CloudDocs::root", "shareAliasCount": 0, "shareCount": 0, "status": "OK", "type": "FOLDER", "zone": "com.apple.CloudDocs" }"#; let response = serde_json::from_str::(data).unwrap(); assert_eq!(response.name, "Downloads"); assert_eq!(response.type_field, "FOLDER"); assert_eq!(response.zone, "com.apple.CloudDocs"); assert_eq!(response.docwsid, "1E013608-C669-43DB-AC14-3D7A4E0A0500"); assert_eq!( response.drivewsid, "FOLDER::com.apple.CloudDocs::1E013608-C669-43DB-AC14-3D7A4E0A0500" ); assert_eq!(response.etag, "w9"); assert_eq!(response.file_count, 22); } #[test] fn test_parse_icloud_webservices() { let data = r#" { "hsaChallengeRequired": false, "hsaTrustedBrowser": true, "webservices": { "docws": { "pcsRequired": true, "status": "active", "url": "https://p219-docws.icloud.com.cn:443" }, "drivews": { "pcsRequired": true, "status": "active", "url": "https://p219-drivews.icloud.com.cn:443" } } } "#; let response = serde_json::from_str::(data).unwrap(); assert!(!response.hsa_challenge_required); assert!(response.hsa_trusted_browser); assert_eq!( response.webservices.docws.url, Some("https://p219-docws.icloud.com.cn:443".to_string()) ); assert_eq!( response.webservices.drivews.url, Some("https://p219-drivews.icloud.com.cn:443".to_string()) ); } } opendal-0.52.0/src/services/icloud/docs.md000064400000000000000000000033721046102023000165020ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [ ] write - [ ] delete - [ ] create_dir - [ ] list - [ ] copy - [ ] rename - [ ] batch # Configuration - `root`: Set the work directory for backend ### Credentials related #### provide Session (Temporary) - `trust_token`: set the trust_token for icloud drive api Please notice its expiration. - `ds_web_auth_token`: set the ds_web_auth_token for icloud drive api Get web trust the session. - `is_china_mainland`: set the is_china_mainland for icloud drive api China region must true to use Otherwise Apple server will return 302. More information you can get [apple.com](https://support.apple.com/en-us/111754) OpenDAL is a library, it cannot do the first step of OAuth2 for you. You need to get authorization code from user by calling icloudDrive's authorize url and save it for session's trust_token and session_token(ds_web_auth_token). Make sure you have enabled Apple icloud Drive API in your Apple icloud ID. And your OAuth scope contains valid `Session`. You can get more information from [pyicloud](https://github.com/picklepete/pyicloud/tree/master?tab=readme-ov-file#authentication) or [iCloud-API](https://github.com/MauriceConrad/iCloud-API?tab=readme-ov-file#getting-started) You can refer to [`IcloudBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Icloud; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Icloud::default() .root("/") .apple_id("") .password("") .trust_token("") .ds_web_auth_token("") .is_china_mainland(true); Ok(()) } opendal-0.52.0/src/services/icloud/mod.rs000064400000000000000000000017731046102023000163600ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-icloud")] mod core; #[cfg(feature = "services-icloud")] mod backend; #[cfg(feature = "services-icloud")] pub use backend::IcloudBuilder as Icloud; mod config; pub use config::IcloudConfig; opendal-0.52.0/src/services/ipfs/backend.rs000064400000000000000000000365321046102023000166530ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Request; use http::Response; use http::StatusCode; use log::debug; use prost::Message; use super::error::parse_error; use super::ipld::PBNode; use crate::raw::*; use crate::services::IpfsConfig; use crate::*; impl Configurator for IpfsConfig { type Builder = IpfsBuilder; fn into_builder(self) -> Self::Builder { IpfsBuilder { config: self, http_client: None, } } } /// IPFS file system support based on [IPFS HTTP Gateway](https://docs.ipfs.tech/concepts/ipfs-gateway/). #[doc = include_str!("docs.md")] #[derive(Default, Clone, Debug)] pub struct IpfsBuilder { config: IpfsConfig, http_client: Option, } impl IpfsBuilder { /// Set root of ipfs backend. /// /// Root must be a valid ipfs address like the following: /// /// - `/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ/` (IPFS with CID v0) /// - `/ipfs/bafybeibozpulxtpv5nhfa2ue3dcjx23ndh3gwr5vwllk7ptoyfwnfjjr4q/` (IPFS with CID v1) /// - `/ipns/opendal.apache.org/` (IPNS) pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set endpoint if ipfs backend. /// /// Endpoint must be a valid ipfs gateway which passed the [IPFS Gateway Checker](https://ipfs.github.io/public-gateway-checker/) /// /// Popular choices including: /// /// - `https://ipfs.io` /// - `https://w3s.link` /// - `https://dweb.link` /// - `https://cloudflare-ipfs.com` /// - `http://127.0.0.1:8080` (ipfs daemon in local) pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()); } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for IpfsBuilder { const SCHEME: Scheme = Scheme::Ipfs; type Config = IpfsConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.unwrap_or_default()); if !root.starts_with("/ipfs/") && !root.starts_with("/ipns/") { return Err(Error::new( ErrorKind::ConfigInvalid, "root must start with /ipfs/ or /ipns/", ) .with_context("service", Scheme::Ipfs) .with_context("root", &root)); } debug!("backend use root {}", root); let endpoint = match &self.config.endpoint { Some(endpoint) => Ok(endpoint.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_context("service", Scheme::Ipfs) .with_context("root", &root)), }?; debug!("backend use endpoint {}", &endpoint); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Ipfs) })? }; Ok(IpfsBackend { root, endpoint, client, }) } } /// Backend for IPFS. #[derive(Clone)] pub struct IpfsBackend { endpoint: String, root: String, client: HttpClient, } impl Debug for IpfsBackend { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("endpoint", &self.endpoint) .field("root", &self.root) .field("client", &self.client) .finish() } } impl Access for IpfsBackend { type Reader = HttpBody; type Writer = (); type Lister = oio::PageLister; type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut ma = AccessorInfo::default(); ma.set_scheme(Scheme::Ipfs) .set_root(&self.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_etag: true, stat_has_content_disposition: true, read: true, list: true, shared: true, ..Default::default() }); ma.into() } /// IPFS's stat behavior highly depends on its implementation. /// /// Based on IPFS [Path Gateway Specification](https://github.com/ipfs/specs/blob/main/http-gateways/PATH_GATEWAY.md), /// response payload could be: /// /// > - UnixFS (implicit default) /// > - File /// > - Bytes representing file contents /// > - Directory /// > - Generated HTML with directory index /// > - When `index.html` is present, gateway can skip generating directory index and return it instead /// > - Raw block (not this case) /// > - CAR (not this case) /// /// When we HEAD a given path, we could have the following responses: /// /// - File /// /// ```http /// :) curl -I https://ipfs.io/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ/normal_file /// HTTP/1.1 200 Connection established /// /// HTTP/2 200 /// server: openresty /// date: Thu, 08 Sep 2022 00:48:50 GMT /// content-type: application/octet-stream /// content-length: 262144 /// access-control-allow-methods: GET /// cache-control: public, max-age=29030400, immutable /// etag: "QmdP6teFTLSNVhT4W5jkhEuUBsjQ3xkp1GmRvDU6937Me1" /// x-ipfs-gateway-host: ipfs-bank11-fr2 /// x-ipfs-path: /ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ/normal_file /// x-ipfs-roots: QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ,QmdP6teFTLSNVhT4W5jkhEuUBsjQ3xkp1GmRvDU6937Me1 /// x-ipfs-pop: ipfs-bank11-fr2 /// timing-allow-origin: * /// x-ipfs-datasize: 262144 /// access-control-allow-origin: * /// access-control-allow-methods: GET, POST, OPTIONS /// access-control-allow-headers: X-Requested-With, Range, Content-Range, X-Chunked-Output, X-Stream-Output /// access-control-expose-headers: Content-Range, X-Chunked-Output, X-Stream-Output /// x-ipfs-lb-pop: gateway-bank1-fr2 /// strict-transport-security: max-age=31536000; includeSubDomains; preload /// x-proxy-cache: MISS /// accept-ranges: bytes /// ``` /// /// - Dir with generated index /// /// ```http /// :( curl -I https://ipfs.io/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ/normal_dir /// HTTP/1.1 200 Connection established /// /// HTTP/2 200 /// server: openresty /// date: Wed, 07 Sep 2022 08:46:13 GMT /// content-type: text/html /// vary: Accept-Encoding /// access-control-allow-methods: GET /// etag: "DirIndex-2b567f6r5vvdg_CID-QmY44DyCDymRN1Qy7sGbupz1ysMkXTWomAQku5vBg7fRQW" /// x-ipfs-gateway-host: ipfs-bank6-sg1 /// x-ipfs-path: /ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ/normal_dir /// x-ipfs-roots: QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ,QmY44DyCDymRN1Qy7sGbupz1ysMkXTWomAQku5vBg7fRQW /// x-ipfs-pop: ipfs-bank6-sg1 /// timing-allow-origin: * /// access-control-allow-origin: * /// access-control-allow-methods: GET, POST, OPTIONS /// access-control-allow-headers: X-Requested-With, Range, Content-Range, X-Chunked-Output, X-Stream-Output /// access-control-expose-headers: Content-Range, X-Chunked-Output, X-Stream-Output /// x-ipfs-lb-pop: gateway-bank3-sg1 /// strict-transport-security: max-age=31536000; includeSubDomains; preload /// x-proxy-cache: MISS /// ``` /// /// - Dir with index.html /// /// ```http /// :) curl -I http://127.0.0.1:8080/ipfs/QmVturFGV3z4WsP7cRV8Ci4avCdGWYXk2qBKvtAwFUp5Az /// HTTP/1.1 302 Found /// Access-Control-Allow-Headers: Content-Type /// Access-Control-Allow-Headers: Range /// Access-Control-Allow-Headers: User-Agent /// Access-Control-Allow-Headers: X-Requested-With /// Access-Control-Allow-Methods: GET /// Access-Control-Allow-Origin: * /// Access-Control-Expose-Headers: Content-Length /// Access-Control-Expose-Headers: Content-Range /// Access-Control-Expose-Headers: X-Chunked-Output /// Access-Control-Expose-Headers: X-Ipfs-Path /// Access-Control-Expose-Headers: X-Ipfs-Roots /// Access-Control-Expose-Headers: X-Stream-Output /// Content-Type: text/html; charset=utf-8 /// Location: /ipfs/QmVturFGV3z4WsP7cRV8Ci4avCdGWYXk2qBKvtAwFUp5Az/ /// X-Ipfs-Path: /ipfs/QmVturFGV3z4WsP7cRV8Ci4avCdGWYXk2qBKvtAwFUp5Az /// X-Ipfs-Roots: QmVturFGV3z4WsP7cRV8Ci4avCdGWYXk2qBKvtAwFUp5Az /// Date: Thu, 08 Sep 2022 00:52:29 GMT /// ``` /// /// In conclusion: /// /// - HTTP Status Code == 302 => directory /// - HTTP Status Code == 200 && ETag starts with `"DirIndex` => directory /// - HTTP Status Code == 200 && ETag not starts with `"DirIndex` => file async fn stat(&self, path: &str, _: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let resp = self.ipfs_head(path).await?; let status = resp.status(); match status { StatusCode::OK => { let mut m = Metadata::new(EntryMode::Unknown); if let Some(v) = parse_content_length(resp.headers())? { m.set_content_length(v); } if let Some(v) = parse_content_type(resp.headers())? { m.set_content_type(v); } if let Some(v) = parse_etag(resp.headers())? { m.set_etag(v); if v.starts_with("\"DirIndex") { m.set_mode(EntryMode::DIR); } else { m.set_mode(EntryMode::FILE); } } else { // Some service will stream the output of DirIndex. // If we don't have an etag, it's highly to be a dir. m.set_mode(EntryMode::DIR); } if let Some(v) = parse_content_disposition(resp.headers())? { m.set_content_disposition(v); } Ok(RpStat::new(m)) } StatusCode::FOUND | StatusCode::MOVED_PERMANENTLY => { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.ipfs_get(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn list(&self, path: &str, _: OpList) -> Result<(RpList, Self::Lister)> { let l = DirStream::new(Arc::new(self.clone()), path); Ok((RpList::default(), oio::PageLister::new(l))) } } impl IpfsBackend { pub async fn ipfs_get(&self, path: &str, range: BytesRange) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!("{}{}", self.endpoint, percent_encode_path(&p)); let mut req = Request::get(&url); if !range.is_full() { req = req.header(http::header::RANGE, range.to_header()); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } async fn ipfs_head(&self, path: &str) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!("{}{}", self.endpoint, percent_encode_path(&p)); let req = Request::head(&url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } async fn ipfs_list(&self, path: &str) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!("{}{}", self.endpoint, percent_encode_path(&p)); let mut req = Request::get(&url); // Use "application/vnd.ipld.raw" to disable IPLD codec deserialization // OpenDAL will parse ipld data directly. // // ref: https://github.com/ipfs/specs/blob/main/http-gateways/PATH_GATEWAY.md req = req.header(http::header::ACCEPT, "application/vnd.ipld.raw"); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } } pub struct DirStream { backend: Arc, path: String, } impl DirStream { fn new(backend: Arc, path: &str) -> Self { Self { backend, path: path.to_string(), } } } impl oio::PageList for DirStream { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self.backend.ipfs_list(&self.path).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let pb_node = PBNode::decode(bs).map_err(|e| { Error::new(ErrorKind::Unexpected, "deserialize protobuf from response").set_source(e) })?; let names = pb_node .links .into_iter() .map(|v| v.name.unwrap()) .collect::>(); for mut name in names { let meta = self .backend .stat(&name, OpStat::new()) .await? .into_metadata(); if meta.mode().is_dir() { name += "/"; } ctx.entries.push_back(oio::Entry::new(&name, meta)) } ctx.done = true; Ok(()) } } opendal-0.52.0/src/services/ipfs/config.rs000064400000000000000000000022101046102023000165130ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// Config for IPFS file system support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct IpfsConfig { /// IPFS gateway endpoint. pub endpoint: Option, /// IPFS root. pub root: Option, } opendal-0.52.0/src/services/ipfs/docs.md000064400000000000000000000015321046102023000161600ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [ ] ~~write~~ - [ ] ~~create_dir~~ - [ ] ~~delete~~ - [ ] ~~copy~~ - [ ] ~~rename~~ - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `endpoint`: Customizable endpoint setting You can refer to [`IpfsBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Ipfs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Ipfs::default() // set the endpoint for OpenDAL .endpoint("https://ipfs.io") // set the root for OpenDAL .root("/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/ipfs/error.rs000064400000000000000000000034371046102023000164130ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use crate::raw::*; use crate::*; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT // IPFS Gateway will return `408 REQUEST_TIMEOUT` while `ipfs resolve -r` failed. | StatusCode::REQUEST_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = String::from_utf8_lossy(&bs); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/ipfs/ipld.rs000064400000000000000000000200101046102023000161740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. /// Ref: #[derive(Clone, PartialEq, Eq, prost::Message)] pub struct PBNode { #[prost(bytes = "vec", optional, tag = "1")] pub data: Option>, #[prost(message, repeated, tag = "2")] pub links: Vec, } /// Ref: #[derive(Clone, PartialEq, Eq, prost::Message)] pub struct PBLink { #[prost(bytes = "vec", optional, tag = "1")] pub hash: Option>, #[prost(string, optional, tag = "2")] pub name: Option, #[prost(uint64, optional, tag = "3")] pub tsize: Option, } /// This type is generated by [prost_build](https://docs.rs/prost-build/latest/prost_build/) via proto file `https://github.com/ipfs/go-unixfs/raw/master/pb/unixfs.proto`. // /// No modification has been and will be made from OpenDAL. #[derive(Clone, PartialEq, Eq, prost::Message)] pub struct Data { #[prost(enumeration = "data::DataType", required, tag = "1")] pub r#type: i32, #[prost(bytes = "vec", optional, tag = "2")] pub data: Option>, #[prost(uint64, optional, tag = "3")] pub filesize: Option, #[prost(uint64, repeated, packed = "false", tag = "4")] pub blocksizes: Vec, #[prost(uint64, optional, tag = "5")] pub hash_type: Option, #[prost(uint64, optional, tag = "6")] pub fanout: Option, } /// Nested message and enum types in `Data`. pub mod data { #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, prost::Enumeration)] #[repr(i32)] pub enum DataType { Raw = 0, Directory = 1, File = 2, Metadata = 3, Symlink = 4, HamtShard = 5, } impl DataType { /// String value of the enum field names used in the ProtoBuf definition. /// /// The values are not transformed in any way and thus are considered stable /// (if the ProtoBuf definition does not change) and safe for programmatic use. pub fn as_str_name(&self) -> &'static str { match self { DataType::Raw => "Raw", DataType::Directory => "Directory", DataType::File => "File", DataType::Metadata => "Metadata", DataType::Symlink => "Symlink", DataType::HamtShard => "HAMTShard", } } } } #[derive(Clone, PartialEq, Eq, prost::Message)] pub struct Metadata { #[prost(string, optional, tag = "1")] pub mime_type: Option, } #[cfg(test)] mod tests { use bytes::Bytes; use data::DataType; use prost::Message; use super::*; /// Content is generated from `https://ipfs.io/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ` /// with `accept: application/vnd.ipld.raw` #[test] fn test_message() { let bs: Vec = vec![ 0o022, 0o062, 0o012, 0o042, 0o022, 0o040, 0o220, 0o124, 0o225, 0o170, 0o152, 0o036, 0o040, 0o072, 0o032, 0o053, 0o143, 0o274, 0o044, 0o366, 0o215, 0o306, 0o177, 0o041, 0o260, 0o310, 0o231, 0o271, 0o142, 0o163, 0o006, 0o150, 0o054, 0o315, 0o227, 0o375, 0o347, 0o143, 0o022, 0o012, 0o156, 0o157, 0o162, 0o155, 0o141, 0o154, 0o137, 0o144, 0o151, 0o162, 0o030, 0o074, 0o022, 0o065, 0o012, 0o042, 0o022, 0o040, 0o337, 0o200, 0o021, 0o267, 0o364, 0o242, 0o030, 0o145, 0o257, 0o150, 0o205, 0o335, 0o100, 0o173, 0o222, 0o053, 0o074, 0o051, 0o041, 0o020, 0o203, 0o265, 0o116, 0o223, 0o111, 0o145, 0o040, 0o114, 0o350, 0o143, 0o354, 0o002, 0o022, 0o013, 0o156, 0o157, 0o162, 0o155, 0o141, 0o154, 0o137, 0o146, 0o151, 0o154, 0o145, 0o030, 0o216, 0o200, 0o020, 0o022, 0o064, 0o012, 0o042, 0o022, 0o040, 0o051, 0o305, 0o176, 0o211, 0o142, 0o044, 0o020, 0o267, 0o344, 0o172, 0o166, 0o374, 0o043, 0o010, 0o354, 0o047, 0o061, 0o031, 0o021, 0o121, 0o367, 0o014, 0o003, 0o002, 0o343, 0o032, 0o250, 0o353, 0o316, 0o263, 0o224, 0o142, 0o022, 0o012, 0o157, 0o156, 0o164, 0o151, 0o155, 0o145, 0o056, 0o143, 0o163, 0o166, 0o030, 0o215, 0o307, 0o005, 0o022, 0o067, 0o012, 0o042, 0o022, 0o040, 0o215, 0o225, 0o000, 0o035, 0o077, 0o302, 0o322, 0o271, 0o150, 0o077, 0o364, 0o202, 0o062, 0o144, 0o104, 0o074, 0o327, 0o111, 0o131, 0o233, 0o176, 0o144, 0o033, 0o076, 0o266, 0o144, 0o203, 0o256, 0o107, 0o257, 0o161, 0o112, 0o022, 0o016, 0o157, 0o156, 0o164, 0o151, 0o155, 0o145, 0o056, 0o143, 0o163, 0o166, 0o056, 0o142, 0o172, 0o062, 0o030, 0o206, 0o065, 0o022, 0o066, 0o012, 0o042, 0o022, 0o040, 0o205, 0o065, 0o360, 0o241, 0o222, 0o377, 0o267, 0o213, 0o334, 0o057, 0o060, 0o130, 0o230, 0o154, 0o213, 0o260, 0o123, 0o143, 0o011, 0o055, 0o365, 0o103, 0o002, 0o332, 0o213, 0o150, 0o275, 0o164, 0o162, 0o223, 0o350, 0o117, 0o022, 0o015, 0o157, 0o156, 0o164, 0o151, 0o155, 0o145, 0o056, 0o143, 0o163, 0o166, 0o056, 0o147, 0o172, 0o030, 0o217, 0o101, 0o022, 0o067, 0o012, 0o042, 0o022, 0o040, 0o202, 0o041, 0o276, 0o325, 0o105, 0o143, 0o237, 0o357, 0o121, 0o152, 0o300, 0o112, 0o067, 0o205, 0o022, 0o226, 0o021, 0o015, 0o302, 0o061, 0o135, 0o225, 0o320, 0o123, 0o030, 0o101, 0o007, 0o367, 0o157, 0o273, 0o154, 0o306, 0o022, 0o016, 0o157, 0o156, 0o164, 0o151, 0o155, 0o145, 0o056, 0o143, 0o163, 0o166, 0o056, 0o172, 0o163, 0o164, 0o030, 0o251, 0o103, 0o022, 0o111, 0o012, 0o042, 0o022, 0o040, 0o220, 0o124, 0o225, 0o170, 0o152, 0o036, 0o040, 0o072, 0o032, 0o053, 0o143, 0o274, 0o044, 0o366, 0o215, 0o306, 0o177, 0o041, 0o260, 0o310, 0o231, 0o271, 0o142, 0o163, 0o006, 0o150, 0o054, 0o315, 0o227, 0o375, 0o347, 0o143, 0o022, 0o041, 0o163, 0o160, 0o145, 0o143, 0o151, 0o141, 0o154, 0o137, 0o144, 0o151, 0o162, 0o040, 0o040, 0o041, 0o100, 0o043, 0o044, 0o045, 0o136, 0o046, 0o052, 0o050, 0o051, 0o137, 0o053, 0o055, 0o075, 0o073, 0o047, 0o076, 0o074, 0o054, 0o077, 0o030, 0o074, 0o022, 0o114, 0o012, 0o042, 0o022, 0o040, 0o337, 0o200, 0o021, 0o267, 0o364, 0o242, 0o030, 0o145, 0o257, 0o150, 0o205, 0o335, 0o100, 0o173, 0o222, 0o053, 0o074, 0o051, 0o041, 0o020, 0o203, 0o265, 0o116, 0o223, 0o111, 0o145, 0o040, 0o114, 0o350, 0o143, 0o354, 0o002, 0o022, 0o042, 0o163, 0o160, 0o145, 0o143, 0o151, 0o141, 0o154, 0o137, 0o146, 0o151, 0o154, 0o145, 0o040, 0o040, 0o041, 0o100, 0o043, 0o044, 0o045, 0o136, 0o046, 0o052, 0o050, 0o051, 0o137, 0o053, 0o055, 0o075, 0o073, 0o047, 0o076, 0o074, 0o054, 0o077, 0o030, 0o216, 0o200, 0o020, 0o012, 0o002, 0o010, 0o001, ]; let data = PBNode::decode(Bytes::from(bs)).expect("decode must succeed"); if let Some(bs) = data.data.clone() { let d = Data::decode(Bytes::from(bs)).expect("decode must succeed"); assert_eq!(d.r#type, DataType::Directory as i32); } assert_eq!(data.links.len(), 8); assert_eq!(data.links[0].name.as_ref().unwrap(), "normal_dir"); assert_eq!(data.links[0].tsize.unwrap(), 60); assert_eq!(data.links[1].name.as_ref().unwrap(), "normal_file"); assert_eq!(data.links[1].tsize.unwrap(), 262158); } } opendal-0.52.0/src/services/ipfs/mod.rs000064400000000000000000000020341046102023000160310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-ipfs")] mod error; #[cfg(feature = "services-ipfs")] mod ipld; #[cfg(feature = "services-ipfs")] mod backend; #[cfg(feature = "services-ipfs")] pub use backend::IpfsBuilder as Ipfs; mod config; pub use config::IpfsConfig; opendal-0.52.0/src/services/ipmfs/backend.rs000064400000000000000000000204561046102023000170260ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt; use std::fmt::Write; use std::str; use std::sync::Arc; use bytes::Buf; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use super::delete::IpmfsDeleter; use super::error::parse_error; use super::lister::IpmfsLister; use super::writer::IpmfsWriter; use crate::raw::*; use crate::*; /// IPFS Mutable File System (IPMFS) backend. #[doc = include_str!("docs.md")] #[derive(Clone)] pub struct IpmfsBackend { root: String, endpoint: String, client: HttpClient, } impl fmt::Debug for IpmfsBackend { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("endpoint", &self.endpoint) .finish() } } impl IpmfsBackend { pub(crate) fn new(root: String, client: HttpClient, endpoint: String) -> Self { Self { root, client, endpoint, } } } impl Access for IpmfsBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Ipmfs) .set_root(&self.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, read: true, write: true, delete: true, list: true, list_has_content_length: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let resp = self.ipmfs_mkdir(path).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(RpCreateDir::default()), _ => Err(parse_error(resp)), } } async fn stat(&self, path: &str, _: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let resp = self.ipmfs_stat(path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let res: IpfsStatResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let mode = match res.file_type.as_str() { "file" => EntryMode::FILE, "directory" => EntryMode::DIR, _ => EntryMode::Unknown, }; let mut meta = Metadata::new(mode); meta.set_content_length(res.size); Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.ipmfs_read(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, _: OpWrite) -> Result<(RpWrite, Self::Writer)> { Ok(( RpWrite::default(), oio::OneShotWriter::new(IpmfsWriter::new(self.clone(), path.to_string())), )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(IpmfsDeleter::new(Arc::new(self.clone()))), )) } async fn list(&self, path: &str, _: OpList) -> Result<(RpList, Self::Lister)> { let l = IpmfsLister::new(Arc::new(self.clone()), &self.root, path); Ok((RpList::default(), oio::PageLister::new(l))) } } impl IpmfsBackend { async fn ipmfs_stat(&self, path: &str) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!( "{}/api/v0/files/stat?arg={}", self.endpoint, percent_encode_path(&p) ); let req = Request::post(url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn ipmfs_read(&self, path: &str, range: BytesRange) -> Result> { let p = build_rooted_abs_path(&self.root, path); let mut url = format!( "{}/api/v0/files/read?arg={}", self.endpoint, percent_encode_path(&p) ); write!(url, "&offset={}", range.offset()).expect("write into string must succeed"); if let Some(count) = range.size() { write!(url, "&count={count}").expect("write into string must succeed") } let req = Request::post(url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn ipmfs_rm(&self, path: &str) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!( "{}/api/v0/files/rm?arg={}", self.endpoint, percent_encode_path(&p) ); let req = Request::post(url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub(crate) async fn ipmfs_ls(&self, path: &str) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!( "{}/api/v0/files/ls?arg={}&long=true", self.endpoint, percent_encode_path(&p) ); let req = Request::post(url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } async fn ipmfs_mkdir(&self, path: &str) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!( "{}/api/v0/files/mkdir?arg={}&parents=true", self.endpoint, percent_encode_path(&p) ); let req = Request::post(url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } /// Support write from reader. pub async fn ipmfs_write(&self, path: &str, body: Buffer) -> Result> { let p = build_rooted_abs_path(&self.root, path); let url = format!( "{}/api/v0/files/write?arg={}&parents=true&create=true&truncate=true", self.endpoint, percent_encode_path(&p) ); let multipart = Multipart::new().part(FormDataPart::new("data").content(body)); let req: http::request::Builder = Request::post(url); let req = multipart.apply(req)?; self.client.send(req).await } } #[derive(Deserialize, Default, Debug)] #[serde(default)] struct IpfsStatResponse { #[serde(rename = "Size")] size: u64, #[serde(rename = "Type")] file_type: String, } opendal-0.52.0/src/services/ipmfs/builder.rs000064400000000000000000000073421046102023000170640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use log::debug; use super::backend::IpmfsBackend; use crate::raw::*; use crate::services::IpmfsConfig; use crate::*; impl Configurator for IpmfsConfig { type Builder = IpmfsBuilder; fn into_builder(self) -> Self::Builder { IpmfsBuilder { config: self, http_client: None, } } } /// IPFS file system support based on [IPFS MFS](https://docs.ipfs.tech/concepts/file-systems/) API. /// /// # Capabilities /// /// This service can be used to: /// /// - [x] read /// - [x] write /// - [x] list /// - [ ] presign /// - [ ] blocking /// /// # Configuration /// /// - `root`: Set the work directory for backend /// - `endpoint`: Customizable endpoint setting /// /// You can refer to [`IpmfsBuilder`]'s docs for more information /// /// # Example /// /// ## Via Builder /// /// ```no_run /// use anyhow::Result; /// use opendal::services::Ipmfs; /// use opendal::Operator; /// /// #[tokio::main] /// async fn main() -> Result<()> { /// // create backend builder /// let mut builder = Ipmfs::default() /// // set the storage bucket for OpenDAL /// .endpoint("http://127.0.0.1:5001"); /// /// let op: Operator = Operator::new(builder)?.finish(); /// /// Ok(()) /// } /// ``` #[derive(Default, Debug)] pub struct IpmfsBuilder { config: IpmfsConfig, http_client: Option, } impl IpmfsBuilder { /// Set root for ipfs. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set endpoint for ipfs. /// /// Default: http://localhost:5001 pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.to_string()) }; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for IpmfsBuilder { const SCHEME: Scheme = Scheme::Ipmfs; type Config = IpmfsConfig; fn build(self) -> Result { let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let endpoint = self .config .endpoint .clone() .unwrap_or_else(|| "http://localhost:5001".to_string()); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Ipmfs) })? }; Ok(IpmfsBackend::new(root, client, endpoint)) } } opendal-0.52.0/src/services/ipmfs/config.rs000064400000000000000000000022011046102023000166700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// Config for IPFS MFS support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct IpmfsConfig { /// Root for ipfs. pub root: Option, /// Endpoint for ipfs. pub endpoint: Option, } opendal-0.52.0/src/services/ipmfs/delete.rs000064400000000000000000000026271046102023000167010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::backend::IpmfsBackend; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct IpmfsDeleter { core: Arc, } impl IpmfsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for IpmfsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.ipmfs_rm(&path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/ipmfs/docs.md000064400000000000000000000002571046102023000163400ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [ ] create_dir - [x] delete - [ ] copy - [ ] rename - [x] list - [ ] presign - [ ] blocking opendal-0.52.0/src/services/ipmfs/error.rs000064400000000000000000000052431046102023000165650ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use serde::Deserialize; use serde_json::de; use crate::raw::*; use crate::*; #[derive(Deserialize, Default, Debug)] #[serde(default)] struct IpfsError { #[serde(rename = "Message")] message: String, #[serde(rename = "Code")] code: usize, #[serde(rename = "Type")] ty: String, } /// Parse error response into io::Error. /// /// > Status code 500 means that the function does exist, but IPFS was not /// > able to fulfil the request because of an error. /// > To know that reason, you have to look at the error message that is /// > usually returned with the body of the response /// > (if no error, check the daemon logs). /// /// ref: https://docs.ipfs.tech/reference/kubo/rpc/#http-status-codes pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let ipfs_error = de::from_slice::(&bs).ok(); let (kind, retryable) = match parts.status { StatusCode::INTERNAL_SERVER_ERROR => { if let Some(ie) = &ipfs_error { match ie.message.as_str() { "file does not exist" => (ErrorKind::NotFound, false), _ => (ErrorKind::Unexpected, false), } } else { (ErrorKind::Unexpected, false) } } StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => { (ErrorKind::Unexpected, true) } _ => (ErrorKind::Unexpected, false), }; let message = match ipfs_error { Some(ipfs_error) => format!("{ipfs_error:?}"), None => String::from_utf8_lossy(&bs).into_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/ipmfs/lister.rs000064400000000000000000000064551046102023000167440ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use serde::Deserialize; use super::backend::IpmfsBackend; use super::error::parse_error; use crate::raw::*; use crate::EntryMode; use crate::Metadata; use crate::Result; pub struct IpmfsLister { backend: Arc, root: String, path: String, } impl IpmfsLister { pub fn new(backend: Arc, root: &str, path: &str) -> Self { Self { backend, root: root.to_string(), path: path.to_string(), } } } impl oio::PageList for IpmfsLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self.backend.ipmfs_ls(&self.path).await?; if resp.status() != StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let entries_body: IpfsLsResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; // Mark dir stream has been consumed. ctx.done = true; for object in entries_body.entries.unwrap_or_default() { let path = match object.mode() { EntryMode::FILE => format!("{}{}", &self.path, object.name), EntryMode::DIR => format!("{}{}/", &self.path, object.name), EntryMode::Unknown => unreachable!(), }; let path = build_rel_path(&self.root, &path); ctx.entries.push_back(oio::Entry::new( &path, Metadata::new(object.mode()).with_content_length(object.size), )); } Ok(()) } } #[derive(Deserialize, Default, Debug)] #[serde(default)] struct IpfsLsResponseEntry { #[serde(rename = "Name")] name: String, #[serde(rename = "Type")] file_type: i64, #[serde(rename = "Size")] size: u64, } impl IpfsLsResponseEntry { /// ref: /// /// ```protobuf /// enum DataType { /// Raw = 0; /// Directory = 1; /// File = 2; /// Metadata = 3; /// Symlink = 4; /// HAMTShard = 5; /// } /// ``` fn mode(&self) -> EntryMode { match &self.file_type { 1 => EntryMode::DIR, 0 | 2 => EntryMode::FILE, _ => EntryMode::Unknown, } } } #[derive(Deserialize, Default, Debug)] #[serde(default)] struct IpfsLsResponse { #[serde(rename = "Entries")] entries: Option>, } opendal-0.52.0/src/services/ipmfs/mod.rs000064400000000000000000000022631046102023000162120ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-ipmfs")] mod backend; #[cfg(feature = "services-ipmfs")] mod delete; #[cfg(feature = "services-ipmfs")] mod error; #[cfg(feature = "services-ipmfs")] mod lister; #[cfg(feature = "services-ipmfs")] mod writer; #[cfg(feature = "services-ipmfs")] mod builder; #[cfg(feature = "services-ipmfs")] pub use builder::IpmfsBuilder as Ipmfs; mod config; pub use config::IpmfsConfig; opendal-0.52.0/src/services/ipmfs/writer.rs000064400000000000000000000027311046102023000167470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::StatusCode; use super::backend::IpmfsBackend; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct IpmfsWriter { backend: IpmfsBackend, path: String, } impl IpmfsWriter { pub fn new(backend: IpmfsBackend, path: String) -> Self { IpmfsWriter { backend, path } } } impl oio::OneShotWrite for IpmfsWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self.backend.ipmfs_write(&self.path, bs).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/koofr/backend.rs000064400000000000000000000247271046102023000170350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::Response; use http::StatusCode; use log::debug; use tokio::sync::Mutex; use tokio::sync::OnceCell; use super::core::File; use super::core::KoofrCore; use super::core::KoofrSigner; use super::delete::KoofrDeleter; use super::error::parse_error; use super::lister::KoofrLister; use super::writer::KoofrWriter; use super::writer::KoofrWriters; use crate::raw::*; use crate::services::KoofrConfig; use crate::*; impl Configurator for KoofrConfig { type Builder = KoofrBuilder; fn into_builder(self) -> Self::Builder { KoofrBuilder { config: self, http_client: None, } } } /// [Koofr](https://app.koofr.net/) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct KoofrBuilder { config: KoofrConfig, http_client: Option, } impl Debug for KoofrBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("KoofrBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl KoofrBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// endpoint. /// /// It is required. e.g. `https://api.koofr.net/` pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = endpoint.to_string(); self } /// email. /// /// It is required. e.g. `test@example.com` pub fn email(mut self, email: &str) -> Self { self.config.email = email.to_string(); self } /// Koofr application password. /// /// Go to . /// Click "Generate Password" button to generate a new application password. /// /// # Notes /// /// This is not user's Koofr account password. /// Please use the application password instead. /// Please also remind users of this. pub fn password(mut self, password: &str) -> Self { self.config.password = if password.is_empty() { None } else { Some(password.to_string()) }; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for KoofrBuilder { const SCHEME: Scheme = Scheme::Koofr; type Config = KoofrConfig; /// Builds the backend and returns the result of KoofrBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); if self.config.endpoint.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Koofr)); } debug!("backend use endpoint {}", &self.config.endpoint); if self.config.email.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "email is empty") .with_operation("Builder::build") .with_context("service", Scheme::Koofr)); } debug!("backend use email {}", &self.config.email); let password = match &self.config.password { Some(password) => Ok(password.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty") .with_operation("Builder::build") .with_context("service", Scheme::Koofr)), }?; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Koofr) })? }; let signer = Arc::new(Mutex::new(KoofrSigner::default())); Ok(KoofrBackend { core: Arc::new(KoofrCore { root, endpoint: self.config.endpoint.clone(), email: self.config.email.clone(), password, mount_id: OnceCell::new(), signer, client, }), }) } } /// Backend for Koofr services. #[derive(Debug, Clone)] pub struct KoofrBackend { core: Arc, } impl Access for KoofrBackend { type Reader = HttpBody; type Writer = KoofrWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Koofr) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_last_modified: true, create_dir: true, read: true, write: true, write_can_empty: true, delete: true, rename: true, copy: true, list: true, list_has_content_length: true, list_has_content_type: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { self.core.ensure_dir_exists(path).await?; self.core .create_dir(&build_abs_path(&self.core.root, path)) .await?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _args: OpStat) -> Result { let path = build_rooted_abs_path(&self.core.root, path); let resp = self.core.info(&path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let file: File = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let mode = if file.ty == "dir" { EntryMode::DIR } else { EntryMode::FILE }; let mut md = Metadata::new(mode); md.set_content_length(file.size) .set_content_type(&file.content_type) .set_last_modified(parse_datetime_from_from_timestamp_millis(file.modified)?); Ok(RpStat::new(md)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.get(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, _args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = KoofrWriter::new(self.core.clone(), path.to_string()); let w = oio::OneShotWriter::new(writer); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(KoofrDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, _args: OpList) -> Result<(RpList, Self::Lister)> { let l = KoofrLister::new(self.core.clone(), path); Ok((RpList::default(), oio::PageLister::new(l))) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { self.core.ensure_dir_exists(to).await?; if from == to { return Ok(RpCopy::default()); } let resp = self.core.remove(to).await?; let status = resp.status(); if status != StatusCode::OK && status != StatusCode::NOT_FOUND { return Err(parse_error(resp)); } let resp = self.core.copy(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { self.core.ensure_dir_exists(to).await?; if from == to { return Ok(RpRename::default()); } let resp = self.core.remove(to).await?; let status = resp.status(); if status != StatusCode::OK && status != StatusCode::NOT_FOUND { return Err(parse_error(resp)); } let resp = self.core.move_object(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpRename::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/koofr/config.rs000064400000000000000000000031341046102023000167000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Koofr services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct KoofrConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, /// Koofr endpoint. pub endpoint: String, /// Koofr email. pub email: String, /// password of this backend. (Must be the application password) pub password: Option, } impl Debug for KoofrConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Config"); ds.field("root", &self.root); ds.field("email", &self.email); ds.finish() } } opendal-0.52.0/src/services/koofr/core.rs000064400000000000000000000301501046102023000163610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::VecDeque; use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use bytes::Bytes; use http::header; use http::request; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use serde_json::json; use tokio::sync::Mutex; use tokio::sync::OnceCell; use super::error::parse_error; use crate::raw::*; use crate::*; #[derive(Clone)] pub struct KoofrCore { /// The root of this core. pub root: String, /// The endpoint of this backend. pub endpoint: String, /// Koofr email pub email: String, /// Koofr password pub password: String, /// signer of this backend. pub signer: Arc>, // Koofr mount_id. pub mount_id: OnceCell, pub client: HttpClient, } impl Debug for KoofrCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("email", &self.email) .finish_non_exhaustive() } } impl KoofrCore { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } pub async fn get_mount_id(&self) -> Result<&String> { self.mount_id .get_or_try_init(|| async { let req = Request::get(format!("{}/api/v2/mounts", self.endpoint)); let req = self.sign(req).await?; let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let resp: MountsResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; for mount in resp.mounts { if mount.is_primary { return Ok(mount.id); } } Err(Error::new(ErrorKind::Unexpected, "No primary mount found")) }) .await } pub async fn sign(&self, req: request::Builder) -> Result { let mut signer = self.signer.lock().await; if !signer.token.is_empty() { return Ok(req.header( header::AUTHORIZATION, format!("Token token={}", signer.token), )); } let url = format!("{}/token", self.endpoint); let body = json!({ "email": self.email, "password": self.password, }); let bs = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let auth_req = Request::post(url) .header(header::CONTENT_TYPE, "application/json") .body(Buffer::from(Bytes::from(bs))) .map_err(new_request_build_error)?; let resp = self.client.send(auth_req).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let resp: TokenResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; signer.token = resp.token; Ok(req.header( header::AUTHORIZATION, format!("Token token={}", signer.token), )) } } impl KoofrCore { pub async fn ensure_dir_exists(&self, path: &str) -> Result<()> { let mut dirs = VecDeque::default(); let mut p = build_abs_path(&self.root, path); while p != "/" { let parent = get_parent(&p).to_string(); dirs.push_front(parent.clone()); p = parent; } for dir in dirs { self.create_dir(&dir).await?; } Ok(()) } pub async fn create_dir(&self, path: &str) -> Result<()> { let resp = self.info(path).await?; let status = resp.status(); match status { StatusCode::NOT_FOUND => { let name = get_basename(path).trim_end_matches('/'); let parent = get_parent(path); let mount_id = self.get_mount_id().await?; let url = format!( "{}/api/v2/mounts/{}/files/folder?path={}", self.endpoint, mount_id, percent_encode_path(parent) ); let body = json!({ "name": name }); let bs = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let req = Request::post(url); let req = self.sign(req).await?; let req = req .header(header::CONTENT_TYPE, "application/json") .body(Buffer::from(Bytes::from(bs))) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { // When the directory already exists, Koofr returns 400 Bad Request. // We should treat it as success. StatusCode::OK | StatusCode::CREATED | StatusCode::BAD_REQUEST => Ok(()), _ => Err(parse_error(resp)), } } StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } pub async fn info(&self, path: &str) -> Result> { let mount_id = self.get_mount_id().await?; let url = format!( "{}/api/v2/mounts/{}/files/info?path={}", self.endpoint, mount_id, percent_encode_path(path) ); let req = Request::get(url); let req = self.sign(req).await?; let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn get(&self, path: &str, range: BytesRange) -> Result> { let path = build_rooted_abs_path(&self.root, path); let mount_id = self.get_mount_id().await?; let url = format!( "{}/api/v2/mounts/{}/files/get?path={}", self.endpoint, mount_id, percent_encode_path(&path) ); let req = Request::get(url).header(header::RANGE, range.to_header()); let req = self.sign(req).await?; let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn put(&self, path: &str, bs: Buffer) -> Result> { let path = build_rooted_abs_path(&self.root, path); let filename = get_basename(&path); let parent = get_parent(&path); let mount_id = self.get_mount_id().await?; let url = format!( "{}/content/api/v2/mounts/{}/files/put?path={}&filename={}&info=true&overwriteIgnoreNonexisting=&autorename=false&overwrite=true", self.endpoint, mount_id, percent_encode_path(parent), percent_encode_path(filename) ); let file_part = FormDataPart::new("file") .header( header::CONTENT_DISPOSITION, format!("form-data; name=\"file\"; filename=\"{filename}\"") .parse() .unwrap(), ) .content(bs); let multipart = Multipart::new().part(file_part); let req = Request::post(url); let req = self.sign(req).await?; let req = multipart.apply(req)?; self.send(req).await } pub async fn remove(&self, path: &str) -> Result> { let path = build_rooted_abs_path(&self.root, path); let mount_id = self.get_mount_id().await?; let url = format!( "{}/api/v2/mounts/{}/files/remove?path={}", self.endpoint, mount_id, percent_encode_path(&path) ); let req = Request::delete(url); let req = self.sign(req).await?; let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn copy(&self, from: &str, to: &str) -> Result> { let from = build_rooted_abs_path(&self.root, from); let to = build_rooted_abs_path(&self.root, to); let mount_id = self.get_mount_id().await?; let url = format!( "{}/api/v2/mounts/{}/files/copy?path={}", self.endpoint, mount_id, percent_encode_path(&from), ); let body = json!({ "toMountId": mount_id, "toPath": to, }); let bs = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let req = Request::put(url); let req = self.sign(req).await?; let req = req .header(header::CONTENT_TYPE, "application/json") .body(Buffer::from(Bytes::from(bs))) .map_err(new_request_build_error)?; self.send(req).await } pub async fn move_object(&self, from: &str, to: &str) -> Result> { let from = build_rooted_abs_path(&self.root, from); let to = build_rooted_abs_path(&self.root, to); let mount_id = self.get_mount_id().await?; let url = format!( "{}/api/v2/mounts/{}/files/move?path={}", self.endpoint, mount_id, percent_encode_path(&from), ); let body = json!({ "toMountId": mount_id, "toPath": to, }); let bs = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let req = Request::put(url); let req = self.sign(req).await?; let req = req .header(header::CONTENT_TYPE, "application/json") .body(Buffer::from(Bytes::from(bs))) .map_err(new_request_build_error)?; self.send(req).await } pub async fn list(&self, path: &str) -> Result> { let path = build_rooted_abs_path(&self.root, path); let mount_id = self.get_mount_id().await?; let url = format!( "{}/api/v2/mounts/{}/files/list?path={}", self.endpoint, mount_id, percent_encode_path(&path) ); let req = Request::get(url); let req = self.sign(req).await?; let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } } #[derive(Clone, Default)] pub struct KoofrSigner { pub token: String, } #[derive(Debug, Deserialize)] pub struct TokenResponse { pub token: String, } #[derive(Debug, Deserialize)] pub struct MountsResponse { pub mounts: Vec, } #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct Mount { pub id: String, pub is_primary: bool, } #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct ListResponse { pub files: Vec, } #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub struct File { pub name: String, #[serde(rename = "type")] pub ty: String, pub size: u64, pub modified: i64, pub content_type: String, } opendal-0.52.0/src/services/koofr/delete.rs000064400000000000000000000027531046102023000167030ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct KoofrDeleter { core: Arc, } impl KoofrDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for KoofrDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.remove(&path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), // Allow 404 when deleting a non-existing object StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/koofr/docs.md000064400000000000000000000017331046102023000163420ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `endpoint`: Koofr endpoint - `email` Koofr email - `password` Koofr password You can refer to [`KoofrBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Koofr; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Koofr::default() // set the storage bucket for OpenDAL .root("/") // set the bucket for OpenDAL .endpoint("https://api.koofr.net/") // set the email for OpenDAL .email("me@example.com") // set the password for OpenDAL .password("xxx xxx xxx xxx"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/koofr/error.rs000064400000000000000000000043351046102023000165700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use crate::raw::*; use crate::*; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status.as_u16() { 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), 304 | 412 => (ErrorKind::ConditionNotMatch, false), // Service like Koofr could return 499 error with a message like: // Client Disconnect, we should retry it. 499 => (ErrorKind::Unexpected, true), 500 | 502 | 503 | 504 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = String::from_utf8_lossy(&bs).into_owned(); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[tokio::test] async fn test_parse_error() { let err_res = vec![(r#""#, ErrorKind::NotFound, StatusCode::NOT_FOUND)]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/koofr/lister.rs000064400000000000000000000051001046102023000167300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use super::core::KoofrCore; use super::core::ListResponse; use super::error::parse_error; use crate::raw::oio::Entry; use crate::raw::*; use crate::EntryMode; use crate::Metadata; use crate::Result; pub struct KoofrLister { core: Arc, path: String, } impl KoofrLister { pub(super) fn new(core: Arc, path: &str) -> Self { KoofrLister { core, path: path.to_string(), } } } impl oio::PageList for KoofrLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self.core.list(&self.path).await?; if resp.status() == http::StatusCode::NOT_FOUND { ctx.done = true; return Ok(()); } match resp.status() { http::StatusCode::OK => {} _ => { return Err(parse_error(resp)); } } let bs = resp.into_body(); let response: ListResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; for file in response.files { let path = build_abs_path(&normalize_root(&self.path), &file.name); let entry = if file.ty == "dir" { let path = format!("{}/", path); Entry::new(&path, Metadata::new(EntryMode::DIR)) } else { let m = Metadata::new(EntryMode::FILE) .with_content_length(file.size) .with_content_type(file.content_type) .with_last_modified(parse_datetime_from_from_timestamp_millis(file.modified)?); Entry::new(&path, m) }; ctx.entries.push_back(entry); } ctx.done = true; Ok(()) } } opendal-0.52.0/src/services/koofr/mod.rs000064400000000000000000000022601046102023000162110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-koofr")] mod core; #[cfg(feature = "services-koofr")] mod delete; #[cfg(feature = "services-koofr")] mod error; #[cfg(feature = "services-koofr")] mod lister; #[cfg(feature = "services-koofr")] mod writer; #[cfg(feature = "services-koofr")] mod backend; #[cfg(feature = "services-koofr")] pub use backend::KoofrBuilder as Koofr; mod config; pub use config::KoofrConfig; opendal-0.52.0/src/services/koofr/writer.rs000064400000000000000000000031121046102023000167430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::KoofrCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type KoofrWriters = oio::OneShotWriter; pub struct KoofrWriter { core: Arc, path: String, } impl KoofrWriter { pub fn new(core: Arc, path: String) -> Self { KoofrWriter { core, path } } } impl oio::OneShotWrite for KoofrWriter { async fn write_once(&self, bs: Buffer) -> Result { self.core.ensure_dir_exists(&self.path).await?; let resp = self.core.put(&self.path, bs).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/lakefs/backend.rs000064400000000000000000000233401046102023000171500ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use chrono::TimeZone; use chrono::Utc; use http::Response; use http::StatusCode; use log::debug; use super::core::LakefsCore; use super::core::LakefsStatus; use super::delete::LakefsDeleter; use super::error::parse_error; use super::lister::LakefsLister; use super::writer::LakefsWriter; use crate::raw::*; use crate::services::LakefsConfig; use crate::*; impl Configurator for LakefsConfig { type Builder = LakefsBuilder; fn into_builder(self) -> Self::Builder { LakefsBuilder { config: self } } } /// [Lakefs](https://docs.lakefs.io/reference/api.html#/)'s API support. #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct LakefsBuilder { config: LakefsConfig, } impl Debug for LakefsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Builder"); ds.field("config", &self.config); ds.finish() } } impl LakefsBuilder { /// Set the endpoint of this backend. /// /// endpoint must be full uri. /// /// This is required. /// - `http://127.0.0.1:8000` (lakefs daemon in local) /// - `https://my-lakefs.example.com` (lakefs server) pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { self.config.endpoint = Some(endpoint.to_string()); } self } /// Set username of this backend. This is required. pub fn username(mut self, username: &str) -> Self { if !username.is_empty() { self.config.username = Some(username.to_string()); } self } /// Set password of this backend. This is required. pub fn password(mut self, password: &str) -> Self { if !password.is_empty() { self.config.password = Some(password.to_string()); } self } /// Set branch of this backend or a commit ID. Default is main. /// /// Branch can be a branch name. /// /// For example, branch can be: /// - main /// - 1d0c4eb pub fn branch(mut self, branch: &str) -> Self { if !branch.is_empty() { self.config.branch = Some(branch.to_string()); } self } /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { if !root.is_empty() { self.config.root = Some(root.to_string()); } self } /// Set the repository of this backend. /// /// This is required. pub fn repository(mut self, repository: &str) -> Self { if !repository.is_empty() { self.config.repository = Some(repository.to_string()); } self } } impl Builder for LakefsBuilder { const SCHEME: Scheme = Scheme::Lakefs; type Config = LakefsConfig; /// Build a LakefsBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let endpoint = match self.config.endpoint { Some(endpoint) => Ok(endpoint.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Lakefs)), }?; debug!("backend use endpoint: {:?}", &endpoint); let repository = match &self.config.repository { Some(repository) => Ok(repository.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "repository is empty") .with_operation("Builder::build") .with_context("service", Scheme::Lakefs)), }?; debug!("backend use repository: {}", &repository); let branch = match &self.config.branch { Some(branch) => branch.clone(), None => "main".to_string(), }; debug!("backend use branch: {}", &branch); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root: {}", &root); let username = match &self.config.username { Some(username) => Ok(username.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "username is empty") .with_operation("Builder::build") .with_context("service", Scheme::Lakefs)), }?; let password = match &self.config.password { Some(password) => Ok(password.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty") .with_operation("Builder::build") .with_context("service", Scheme::Lakefs)), }?; let client = HttpClient::new()?; Ok(LakefsBackend { core: Arc::new(LakefsCore { endpoint, repository, branch, root, username, password, client, }), }) } } /// Backend for Lakefs service #[derive(Debug, Clone)] pub struct LakefsBackend { core: Arc, } impl Access for LakefsBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Lakefs) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_content_disposition: true, stat_has_last_modified: true, list: true, list_has_content_length: true, list_has_last_modified: true, read: true, write: true, delete: true, copy: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, _: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let resp = self.core.get_object_metadata(path).await?; let status = resp.status(); match status { StatusCode::OK => { let mut meta = parse_into_metadata(path, resp.headers())?; let bs = resp.clone().into_body(); let decoded_response: LakefsStatus = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; if let Some(size_bytes) = decoded_response.size_bytes { meta.set_content_length(size_bytes); } meta.set_mode(EntryMode::FILE); if let Some(v) = parse_content_disposition(resp.headers())? { meta.set_content_disposition(v); } meta.set_last_modified(Utc.timestamp_opt(decoded_response.mtime, 0).unwrap()); Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self .core .get_object_content(path, args.range(), &args) .await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = LakefsLister::new( self.core.clone(), path.to_string(), args.limit(), args.start_after(), args.recursive(), ); Ok((RpList::default(), oio::PageLister::new(l))) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { Ok(( RpWrite::default(), oio::OneShotWriter::new(LakefsWriter::new(self.core.clone(), path.to_string(), args)), )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(LakefsDeleter::new(self.core.clone())), )) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.copy_object(from, to).await?; let status = resp.status(); match status { StatusCode::CREATED => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/lakefs/config.rs000064400000000000000000000047401046102023000170310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Configuration for Lakefs service support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct LakefsConfig { /// Base url. /// /// This is required. pub endpoint: Option, /// Username for Lakefs basic authentication. /// /// This is required. pub username: Option, /// Password for Lakefs basic authentication. /// /// This is required. pub password: Option, /// Root of this backend. Can be "/path/to/dir". /// /// Default is "/". pub root: Option, /// The repository name /// /// This is required. pub repository: Option, /// Name of the branch or a commit ID. Default is main. /// /// This is optional. pub branch: Option, } impl Debug for LakefsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("LakefsConfig"); if let Some(endpoint) = &self.endpoint { ds.field("endpoint", &endpoint); } if let Some(_username) = &self.username { ds.field("username", &""); } if let Some(_password) = &self.password { ds.field("password", &""); } if let Some(root) = &self.root { ds.field("root", &root); } if let Some(repository) = &self.repository { ds.field("repository", &repository); } if let Some(branch) = &self.branch { ds.field("branch", &branch); } ds.finish() } } opendal-0.52.0/src/services/lakefs/core.rs000064400000000000000000000171471046102023000165210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::header; use http::Request; use http::Response; use serde::Deserialize; use std::collections::HashMap; use std::fmt::Debug; use crate::raw::*; use crate::*; pub struct LakefsCore { pub endpoint: String, pub repository: String, pub branch: String, pub root: String, pub username: String, pub password: String, pub client: HttpClient, } impl Debug for LakefsCore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("LakefsCore") .field("endpoint", &self.endpoint) .field("username", &self.username) .field("password", &self.password) .field("root", &self.root) .field("repository", &self.repository) .field("branch", &self.branch) .finish_non_exhaustive() } } impl LakefsCore { pub async fn get_object_metadata(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/api/v1/repositories/{}/refs/{}/objects/stat?path={}", self.endpoint, self.repository, self.branch, percent_encode_path(&p) ); let mut req = Request::get(&url); let auth_header_content = format_authorization_by_basic(&self.username, &self.password)?; req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn get_object_content( &self, path: &str, range: BytesRange, _args: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/api/v1/repositories/{}/refs/{}/objects?path={}", self.endpoint, self.repository, self.branch, percent_encode_path(&p) ); let mut req = Request::get(&url); let auth_header_content = format_authorization_by_basic(&self.username, &self.password)?; req = req.header(header::AUTHORIZATION, auth_header_content); if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn list_objects( &self, path: &str, delimiter: &str, amount: &Option, after: Option, ) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/api/v1/repositories/{}/refs/{}/objects/ls?", self.endpoint, self.repository, self.branch ); if !p.is_empty() { url.push_str(&format!("&prefix={}", percent_encode_path(&p))); } if !delimiter.is_empty() { url.push_str(&format!("&delimiter={}", delimiter)); } if let Some(amount) = amount { url.push_str(&format!("&amount={}", amount)); } if let Some(after) = after { url.push_str(&format!("&after={}", after)); } let mut req = Request::get(&url); let auth_header_content = format_authorization_by_basic(&self.username, &self.password)?; req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn upload_object( &self, path: &str, _args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/api/v1/repositories/{}/branches/{}/objects?path={}", self.endpoint, self.repository, self.branch, percent_encode_path(&p) ); let mut req = Request::post(&url); let auth_header_content = format_authorization_by_basic(&self.username, &self.password)?; req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn delete_object(&self, path: &str, _args: &OpDelete) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/api/v1/repositories/{}/branches/{}/objects?path={}", self.endpoint, self.repository, self.branch, percent_encode_path(&p) ); let mut req = Request::delete(&url); let auth_header_content = format_authorization_by_basic(&self.username, &self.password)?; req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn copy_object(&self, path: &str, dest: &str) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let d = build_abs_path(&self.root, dest) .trim_end_matches('/') .to_string(); let url = format!( "{}/api/v1/repositories/{}/branches/{}/objects/copy?dest_path={}", self.endpoint, self.repository, self.branch, percent_encode_path(&d) ); let mut req = Request::post(&url); let auth_header_content = format_authorization_by_basic(&self.username, &self.password)?; req = req.header(header::AUTHORIZATION, auth_header_content); req = req.header(header::CONTENT_TYPE, "application/json"); let mut map = HashMap::new(); map.insert("src_path", p); let req = req .body(serde_json::to_vec(&map).unwrap().into()) .map_err(new_request_build_error)?; self.client.send(req).await } } #[derive(Deserialize, Eq, PartialEq, Debug)] pub(super) struct LakefsStatus { pub path: String, pub path_type: String, pub physical_address: String, pub checksum: String, pub size_bytes: Option, pub mtime: i64, pub content_type: Option, } #[derive(Deserialize, Eq, PartialEq, Debug)] pub(super) struct LakefsListResponse { pub pagination: Pagination, pub results: Vec, } #[derive(Deserialize, Eq, PartialEq, Debug)] pub(super) struct Pagination { pub has_more: bool, pub max_per_page: u64, pub next_offset: String, pub results: u64, } opendal-0.52.0/src/services/lakefs/delete.rs000064400000000000000000000031341046102023000170220ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct LakefsDeleter { core: Arc, } impl LakefsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for LakefsDeleter { async fn delete_once(&self, path: String, args: OpDelete) -> Result<()> { // This would delete the bucket, do not perform if self.core.root == "/" && path == "/" { return Ok(()); } let resp = self.core.delete_object(&path, &args).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT => Ok(()), StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/lakefs/docs.md000064400000000000000000000034431046102023000164670ustar 00000000000000This service will visit the [Lakefs API](https://Lakefs.co/docs/Lakefs_hub/package_reference/hf_api) to access the Lakefs File System. Currently, we only support the `model` and `dataset` types of repositories, and operations are limited to reading and listing/stating. Lakefs doesn't host official HTTP API docs. Detailed HTTP request API information can be found on the [`Lakefs_hub` Source Code](https://github.com/Lakefs/Lakefs_hub). ## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [ ] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [ ] ~~presign~~ - [ ] blocking ## Configurations - `endpoint`: The endpoint of the Lakefs repository. - `repository`: The id of the repository. - `branch`: The branch of the repository. - `root`: Set the work directory for backend. - `username`: The username for accessing the repository. - `password`: The password for accessing the repository. Refer to [`LakefsBuilder`]'s public API docs for more information. ## Examples ### Via Builder ```rust,no_run use opendal::Operator; use opendal::services::Lakefs; use anyhow::Result; #[tokio::main] async fn main() -> Result<()> { // Create Lakefs backend builder let mut builder = Lakefs::default() // set the type of Lakefs endpoint .endpoint("https://whole-llama-mh6mux.us-east-1.lakefscloud.io") // set the id of Lakefs repository .repository("sample-repo") // set the branch of Lakefs repository .branch("main") // set the username for accessing the repository .username("xxx") // set the password for accessing the repository .password("xxx"); let op: Operator = Operator::new(builder)?.finish(); let stat = op.stat("README.md").await?; println!("{:?}", stat); Ok(()) } ``` opendal-0.52.0/src/services/lakefs/error.rs000064400000000000000000000054771046102023000167250ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use http::Response; use http::StatusCode; use serde::Deserialize; use crate::raw::*; use crate::*; /// LakefsError is the error returned by Lakefs File System. #[derive(Default, Deserialize)] struct LakefsError { error: String, } impl Debug for LakefsError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("LakefsError"); de.field("message", &self.error.replace('\n', " ")); de.finish() } } pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED => (ErrorKind::ConditionNotMatch, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = match serde_json::from_slice::(&bs) { Ok(hf_error) => format!("{:?}", hf_error.error), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod test { use super::*; use crate::raw::new_json_deserialize_error; use crate::types::Result; #[test] fn test_parse_error() -> Result<()> { let resp = r#" { "error": "Invalid username or password." } "#; let decoded_response = serde_json::from_slice::(resp.as_bytes()) .map_err(new_json_deserialize_error)?; assert_eq!(decoded_response.error, "Invalid username or password."); Ok(()) } } opendal-0.52.0/src/services/lakefs/lister.rs000064400000000000000000000066501046102023000170700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use chrono::TimeZone; use chrono::Utc; use super::core::LakefsCore; use super::core::LakefsListResponse; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct LakefsLister { core: Arc, path: String, delimiter: &'static str, amount: Option, after: Option, } impl LakefsLister { pub fn new( core: Arc, path: String, amount: Option, after: Option<&str>, recursive: bool, ) -> Self { let delimiter = if recursive { "" } else { "/" }; Self { core, path, delimiter, amount, after: after.map(String::from), } } } impl oio::PageList for LakefsLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let response = self .core .list_objects( &self.path, self.delimiter, &self.amount, // start after should only be set for the first page. if ctx.token.is_empty() { self.after.clone() } else { None }, ) .await?; let status_code = response.status(); if !status_code.is_success() { let error = parse_error(response); return Err(error); } let bytes = response.into_body(); let decoded_response: LakefsListResponse = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; ctx.done = true; for status in decoded_response.results { let entry_type = match status.path_type.as_str() { "common_prefix" => EntryMode::DIR, "object" => EntryMode::FILE, _ => EntryMode::Unknown, }; let mut meta = Metadata::new(entry_type); if status.mtime != 0 { meta.set_last_modified(Utc.timestamp_opt(status.mtime, 0).unwrap()); } if entry_type == EntryMode::FILE { if let Some(size_bytes) = status.size_bytes { meta.set_content_length(size_bytes); } } let path = if entry_type == EntryMode::DIR { format!("{}/", &status.path) } else { status.path.clone() }; ctx.entries.push_back(oio::Entry::new( &build_rel_path(&self.core.root, &path), meta, )); } Ok(()) } } opendal-0.52.0/src/services/lakefs/mod.rs000064400000000000000000000022721046102023000163410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-lakefs")] mod core; #[cfg(feature = "services-lakefs")] mod delete; #[cfg(feature = "services-lakefs")] mod error; #[cfg(feature = "services-lakefs")] mod lister; #[cfg(feature = "services-lakefs")] mod writer; #[cfg(feature = "services-lakefs")] mod backend; #[cfg(feature = "services-lakefs")] pub use backend::LakefsBuilder as Lakefs; mod config; pub use config::LakefsConfig; opendal-0.52.0/src/services/lakefs/writer.rs000064400000000000000000000030241046102023000170720ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::LakefsCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct LakefsWriter { core: Arc, op: OpWrite, path: String, } impl LakefsWriter { pub fn new(core: Arc, path: String, op: OpWrite) -> Self { LakefsWriter { core, path, op } } } impl oio::OneShotWrite for LakefsWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self.core.upload_object(&self.path, &self.op, bs).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/memcached/backend.rs000064400000000000000000000207741046102023000176210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::time::Duration; use bb8::RunError; use tokio::net::TcpStream; use tokio::sync::OnceCell; use super::binary; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::MemcachedConfig; use crate::*; impl Configurator for MemcachedConfig { type Builder = MemcachedBuilder; fn into_builder(self) -> Self::Builder { MemcachedBuilder { config: self } } } /// [Memcached](https://memcached.org/) service support. #[doc = include_str!("docs.md")] #[derive(Clone, Default)] pub struct MemcachedBuilder { config: MemcachedConfig, } impl MemcachedBuilder { /// set the network address of memcached service. /// /// For example: "tcp://localhost:11211" pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { self.config.endpoint = Some(endpoint.to_owned()); } self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// set the username. pub fn username(mut self, username: &str) -> Self { self.config.username = Some(username.to_string()); self } /// set the password. pub fn password(mut self, password: &str) -> Self { self.config.password = Some(password.to_string()); self } /// Set the default ttl for memcached services. pub fn default_ttl(mut self, ttl: Duration) -> Self { self.config.default_ttl = Some(ttl); self } } impl Builder for MemcachedBuilder { const SCHEME: Scheme = Scheme::Memcached; type Config = MemcachedConfig; fn build(self) -> Result { let endpoint = self.config.endpoint.clone().ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_context("service", Scheme::Memcached) })?; let uri = http::Uri::try_from(&endpoint).map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "endpoint is invalid") .with_context("service", Scheme::Memcached) .with_context("endpoint", &endpoint) .set_source(err) })?; match uri.scheme_str() { // If scheme is none, we will use tcp by default. None => (), Some(scheme) => { // We only support tcp by now. if scheme != "tcp" { return Err(Error::new( ErrorKind::ConfigInvalid, "endpoint is using invalid scheme", ) .with_context("service", Scheme::Memcached) .with_context("endpoint", &endpoint) .with_context("scheme", scheme.to_string())); } } }; let host = if let Some(host) = uri.host() { host.to_string() } else { return Err( Error::new(ErrorKind::ConfigInvalid, "endpoint doesn't have host") .with_context("service", Scheme::Memcached) .with_context("endpoint", &endpoint), ); }; let port = if let Some(port) = uri.port_u16() { port } else { return Err( Error::new(ErrorKind::ConfigInvalid, "endpoint doesn't have port") .with_context("service", Scheme::Memcached) .with_context("endpoint", &endpoint), ); }; let endpoint = format!("{host}:{port}",); let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); let conn = OnceCell::new(); Ok(MemcachedBackend::new(Adapter { endpoint, username: self.config.username.clone(), password: self.config.password.clone(), conn, default_ttl: self.config.default_ttl, }) .with_normalized_root(root)) } } /// Backend for memcached services. pub type MemcachedBackend = kv::Backend; #[derive(Clone, Debug)] pub struct Adapter { endpoint: String, username: Option, password: Option, default_ttl: Option, conn: OnceCell>, } impl Adapter { async fn conn(&self) -> Result> { let pool = self .conn .get_or_try_init(|| async { let mgr = MemcacheConnectionManager::new( &self.endpoint, self.username.clone(), self.password.clone(), ); bb8::Pool::builder().build(mgr).await.map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "connect to memecached failed") .set_source(err) }) }) .await?; pool.get().await.map_err(|err| match err { RunError::TimedOut => { Error::new(ErrorKind::Unexpected, "get connection from pool failed").set_temporary() } RunError::User(err) => err, }) } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Memcached, "memcached", Capability { read: true, write: true, shared: true, ..Default::default() }, ) } async fn get(&self, key: &str) -> Result> { let mut conn = self.conn().await?; let result = conn.get(&percent_encode_path(key)).await?; Ok(result.map(Buffer::from)) } async fn set(&self, key: &str, value: Buffer) -> Result<()> { let mut conn = self.conn().await?; conn.set( &percent_encode_path(key), &value.to_vec(), // Set expiration to 0 if ttl not set. self.default_ttl .map(|v| v.as_secs() as u32) .unwrap_or_default(), ) .await } async fn delete(&self, key: &str) -> Result<()> { let mut conn = self.conn().await?; conn.delete(&percent_encode_path(key)).await } } /// A `bb8::ManageConnection` for `memcache_async::ascii::Protocol`. #[derive(Clone, Debug)] struct MemcacheConnectionManager { address: String, username: Option, password: Option, } impl MemcacheConnectionManager { fn new(address: &str, username: Option, password: Option) -> Self { Self { address: address.to_string(), username, password, } } } #[async_trait::async_trait] impl bb8::ManageConnection for MemcacheConnectionManager { type Connection = binary::Connection; type Error = Error; /// TODO: Implement unix stream support. async fn connect(&self) -> Result { let conn = TcpStream::connect(&self.address) .await .map_err(new_std_io_error)?; let mut conn = binary::Connection::new(conn); if let (Some(username), Some(password)) = (self.username.as_ref(), self.password.as_ref()) { conn.auth(username, password).await?; } Ok(conn) } async fn is_valid(&self, conn: &mut Self::Connection) -> Result<(), Self::Error> { conn.version().await.map(|_| ()) } fn has_broken(&self, _: &mut Self::Connection) -> bool { false } } opendal-0.52.0/src/services/memcached/binary.rs000064400000000000000000000211111046102023000175000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use tokio::io::AsyncReadExt; use tokio::io::AsyncWriteExt; use tokio::io::BufReader; use tokio::io::{self}; use tokio::net::TcpStream; use crate::raw::*; use crate::*; pub(super) mod constants { pub const OK_STATUS: u16 = 0x0; pub const KEY_NOT_FOUND: u16 = 0x1; } pub enum Opcode { Get = 0x00, Set = 0x01, Delete = 0x04, Version = 0x0b, StartAuth = 0x21, } pub enum Magic { Request = 0x80, } #[derive(Debug)] pub struct StoreExtras { pub flags: u32, pub expiration: u32, } #[derive(Debug, Default)] pub struct PacketHeader { pub magic: u8, pub opcode: u8, pub key_length: u16, pub extras_length: u8, pub data_type: u8, pub vbucket_id_or_status: u16, pub total_body_length: u32, pub opaque: u32, pub cas: u64, } impl PacketHeader { pub async fn write(self, writer: &mut TcpStream) -> io::Result<()> { writer.write_u8(self.magic).await?; writer.write_u8(self.opcode).await?; writer.write_u16(self.key_length).await?; writer.write_u8(self.extras_length).await?; writer.write_u8(self.data_type).await?; writer.write_u16(self.vbucket_id_or_status).await?; writer.write_u32(self.total_body_length).await?; writer.write_u32(self.opaque).await?; writer.write_u64(self.cas).await?; Ok(()) } pub async fn read(reader: &mut TcpStream) -> Result { let header = PacketHeader { magic: reader.read_u8().await?, opcode: reader.read_u8().await?, key_length: reader.read_u16().await?, extras_length: reader.read_u8().await?, data_type: reader.read_u8().await?, vbucket_id_or_status: reader.read_u16().await?, total_body_length: reader.read_u32().await?, opaque: reader.read_u32().await?, cas: reader.read_u64().await?, }; Ok(header) } } pub struct Response { header: PacketHeader, _key: Vec, _extras: Vec, value: Vec, } pub struct Connection { io: BufReader, } impl Connection { pub fn new(io: TcpStream) -> Self { Self { io: BufReader::new(io), } } pub async fn auth(&mut self, username: &str, password: &str) -> Result<()> { let writer = self.io.get_mut(); let key = "PLAIN"; let request_header = PacketHeader { magic: Magic::Request as u8, opcode: Opcode::StartAuth as u8, key_length: key.len() as u16, total_body_length: (key.len() + username.len() + password.len() + 2) as u32, ..Default::default() }; request_header .write(writer) .await .map_err(new_std_io_error)?; writer .write_all(key.as_bytes()) .await .map_err(new_std_io_error)?; writer .write_all(format!("\x00{}\x00{}", username, password).as_bytes()) .await .map_err(new_std_io_error)?; writer.flush().await.map_err(new_std_io_error)?; parse_response(writer).await?; Ok(()) } pub async fn version(&mut self) -> Result { let writer = self.io.get_mut(); let request_header = PacketHeader { magic: Magic::Request as u8, opcode: Opcode::Version as u8, ..Default::default() }; request_header .write(writer) .await .map_err(new_std_io_error)?; writer.flush().await.map_err(new_std_io_error)?; let response = parse_response(writer).await?; let version = String::from_utf8(response.value); match version { Ok(version) => Ok(version), Err(e) => { Err(Error::new(ErrorKind::Unexpected, "unexpected data received").set_source(e)) } } } pub async fn get(&mut self, key: &str) -> Result>> { let writer = self.io.get_mut(); let request_header = PacketHeader { magic: Magic::Request as u8, opcode: Opcode::Get as u8, key_length: key.len() as u16, total_body_length: key.len() as u32, ..Default::default() }; request_header .write(writer) .await .map_err(new_std_io_error)?; writer .write_all(key.as_bytes()) .await .map_err(new_std_io_error)?; writer.flush().await.map_err(new_std_io_error)?; match parse_response(writer).await { Ok(response) => { if response.header.vbucket_id_or_status == 0x1 { return Ok(None); } Ok(Some(response.value)) } Err(e) => Err(e), } } pub async fn set(&mut self, key: &str, val: &[u8], expiration: u32) -> Result<()> { let writer = self.io.get_mut(); let request_header = PacketHeader { magic: Magic::Request as u8, opcode: Opcode::Set as u8, key_length: key.len() as u16, extras_length: 8, total_body_length: (8 + key.len() + val.len()) as u32, ..Default::default() }; let extras = StoreExtras { flags: 0, expiration, }; request_header .write(writer) .await .map_err(new_std_io_error)?; writer .write_u32(extras.flags) .await .map_err(new_std_io_error)?; writer .write_u32(extras.expiration) .await .map_err(new_std_io_error)?; writer .write_all(key.as_bytes()) .await .map_err(new_std_io_error)?; writer.write_all(val).await.map_err(new_std_io_error)?; writer.flush().await.map_err(new_std_io_error)?; parse_response(writer).await?; Ok(()) } pub async fn delete(&mut self, key: &str) -> Result<()> { let writer = self.io.get_mut(); let request_header = PacketHeader { magic: Magic::Request as u8, opcode: Opcode::Delete as u8, key_length: key.len() as u16, total_body_length: key.len() as u32, ..Default::default() }; request_header .write(writer) .await .map_err(new_std_io_error)?; writer .write_all(key.as_bytes()) .await .map_err(new_std_io_error)?; writer.flush().await.map_err(new_std_io_error)?; parse_response(writer).await?; Ok(()) } } pub async fn parse_response(reader: &mut TcpStream) -> Result { let header = PacketHeader::read(reader).await.map_err(new_std_io_error)?; if header.vbucket_id_or_status != constants::OK_STATUS && header.vbucket_id_or_status != constants::KEY_NOT_FOUND { return Err( Error::new(ErrorKind::Unexpected, "unexpected status received") .with_context("message", format!("{}", header.vbucket_id_or_status)), ); } let mut extras = vec![0x0; header.extras_length as usize]; reader .read_exact(extras.as_mut_slice()) .await .map_err(new_std_io_error)?; let mut key = vec![0x0; header.key_length as usize]; reader .read_exact(key.as_mut_slice()) .await .map_err(new_std_io_error)?; let mut value = vec![ 0x0; (header.total_body_length - u32::from(header.key_length) - u32::from(header.extras_length)) as usize ]; reader .read_exact(value.as_mut_slice()) .await .map_err(new_std_io_error)?; Ok(Response { header, _key: key, _extras: extras, value, }) } opendal-0.52.0/src/services/memcached/config.rs000064400000000000000000000030421046102023000174640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::time::Duration; use serde::Deserialize; use serde::Serialize; /// Config for MemCached services support #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct MemcachedConfig { /// network address of the memcached service. /// /// For example: "tcp://localhost:11211" pub endpoint: Option, /// the working directory of the service. Can be "/path/to/dir" /// /// default is "/" pub root: Option, /// Memcached username, optional. pub username: Option, /// Memcached password, optional. pub password: Option, /// The default ttl for put operations. pub default_ttl: Option, } opendal-0.52.0/src/services/memcached/docs.md000064400000000000000000000020521046102023000171230ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [ ] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `username`: Set the username for authentication. - `password`: Set the password for authentication. - `endpoint`: Set the network address of memcached server - `default_ttl`: Set the ttl for memcached service. You can refer to [`MemcachedBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Memcached; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create memcached backend builder let mut builder = Memcached::default() .endpoint("tcp://127.0.0.1:11211"); // if you enable authentication, set username and password for authentication // builder.username("admin") // builder.password("password"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/memcached/mod.rs000064400000000000000000000020171046102023000167770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-memcached")] mod binary; #[cfg(feature = "services-memcached")] mod backend; #[cfg(feature = "services-memcached")] pub use backend::MemcachedBuilder as Memcached; mod config; pub use config::MemcachedConfig; opendal-0.52.0/src/services/memory/backend.rs000064400000000000000000000121101046102023000172040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::BTreeMap; use std::fmt::Debug; use std::sync::Arc; use std::sync::Mutex; use crate::raw::adapters::typed_kv; use crate::raw::Access; use crate::services::MemoryConfig; use crate::*; impl Configurator for MemoryConfig { type Builder = MemoryBuilder; fn into_builder(self) -> Self::Builder { MemoryBuilder { config: self } } } /// In memory service support. (BTreeMap Based) #[doc = include_str!("docs.md")] #[derive(Default)] pub struct MemoryBuilder { config: MemoryConfig, } impl MemoryBuilder { /// Set the root for BTreeMap. pub fn root(mut self, path: &str) -> Self { self.config.root = Some(path.into()); self } } impl Builder for MemoryBuilder { const SCHEME: Scheme = Scheme::Memory; type Config = MemoryConfig; fn build(self) -> Result { let adapter = Adapter { inner: Arc::new(Mutex::new(BTreeMap::default())), }; Ok(MemoryBackend::new(adapter).with_root(self.config.root.as_deref().unwrap_or_default())) } } /// Backend is used to serve `Accessor` support in memory. pub type MemoryBackend = typed_kv::Backend; #[derive(Clone)] pub struct Adapter { inner: Arc>>, } impl Debug for Adapter { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("MemoryBackend").finish_non_exhaustive() } } impl typed_kv::Adapter for Adapter { fn info(&self) -> typed_kv::Info { typed_kv::Info::new( Scheme::Memory, &format!("{:?}", &self.inner as *const _), typed_kv::Capability { get: true, set: true, delete: true, scan: true, shared: false, }, ) } async fn get(&self, path: &str) -> Result> { self.blocking_get(path) } fn blocking_get(&self, path: &str) -> Result> { match self.inner.lock().unwrap().get(path) { None => Ok(None), Some(bs) => Ok(Some(bs.to_owned())), } } async fn set(&self, path: &str, value: typed_kv::Value) -> Result<()> { self.blocking_set(path, value) } fn blocking_set(&self, path: &str, value: typed_kv::Value) -> Result<()> { self.inner.lock().unwrap().insert(path.to_string(), value); Ok(()) } async fn delete(&self, path: &str) -> Result<()> { self.blocking_delete(path) } fn blocking_delete(&self, path: &str) -> Result<()> { self.inner.lock().unwrap().remove(path); Ok(()) } async fn scan(&self, path: &str) -> Result> { self.blocking_scan(path) } fn blocking_scan(&self, path: &str) -> Result> { let inner = self.inner.lock().unwrap(); if path.is_empty() { return Ok(inner.keys().cloned().collect()); } let mut keys = Vec::new(); for (key, _) in inner.range(path.to_string()..) { if !key.starts_with(path) { break; } keys.push(key.to_string()); } Ok(keys) } } #[cfg(test)] mod tests { use super::*; use crate::raw::adapters::typed_kv::{Adapter, Value}; #[test] fn test_accessor_metadata_name() { let b1 = MemoryBuilder::default().build().unwrap(); assert_eq!(b1.info().name(), b1.info().name()); let b2 = MemoryBuilder::default().build().unwrap(); assert_ne!(b1.info().name(), b2.info().name()) } #[test] fn test_blocking_scan() { let adapter = super::Adapter { inner: Arc::new(Mutex::new(BTreeMap::default())), }; adapter.blocking_set("aaa/bbb/", Value::new_dir()).unwrap(); adapter.blocking_set("aab/bbb/", Value::new_dir()).unwrap(); adapter.blocking_set("aab/ccc/", Value::new_dir()).unwrap(); adapter .blocking_set(&format!("aab{}aaa/", std::char::MAX), Value::new_dir()) .unwrap(); adapter.blocking_set("aac/bbb/", Value::new_dir()).unwrap(); let data = adapter.blocking_scan("aab").unwrap(); assert_eq!(data.len(), 3); for path in data { assert!(path.starts_with("aab")); } } } opendal-0.52.0/src/services/memory/config.rs000064400000000000000000000021011046102023000170610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// Config for memory. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct MemoryConfig { /// root of the backend. pub root: Option, } opendal-0.52.0/src/services/memory/docs.md000064400000000000000000000007561046102023000165360ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] list - [ ] presign - [ ] blocking ## Example ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Memory; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Memory::default().root("/tmp"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/memory/mod.rs000064400000000000000000000017141046102023000164040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-memory")] mod backend; #[cfg(feature = "services-memory")] pub use backend::MemoryBuilder as Memory; mod config; pub use config::MemoryConfig; opendal-0.52.0/src/services/mini_moka/backend.rs000064400000000000000000000133551046102023000176530ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::time::Duration; use log::debug; use mini_moka::sync::Cache; use mini_moka::sync::CacheBuilder; use crate::raw::adapters::typed_kv; use crate::raw::Access; use crate::services::MiniMokaConfig; use crate::*; impl Configurator for MiniMokaConfig { type Builder = MiniMokaBuilder; fn into_builder(self) -> Self::Builder { MiniMokaBuilder { config: self } } } /// [mini-moka](https://github.com/moka-rs/mini-moka) backend support. #[doc = include_str!("docs.md")] #[derive(Default, Debug)] pub struct MiniMokaBuilder { config: MiniMokaConfig, } impl MiniMokaBuilder { /// Sets the max capacity of the cache. /// /// Refer to [`mini-moka::sync::CacheBuilder::max_capacity`](https://docs.rs/mini-moka/latest/mini_moka/sync/struct.CacheBuilder.html#method.max_capacity) pub fn max_capacity(mut self, v: u64) -> Self { if v != 0 { self.config.max_capacity = Some(v); } self } /// Sets the time to live of the cache. /// /// Refer to [`mini-moka::sync::CacheBuilder::time_to_live`](https://docs.rs/mini-moka/latest/mini_moka/sync/struct.CacheBuilder.html#method.time_to_live) pub fn time_to_live(mut self, v: Duration) -> Self { if !v.is_zero() { self.config.time_to_live = Some(v); } self } /// Sets the time to idle of the cache. /// /// Refer to [`mini-moka::sync::CacheBuilder::time_to_idle`](https://docs.rs/mini-moka/latest/mini_moka/sync/struct.CacheBuilder.html#method.time_to_idle) pub fn time_to_idle(mut self, v: Duration) -> Self { if !v.is_zero() { self.config.time_to_idle = Some(v); } self } /// Set root path of this backend pub fn root(mut self, path: &str) -> Self { self.config.root = if path.is_empty() { None } else { Some(path.to_string()) }; self } } impl Builder for MiniMokaBuilder { const SCHEME: Scheme = Scheme::MiniMoka; type Config = MiniMokaConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let mut builder: CacheBuilder = Cache::builder(); // Use entries' bytes as capacity weigher. builder = builder.weigher(|k, v| (k.len() + v.size()) as u32); if let Some(v) = self.config.max_capacity { builder = builder.max_capacity(v) } if let Some(v) = self.config.time_to_live { builder = builder.time_to_live(v) } if let Some(v) = self.config.time_to_idle { builder = builder.time_to_idle(v) } debug!("backend build finished: {:?}", &self); let mut backend = MiniMokaBackend::new(Adapter { inner: builder.build(), }); if let Some(v) = self.config.root { backend = backend.with_root(&v); } Ok(backend) } } /// Backend is used to serve `Accessor` support in mini-moka. pub type MiniMokaBackend = typed_kv::Backend; #[derive(Clone)] pub struct Adapter { inner: Cache, } impl Debug for Adapter { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("Adapter") .field("size", &self.inner.weighted_size()) .field("count", &self.inner.entry_count()) .finish() } } impl typed_kv::Adapter for Adapter { fn info(&self) -> typed_kv::Info { typed_kv::Info::new( Scheme::MiniMoka, "mini-moka", typed_kv::Capability { get: true, set: true, delete: true, scan: true, shared: false, }, ) } async fn get(&self, path: &str) -> Result> { self.blocking_get(path) } fn blocking_get(&self, path: &str) -> Result> { match self.inner.get(&path.to_string()) { None => Ok(None), Some(bs) => Ok(Some(bs)), } } async fn set(&self, path: &str, value: typed_kv::Value) -> Result<()> { self.blocking_set(path, value) } fn blocking_set(&self, path: &str, value: typed_kv::Value) -> Result<()> { self.inner.insert(path.to_string(), value); Ok(()) } async fn delete(&self, path: &str) -> Result<()> { self.blocking_delete(path) } fn blocking_delete(&self, path: &str) -> Result<()> { self.inner.invalidate(&path.to_string()); Ok(()) } async fn scan(&self, path: &str) -> Result> { self.blocking_scan(path) } fn blocking_scan(&self, path: &str) -> Result> { let keys = self.inner.iter().map(|kv| kv.key().to_string()); if path.is_empty() { Ok(keys.collect()) } else { Ok(keys.filter(|k| k.starts_with(path)).collect()) } } } opendal-0.52.0/src/services/mini_moka/config.rs000064400000000000000000000035311046102023000175240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::time::Duration; use serde::Deserialize; use serde::Serialize; /// Config for mini-moka support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct MiniMokaConfig { /// Sets the max capacity of the cache. /// /// Refer to [`mini-moka::sync::CacheBuilder::max_capacity`](https://docs.rs/mini-moka/latest/mini_moka/sync/struct.CacheBuilder.html#method.max_capacity) pub max_capacity: Option, /// Sets the time to live of the cache. /// /// Refer to [`mini-moka::sync::CacheBuilder::time_to_live`](https://docs.rs/mini-moka/latest/mini_moka/sync/struct.CacheBuilder.html#method.time_to_live) pub time_to_live: Option, /// Sets the time to idle of the cache. /// /// Refer to [`mini-moka::sync::CacheBuilder::time_to_idle`](https://docs.rs/mini-moka/latest/mini_moka/sync/struct.CacheBuilder.html#method.time_to_idle) pub time_to_idle: Option, /// root path of this backend pub root: Option, } opendal-0.52.0/src/services/mini_moka/docs.md000064400000000000000000000005601046102023000171620ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] list - [ ] presign - [ ] blocking ## Notes To better assist you in choosing the right cache for your use case, Here's a comparison table with [moka](https://github.com/moka-rs/moka#choosing-the-right-cache-for-your-use-case) opendal-0.52.0/src/services/mini_moka/mod.rs000064400000000000000000000017301046102023000170350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-mini-moka")] mod backend; #[cfg(feature = "services-mini-moka")] pub use backend::MiniMokaBuilder as MiniMoka; mod config; pub use config::MiniMokaConfig; opendal-0.52.0/src/services/mod.rs000064400000000000000000000064471046102023000151040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Services will provide builders to build underlying backends. //! //! More ongoing services support is tracked at [opendal#5](https://github.com/apache/opendal/issues/5). Please feel free to submit issues if there are services not covered. mod aliyun_drive; pub use aliyun_drive::*; mod alluxio; pub use alluxio::*; mod atomicserver; pub use self::atomicserver::*; mod azblob; pub use azblob::*; mod azdls; pub use azdls::*; mod azfile; pub use azfile::*; mod b2; pub use b2::*; mod cacache; pub use self::cacache::*; mod chainsafe; pub use chainsafe::*; mod cloudflare_kv; pub use self::cloudflare_kv::*; mod compfs; pub use compfs::*; mod cos; pub use cos::*; mod d1; pub use self::d1::*; mod dashmap; pub use self::dashmap::*; mod dbfs; pub use self::dbfs::*; mod dropbox; pub use dropbox::*; mod etcd; pub use self::etcd::*; mod foundationdb; pub use self::foundationdb::*; mod fs; pub use fs::*; mod ftp; pub use ftp::*; mod gcs; pub use gcs::*; mod gdrive; pub use gdrive::*; mod ghac; pub use ghac::*; mod github; pub use github::*; mod gridfs; pub use gridfs::*; mod hdfs; pub use self::hdfs::*; mod hdfs_native; pub use hdfs_native::*; mod http; pub use self::http::*; mod huggingface; pub use huggingface::*; mod icloud; pub use icloud::*; mod ipfs; pub use self::ipfs::*; mod ipmfs; pub use ipmfs::*; mod koofr; pub use koofr::*; mod lakefs; pub use lakefs::*; mod memcached; pub use memcached::*; mod memory; pub use self::memory::*; mod mini_moka; pub use self::mini_moka::*; mod moka; pub use self::moka::*; mod mongodb; pub use self::mongodb::*; mod monoiofs; pub use monoiofs::*; mod mysql; pub use self::mysql::*; mod nebula_graph; pub use nebula_graph::*; mod obs; pub use obs::*; mod onedrive; pub use onedrive::*; mod oss; pub use oss::*; mod pcloud; pub use pcloud::*; mod persy; pub use self::persy::*; mod postgresql; pub use self::postgresql::*; mod redb; pub use self::redb::*; mod redis; pub use self::redis::*; mod rocksdb; pub use self::rocksdb::*; mod s3; pub use s3::*; mod seafile; pub use seafile::*; mod sftp; pub use sftp::*; mod sled; pub use self::sled::*; mod sqlite; pub use self::sqlite::*; mod supabase; pub use supabase::*; mod surrealdb; pub use surrealdb::*; mod swift; pub use self::swift::*; mod tikv; pub use self::tikv::*; mod upyun; pub use upyun::*; mod vercel_artifacts; pub use vercel_artifacts::*; mod vercel_blob; pub use vercel_blob::*; mod webdav; pub use webdav::*; mod webhdfs; pub use webhdfs::*; mod yandex_disk; pub use yandex_disk::*; opendal-0.52.0/src/services/moka/backend.rs000064400000000000000000000144531046102023000166370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::time::Duration; use log::debug; use moka::sync::CacheBuilder; use moka::sync::SegmentedCache; use crate::raw::adapters::typed_kv; use crate::raw::*; use crate::services::MokaConfig; use crate::*; impl Configurator for MokaConfig { type Builder = MokaBuilder; fn into_builder(self) -> Self::Builder { MokaBuilder { config: self } } } /// [moka](https://github.com/moka-rs/moka) backend support. #[doc = include_str!("docs.md")] #[derive(Default, Debug)] pub struct MokaBuilder { config: MokaConfig, } impl MokaBuilder { /// Name for this cache instance. pub fn name(mut self, v: &str) -> Self { if !v.is_empty() { self.config.name = Some(v.to_owned()); } self } /// Sets the max capacity of the cache. /// /// Refer to [`moka::sync::CacheBuilder::max_capacity`](https://docs.rs/moka/latest/moka/sync/struct.CacheBuilder.html#method.max_capacity) pub fn max_capacity(mut self, v: u64) -> Self { if v != 0 { self.config.max_capacity = Some(v); } self } /// Sets the time to live of the cache. /// /// Refer to [`moka::sync::CacheBuilder::time_to_live`](https://docs.rs/moka/latest/moka/sync/struct.CacheBuilder.html#method.time_to_live) pub fn time_to_live(mut self, v: Duration) -> Self { if !v.is_zero() { self.config.time_to_live = Some(v); } self } /// Sets the time to idle of the cache. /// /// Refer to [`moka::sync::CacheBuilder::time_to_idle`](https://docs.rs/moka/latest/moka/sync/struct.CacheBuilder.html#method.time_to_idle) pub fn time_to_idle(mut self, v: Duration) -> Self { if !v.is_zero() { self.config.time_to_idle = Some(v); } self } /// Sets the segments number of the cache. /// /// Refer to [`moka::sync::CacheBuilder::segments`](https://docs.rs/moka/latest/moka/sync/struct.CacheBuilder.html#method.segments) pub fn segments(mut self, v: usize) -> Self { assert!(v != 0); self.config.num_segments = Some(v); self } /// Set root path of this backend pub fn root(mut self, path: &str) -> Self { self.config.root = if path.is_empty() { None } else { Some(path.to_string()) }; self } } impl Builder for MokaBuilder { const SCHEME: Scheme = Scheme::Moka; type Config = MokaConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let mut builder: CacheBuilder = SegmentedCache::builder(self.config.num_segments.unwrap_or(1)); // Use entries' bytes as capacity weigher. builder = builder.weigher(|k, v| (k.len() + v.size()) as u32); if let Some(v) = &self.config.name { builder = builder.name(v); } if let Some(v) = self.config.max_capacity { builder = builder.max_capacity(v) } if let Some(v) = self.config.time_to_live { builder = builder.time_to_live(v) } if let Some(v) = self.config.time_to_idle { builder = builder.time_to_idle(v) } debug!("backend build finished: {:?}", &self); let mut backend = MokaBackend::new(Adapter { inner: builder.build(), }); if let Some(v) = self.config.root { backend = backend.with_root(&v); } Ok(backend) } } /// Backend is used to serve `Accessor` support in moka. pub type MokaBackend = typed_kv::Backend; #[derive(Clone)] pub struct Adapter { inner: SegmentedCache, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Adapter") .field("size", &self.inner.weighted_size()) .field("count", &self.inner.entry_count()) .finish() } } impl typed_kv::Adapter for Adapter { fn info(&self) -> typed_kv::Info { typed_kv::Info::new( Scheme::Moka, self.inner.name().unwrap_or("moka"), typed_kv::Capability { get: true, set: true, delete: true, scan: true, shared: false, }, ) } async fn get(&self, path: &str) -> Result> { self.blocking_get(path) } fn blocking_get(&self, path: &str) -> Result> { match self.inner.get(path) { None => Ok(None), Some(bs) => Ok(Some(bs)), } } async fn set(&self, path: &str, value: typed_kv::Value) -> Result<()> { self.blocking_set(path, value) } fn blocking_set(&self, path: &str, value: typed_kv::Value) -> Result<()> { self.inner.insert(path.to_string(), value); Ok(()) } async fn delete(&self, path: &str) -> Result<()> { self.blocking_delete(path) } fn blocking_delete(&self, path: &str) -> Result<()> { self.inner.invalidate(path); Ok(()) } async fn scan(&self, path: &str) -> Result> { self.blocking_scan(path) } fn blocking_scan(&self, path: &str) -> Result> { let keys = self.inner.iter().map(|kv| kv.0.to_string()); if path.is_empty() { Ok(keys.collect()) } else { Ok(keys.filter(|k| k.starts_with(path)).collect()) } } } opendal-0.52.0/src/services/moka/config.rs000064400000000000000000000050761046102023000165160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::time::Duration; use serde::Deserialize; use serde::Serialize; /// Config for Moka services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct MokaConfig { /// Name for this cache instance. pub name: Option, /// Sets the max capacity of the cache. /// /// Refer to [`moka::sync::CacheBuilder::max_capacity`](https://docs.rs/moka/latest/moka/sync/struct.CacheBuilder.html#method.max_capacity) pub max_capacity: Option, /// Sets the time to live of the cache. /// /// Refer to [`moka::sync::CacheBuilder::time_to_live`](https://docs.rs/moka/latest/moka/sync/struct.CacheBuilder.html#method.time_to_live) pub time_to_live: Option, /// Sets the time to idle of the cache. /// /// Refer to [`moka::sync::CacheBuilder::time_to_idle`](https://docs.rs/moka/latest/moka/sync/struct.CacheBuilder.html#method.time_to_idle) pub time_to_idle: Option, /// Sets the segments number of the cache. /// /// Refer to [`moka::sync::CacheBuilder::segments`](https://docs.rs/moka/latest/moka/sync/struct.CacheBuilder.html#method.segments) pub num_segments: Option, /// root path of this backend pub root: Option, } impl Debug for MokaConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("MokaConfig") .field("name", &self.name) .field("max_capacity", &self.max_capacity) .field("time_to_live", &self.time_to_live) .field("time_to_idle", &self.time_to_idle) .field("num_segments", &self.num_segments) .field("root", &self.root) .finish_non_exhaustive() } } opendal-0.52.0/src/services/moka/docs.md000064400000000000000000000014671046102023000161550ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] list - [ ] presign - [ ] blocking ## Configuration - `name`: Set the name for this cache instance. - `max_capacity`: Set the max capacity of the cache. - `time_to_live`: Set the time to live of the cache. - `time_to_idle`: Set the time to idle of the cache. - `num_segments`: Set the segments number of the cache. You can refer to [`MokaBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Moka; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Moka::default() .name("opendal"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/moka/mod.rs000064400000000000000000000017021046102023000160200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-moka")] mod backend; #[cfg(feature = "services-moka")] pub use backend::MokaBuilder as Moka; mod config; pub use config::MokaConfig; opendal-0.52.0/src/services/mongodb/backend.rs000064400000000000000000000226351046102023000173360ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use mongodb::bson::doc; use mongodb::bson::Binary; use mongodb::bson::Document; use mongodb::options::ClientOptions; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::MongodbConfig; use crate::*; impl Configurator for MongodbConfig { type Builder = MongodbBuilder; fn into_builder(self) -> Self::Builder { MongodbBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct MongodbBuilder { config: MongodbConfig, } impl Debug for MongodbBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("MongodbBuilder") .field("config", &self.config) .finish() } } impl MongodbBuilder { /// Set the connection_string of the MongoDB service. /// /// This connection string is used to connect to the MongoDB service. It typically follows the format: /// /// ## Format /// /// `mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]]` /// /// Examples: /// /// - Connecting to a local MongoDB instance: `mongodb://localhost:27017` /// - Using authentication: `mongodb://myUser:myPassword@localhost:27017/myAuthDB` /// - Specifying authentication mechanism: `mongodb://myUser:myPassword@localhost:27017/myAuthDB?authMechanism=SCRAM-SHA-256` /// /// ## Options /// /// - `authMechanism`: Specifies the authentication method to use. Examples include `SCRAM-SHA-1`, `SCRAM-SHA-256`, and `MONGODB-AWS`. /// - ... (any other options you wish to highlight) /// /// For more information, please refer to [MongoDB Connection String URI Format](https://docs.mongodb.com/manual/reference/connection-string/). pub fn connection_string(mut self, v: &str) -> Self { if !v.is_empty() { self.config.connection_string = Some(v.to_string()); } self } /// Set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the database name of the MongoDB service to read/write. pub fn database(mut self, database: &str) -> Self { if !database.is_empty() { self.config.database = Some(database.to_string()); } self } /// Set the collection name of the MongoDB service to read/write. pub fn collection(mut self, collection: &str) -> Self { if !collection.is_empty() { self.config.collection = Some(collection.to_string()); } self } /// Set the key field name of the MongoDB service to read/write. /// /// Default to `key` if not specified. pub fn key_field(mut self, key_field: &str) -> Self { if !key_field.is_empty() { self.config.key_field = Some(key_field.to_string()); } self } /// Set the value field name of the MongoDB service to read/write. /// /// Default to `value` if not specified. pub fn value_field(mut self, value_field: &str) -> Self { if !value_field.is_empty() { self.config.value_field = Some(value_field.to_string()); } self } } impl Builder for MongodbBuilder { const SCHEME: Scheme = Scheme::Mongodb; type Config = MongodbConfig; fn build(self) -> Result { let conn = match &self.config.connection_string.clone() { Some(v) => v.clone(), None => { return Err( Error::new(ErrorKind::ConfigInvalid, "connection_string is required") .with_context("service", Scheme::Mongodb), ) } }; let database = match &self.config.database.clone() { Some(v) => v.clone(), None => { return Err(Error::new(ErrorKind::ConfigInvalid, "database is required") .with_context("service", Scheme::Mongodb)) } }; let collection = match &self.config.collection.clone() { Some(v) => v.clone(), None => { return Err( Error::new(ErrorKind::ConfigInvalid, "collection is required") .with_context("service", Scheme::Mongodb), ) } }; let key_field = match &self.config.key_field.clone() { Some(v) => v.clone(), None => "key".to_string(), }; let value_field = match &self.config.value_field.clone() { Some(v) => v.clone(), None => "value".to_string(), }; let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); Ok(MongodbBackend::new(Adapter { connection_string: conn, database, collection, collection_instance: OnceCell::new(), key_field, value_field, }) .with_normalized_root(root)) } } pub type MongodbBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { connection_string: String, database: String, collection: String, collection_instance: OnceCell>, key_field: String, value_field: String, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Adapter") .field("connection_string", &self.connection_string) .field("database", &self.database) .field("collection", &self.collection) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .finish() } } impl Adapter { async fn get_collection(&self) -> Result<&mongodb::Collection> { self.collection_instance .get_or_try_init(|| async { let client_options = ClientOptions::parse(&self.connection_string) .await .map_err(parse_mongodb_error)?; let client = mongodb::Client::with_options(client_options).map_err(parse_mongodb_error)?; let database = client.database(&self.database); let collection = database.collection(&self.collection); Ok(collection) }) .await } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Mongodb, &format!("{}/{}", self.database, self.collection), Capability { read: true, write: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let collection = self.get_collection().await?; let filter = doc! {self.key_field.as_str():path}; let result = collection .find_one(filter) .await .map_err(parse_mongodb_error)?; match result { Some(doc) => { let value = doc .get_binary_generic(&self.value_field) .map_err(parse_bson_error)?; Ok(Some(Buffer::from(value.to_vec()))) } None => Ok(None), } } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let collection = self.get_collection().await?; let filter = doc! { self.key_field.as_str(): path }; let update = doc! { "$set": { self.value_field.as_str(): Binary { subtype: mongodb::bson::spec::BinarySubtype::Generic, bytes: value.to_vec() } } }; collection .update_one(filter, update) .upsert(true) .await .map_err(parse_mongodb_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let collection = self.get_collection().await?; let filter = doc! {self.key_field.as_str():path}; collection .delete_one(filter) .await .map_err(parse_mongodb_error)?; Ok(()) } } fn parse_mongodb_error(err: mongodb::error::Error) -> Error { Error::new(ErrorKind::Unexpected, "mongodb error").set_source(err) } fn parse_bson_error(err: mongodb::bson::document::ValueAccessError) -> Error { Error::new(ErrorKind::Unexpected, "bson error").set_source(err) } opendal-0.52.0/src/services/mongodb/config.rs000064400000000000000000000036471046102023000172160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Mongodb service support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct MongodbConfig { /// connection string of this backend pub connection_string: Option, /// database of this backend pub database: Option, /// collection of this backend pub collection: Option, /// root of this backend pub root: Option, /// key field of this backend pub key_field: Option, /// value field of this backend pub value_field: Option, } impl Debug for MongodbConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("MongodbConfig") .field("connection_string", &self.connection_string) .field("database", &self.database) .field("collection", &self.collection) .field("root", &self.root) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .finish() } } opendal-0.52.0/src/services/mongodb/docs.md000064400000000000000000000022411046102023000166420ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `connection_string`: Set the connection string of mongodb server - `database`: Set the database of mongodb - `collection`: Set the collection of mongodb - `key_field`: Set the key field of mongodb - `value_field`: Set the value field of mongodb ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Mongodb; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Mongodb::default() .root("/") .connection_string("mongodb://myUser:myPassword@localhost:27017/myAuthDB") .database("your_database") .collection("your_collection") // key field type in the table should be compatible with Rust's &str like text .key_field("key") // value field type in the table should be compatible with Rust's Vec like bytea .value_field("value"); let op = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/mongodb/mod.rs000064400000000000000000000017211046102023000165170ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-mongodb")] mod backend; #[cfg(feature = "services-mongodb")] pub use backend::MongodbBuilder as Mongodb; mod config; pub use config::MongodbConfig; opendal-0.52.0/src/services/monoiofs/backend.rs000064400000000000000000000213431046102023000175350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::io; use std::path::PathBuf; use std::sync::Arc; use chrono::DateTime; use monoio::fs::OpenOptions; use super::core::MonoiofsCore; use super::core::BUFFER_SIZE; use super::delete::MonoiofsDeleter; use super::reader::MonoiofsReader; use super::writer::MonoiofsWriter; use crate::raw::*; use crate::services::MonoiofsConfig; use crate::*; impl Configurator for MonoiofsConfig { type Builder = MonoiofsBuilder; fn into_builder(self) -> Self::Builder { MonoiofsBuilder { config: self } } } /// File system support via [`monoio`]. #[doc = include_str!("docs.md")] #[derive(Default, Debug)] pub struct MonoiofsBuilder { config: MonoiofsConfig, } impl MonoiofsBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } } impl Builder for MonoiofsBuilder { const SCHEME: Scheme = Scheme::Monoiofs; type Config = MonoiofsConfig; fn build(self) -> Result { let root = self.config.root.map(PathBuf::from).ok_or( Error::new(ErrorKind::ConfigInvalid, "root is not specified") .with_operation("Builder::build"), )?; if let Err(e) = std::fs::metadata(&root) { if e.kind() == io::ErrorKind::NotFound { std::fs::create_dir_all(&root).map_err(|e| { Error::new(ErrorKind::Unexpected, "create root dir failed") .with_operation("Builder::build") .with_context("root", root.to_string_lossy()) .set_source(e) })?; } } let root = root.canonicalize().map_err(|e| { Error::new( ErrorKind::Unexpected, "canonicalize of root directory failed", ) .with_operation("Builder::build") .with_context("root", root.to_string_lossy()) .set_source(e) })?; let worker_threads = 1; // TODO: test concurrency and default to available_parallelism and bind cpu let io_uring_entries = 1024; Ok(MonoiofsBackend { core: Arc::new(MonoiofsCore::new(root, worker_threads, io_uring_entries)), }) } } #[derive(Debug, Clone)] pub struct MonoiofsBackend { core: Arc, } impl Access for MonoiofsBackend { type Reader = MonoiofsReader; type Writer = MonoiofsWriter; type Lister = (); type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Monoiofs) .set_root(&self.core.root().to_string_lossy()) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, write: true, write_can_append: true, write_has_last_modified: true, delete: true, rename: true, create_dir: true, copy: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, _args: OpStat) -> Result { let path = self.core.prepare_path(path); let meta = self .core .dispatch(move || monoio::fs::metadata(path)) .await .map_err(new_std_io_error)?; let mode = if meta.is_dir() { EntryMode::DIR } else if meta.is_file() { EntryMode::FILE } else { EntryMode::Unknown }; let m = Metadata::new(mode) .with_content_length(meta.len()) .with_last_modified( meta.modified() .map(DateTime::from) .map_err(new_std_io_error)?, ); Ok(RpStat::new(m)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let path = self.core.prepare_path(path); let reader = MonoiofsReader::new(self.core.clone(), path, args.range()).await?; Ok((RpRead::default(), reader)) } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let path = self.core.prepare_write_path(path).await?; let writer = MonoiofsWriter::new(self.core.clone(), path, args.append()).await?; Ok((RpWrite::default(), writer)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(MonoiofsDeleter::new(self.core.clone())), )) } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let from = self.core.prepare_path(from); // ensure file exists self.core .dispatch({ let from = from.clone(); move || monoio::fs::metadata(from) }) .await .map_err(new_std_io_error)?; let to = self.core.prepare_write_path(to).await?; self.core .dispatch(move || monoio::fs::rename(from, to)) .await .map_err(new_std_io_error)?; Ok(RpRename::default()) } async fn create_dir(&self, path: &str, _args: OpCreateDir) -> Result { let path = self.core.prepare_path(path); self.core .dispatch(move || monoio::fs::create_dir_all(path)) .await .map_err(new_std_io_error)?; Ok(RpCreateDir::default()) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let from = self.core.prepare_path(from); // ensure file exists self.core .dispatch({ let from = from.clone(); move || monoio::fs::metadata(from) }) .await .map_err(new_std_io_error)?; let to = self.core.prepare_write_path(to).await?; self.core .dispatch({ let core = self.core.clone(); move || async move { let from = OpenOptions::new().read(true).open(from).await?; let to = OpenOptions::new() .write(true) .create(true) .truncate(true) .open(to) .await?; // AsyncReadRent and AsyncWriteRent is not implemented // for File, so we can't write this: // monoio::io::copy(&mut from, &mut to).await?; let mut pos = 0; // allocate and resize buffer let mut buf = core.buf_pool.get(); // set capacity of buf to exact size to avoid excessive read buf.reserve(BUFFER_SIZE); let _ = buf.split_off(BUFFER_SIZE); loop { let result; (result, buf) = from.read_at(buf, pos).await; if result? == 0 { // EOF break; } let result; (result, buf) = to.write_all_at(buf, pos).await; result?; pos += buf.len() as u64; buf.clear(); } core.buf_pool.put(buf); Ok(()) } }) .await .map_err(new_std_io_error)?; Ok(RpCopy::default()) } } opendal-0.52.0/src/services/monoiofs/config.rs000064400000000000000000000023241046102023000174110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// Config for monoiofs services support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct MonoiofsConfig { /// The Root of this backend. /// /// All operations will happen under this root. /// /// Builder::build will return error if not set. pub root: Option, } opendal-0.52.0/src/services/monoiofs/core.rs000064400000000000000000000235221046102023000170770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::mem; use std::path::PathBuf; use std::sync::Mutex; use std::time::Duration; use flume::Receiver; use flume::Sender; use futures::channel::oneshot; use futures::Future; use monoio::FusionDriver; use monoio::RuntimeBuilder; use crate::raw::*; use crate::*; pub const BUFFER_SIZE: usize = 2 * 1024 * 1024; // 2 MiB /// a boxed function that spawns task in current monoio runtime type TaskSpawner = Box; #[derive(Debug)] pub struct MonoiofsCore { root: PathBuf, /// sender that sends [`TaskSpawner`] to worker threads tx: Sender, /// join handles of worker threads threads: Mutex>>, pub buf_pool: oio::PooledBuf, } impl MonoiofsCore { pub fn new(root: PathBuf, worker_threads: usize, io_uring_entries: u32) -> Self { // Since users use monoiofs in a context of tokio, all monoio // operations need to be dispatched to a dedicated thread pool // where a monoio runtime runs on each thread. Here we spawn // these worker threads. let (tx, rx) = flume::unbounded(); let threads = (0..worker_threads) .map(move |i| { let rx = rx.clone(); std::thread::Builder::new() .name(format!("monoiofs-worker-{i}")) .spawn(move || Self::worker_entrypoint(rx, io_uring_entries)) .expect("spawn worker thread should success") }) .collect(); let threads = Mutex::new(threads); Self { root, tx, threads, buf_pool: oio::PooledBuf::new(16).with_initial_capacity(BUFFER_SIZE), } } pub fn root(&self) -> &PathBuf { &self.root } /// join root and path pub fn prepare_path(&self, path: &str) -> PathBuf { self.root.join(path.trim_end_matches('/')) } /// join root and path, create parent dirs pub async fn prepare_write_path(&self, path: &str) -> Result { let path = self.prepare_path(path); let parent = path .parent() .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "path should have parent but not, it must be malformed", ) .with_context("input", path.to_string_lossy()) })? .to_path_buf(); self.dispatch(move || monoio::fs::create_dir_all(parent)) .await .map_err(new_std_io_error)?; Ok(path) } /// entrypoint of each worker thread, sets up monoio runtimes and channels fn worker_entrypoint(rx: Receiver, io_uring_entries: u32) { let mut rt = RuntimeBuilder::::new() .enable_all() .with_entries(io_uring_entries) .build() .expect("monoio runtime initialize should success"); // run an infinite loop that receives TaskSpawner and calls // them in a context of monoio rt.block_on(async { while let Ok(spawner) = rx.recv_async().await { spawner(); } }) } /// Create a TaskSpawner, send it to the thread pool and wait /// for its result. Task panic will propagate. pub async fn dispatch(&self, f: F) -> T where F: FnOnce() -> Fut + 'static + Send, Fut: Future, T: 'static + Send, { // oneshot channel to send result back let (tx, rx) = oneshot::channel(); let result = self .tx .send_async(Box::new(move || { // task will be spawned on current thread, task panic // will cause current worker thread panic monoio::spawn(async move { // discard the result if send failed due to // MonoiofsCore::dispatch cancelled let _ = tx.send(f().await); }); })) .await; self.unwrap(result); self.unwrap(rx.await) } /// Create a TaskSpawner, send it to the thread pool and spawn the task. pub async fn spawn(&self, f: F) where F: FnOnce() -> Fut + 'static + Send, Fut: Future + 'static, T: 'static, { let result = self .tx .send_async(Box::new(move || { // task will be spawned on current thread, task panic // will cause current worker thread panic monoio::spawn(f()); })) .await; self.unwrap(result); } /// This method always panics. It is called only when at least a /// worker thread has panicked or meet a broken rx, which is /// unrecoverable. It propagates worker thread's panic if there /// is any and panics on normally exited thread. pub fn propagate_worker_panic(&self) -> ! { let mut guard = self.threads.lock().expect("worker thread has panicked"); // wait until the panicked thread exits std::thread::sleep(Duration::from_millis(100)); let threads = mem::take(&mut *guard); // we don't know which thread panicked, so check them one by one for thread in threads { if thread.is_finished() { // worker thread runs an infinite loop, hence finished // thread must have panicked or meet a broken rx. match thread.join() { // rx is broken Ok(()) => panic!("worker thread should not exit, tx may be dropped"), // thread has panicked Err(e) => std::panic::resume_unwind(e), } } } unreachable!("this method should panic") } /// Unwrap result if result is Ok, otherwise propagates worker thread's /// panic. This method facilitates panic propagation in situation where /// Err returned by broken channel indicates that the worker thread has /// panicked. pub fn unwrap(&self, result: Result) -> T { match result { Ok(result) => result, Err(_) => self.propagate_worker_panic(), } } } #[cfg(test)] mod tests { use std::sync::Arc; use std::time::Duration; use futures::channel::mpsc::UnboundedSender; use futures::channel::mpsc::{self}; use futures::StreamExt; use super::*; fn new_core(worker_threads: usize) -> Arc { Arc::new(MonoiofsCore::new(PathBuf::new(), worker_threads, 1024)) } async fn dispatch_simple(core: Arc) { let result = core.dispatch(|| async { 42 }).await; assert_eq!(result, 42); let bytes: Vec = vec![1, 2, 3, 4, 5, 6, 7, 8]; let bytes_clone = bytes.clone(); let result = core.dispatch(move || async move { bytes }).await; assert_eq!(result, bytes_clone); } async fn dispatch_concurrent(core: Arc) { let (tx, mut rx) = mpsc::unbounded(); fn spawn_task(core: Arc, tx: UnboundedSender, sleep_millis: u64) { tokio::spawn(async move { let result = core .dispatch(move || async move { monoio::time::sleep(Duration::from_millis(sleep_millis)).await; sleep_millis }) .await; assert_eq!(result, sleep_millis); tx.unbounded_send(result).unwrap(); }); } spawn_task(core.clone(), tx.clone(), 200); spawn_task(core.clone(), tx.clone(), 20); drop(tx); let first = rx.next().await; let second = rx.next().await; let third = rx.next().await; assert_eq!(first, Some(20)); assert_eq!(second, Some(200)); assert_eq!(third, None); } async fn dispatch_panic(core: Arc) { core.dispatch(|| async { panic!("BOOM") }).await; } #[tokio::test] async fn test_monoio_single_thread_dispatch() { let core = new_core(1); assert_eq!(core.threads.lock().unwrap().len(), 1); dispatch_simple(core).await; } #[tokio::test] async fn test_monoio_single_thread_dispatch_concurrent() { let core = new_core(1); dispatch_concurrent(core).await; } #[tokio::test] #[should_panic(expected = "BOOM")] async fn test_monoio_single_thread_dispatch_panic() { let core = new_core(1); dispatch_panic(core).await; } #[tokio::test] async fn test_monoio_multi_thread_dispatch() { let core = new_core(4); assert_eq!(core.threads.lock().unwrap().len(), 4); dispatch_simple(core).await; } #[tokio::test] async fn test_monoio_multi_thread_dispatch_concurrent() { let core = new_core(4); dispatch_concurrent(core).await; } #[tokio::test] #[should_panic(expected = "BOOM")] async fn test_monoio_multi_thread_dispatch_panic() { let core = new_core(4); dispatch_panic(core).await; } } opendal-0.52.0/src/services/monoiofs/delete.rs000064400000000000000000000040641046102023000174110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::MonoiofsCore; use crate::raw::*; use crate::*; use std::sync::Arc; pub struct MonoiofsDeleter { core: Arc, } impl MonoiofsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for MonoiofsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let path = self.core.prepare_path(&path); let meta = self .core .dispatch({ let path = path.clone(); move || monoio::fs::metadata(path) }) .await; match meta { Ok(meta) => { if meta.is_dir() { self.core .dispatch(move || monoio::fs::remove_dir(path)) .await .map_err(new_std_io_error)?; } else { self.core .dispatch(move || monoio::fs::remove_file(path)) .await .map_err(new_std_io_error)?; } Ok(()) } Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(()), Err(err) => Err(new_std_io_error(err)), } } } opendal-0.52.0/src/services/monoiofs/docs.md000064400000000000000000000015711046102023000170530ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] append - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] list - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the work dir for backend. You can refer to [`MonoiofsBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Monoiofs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create monoiofs backend builder. let mut builder = Monoiofs::default() // Set the root for monoiofs, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/tmp"); // `Accessor` provides the low level APIs, we will use `Operator` normally. let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/monoiofs/mod.rs000064400000000000000000000022351046102023000167240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-monoiofs")] mod core; #[cfg(feature = "services-monoiofs")] mod delete; #[cfg(feature = "services-monoiofs")] mod reader; #[cfg(feature = "services-monoiofs")] mod writer; #[cfg(feature = "services-monoiofs")] mod backend; #[cfg(feature = "services-monoiofs")] pub use backend::MonoiofsBuilder as Monoiofs; mod config; pub use config::MonoiofsConfig; opendal-0.52.0/src/services/monoiofs/reader.rs000064400000000000000000000116451046102023000174140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::path::PathBuf; use std::sync::Arc; use bytes::BytesMut; use futures::channel::mpsc; use futures::channel::oneshot; use futures::SinkExt; use futures::StreamExt; use monoio::fs::OpenOptions; use super::core::MonoiofsCore; use super::core::BUFFER_SIZE; use crate::raw::*; use crate::*; enum ReaderRequest { Read { pos: u64, buf: BytesMut, tx: oneshot::Sender>, }, } pub struct MonoiofsReader { core: Arc, tx: mpsc::UnboundedSender, pos: u64, end_pos: Option, } impl MonoiofsReader { pub async fn new(core: Arc, path: PathBuf, range: BytesRange) -> Result { let (open_result_tx, open_result_rx) = oneshot::channel(); let (tx, rx) = mpsc::unbounded(); core.spawn(move || Self::worker_entrypoint(path, rx, open_result_tx)) .await; core.unwrap(open_result_rx.await)?; Ok(Self { core, tx, pos: range.offset(), end_pos: range.size().map(|size| range.offset() + size), }) } /// entrypoint of worker task that runs in context of monoio async fn worker_entrypoint( path: PathBuf, mut rx: mpsc::UnboundedReceiver, open_result_tx: oneshot::Sender>, ) { let result = OpenOptions::new().read(true).open(path).await; // [`monoio::fs::File`] is non-Send, hence it is kept within // worker thread let file = match result { Ok(file) => { let Ok(()) = open_result_tx.send(Ok(())) else { // MonoiofsReader::new is cancelled, exit worker task return; }; file } Err(e) => { // discard the result if send failed due to MonoiofsReader::new // cancelled since we are going to exit anyway let _ = open_result_tx.send(Err(new_std_io_error(e))); return; } }; // wait for read request and send back result to main thread loop { let Some(req) = rx.next().await else { // MonoiofsReader is dropped, exit worker task break; }; match req { ReaderRequest::Read { pos, buf, tx } => { let (result, buf) = file.read_at(buf, pos).await; // buf.len() will be set to n by monoio if read // successfully, so n is dropped let result = result.map(move |_| buf).map_err(new_std_io_error); // discard the result if send failed due to // MonoiofsReader::read cancelled let _ = tx.send(result); } } } } } impl oio::Read for MonoiofsReader { /// Send read request to worker thread and wait for result. Actual /// read happens in [`MonoiofsReader::worker_entrypoint`] running /// on worker thread. async fn read(&mut self) -> Result { if let Some(end_pos) = self.end_pos { if self.pos >= end_pos { return Ok(Buffer::new()); } } // allocate and resize buffer let mut buf = self.core.buf_pool.get(); let size = self .end_pos .map_or(BUFFER_SIZE, |end_pos| (end_pos - self.pos) as usize); // set capacity of buf to exact size to avoid excessive read buf.reserve(size); let _ = buf.split_off(size); // send read request to worker thread and wait for result let (tx, rx) = oneshot::channel(); self.core.unwrap( self.tx .send(ReaderRequest::Read { pos: self.pos, buf, tx, }) .await, ); let mut buf = self.core.unwrap(rx.await)?; // advance cursor if read successfully self.pos += buf.len() as u64; let buffer = Buffer::from(buf.split().freeze()); self.core.buf_pool.put(buf); Ok(buffer) } } opendal-0.52.0/src/services/monoiofs/writer.rs000064400000000000000000000143541046102023000174660ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::path::PathBuf; use std::sync::Arc; use bytes::Buf; use bytes::Bytes; use chrono::DateTime; use futures::channel::mpsc; use futures::channel::oneshot; use futures::SinkExt; use futures::StreamExt; use monoio::fs::OpenOptions; use super::core::MonoiofsCore; use crate::raw::*; use crate::*; enum WriterRequest { Write { pos: u64, buf: Bytes, tx: oneshot::Sender>, }, Stat { tx: oneshot::Sender>, }, Close { tx: oneshot::Sender>, }, } pub struct MonoiofsWriter { core: Arc, tx: mpsc::UnboundedSender, pos: u64, } impl MonoiofsWriter { pub async fn new(core: Arc, path: PathBuf, append: bool) -> Result { let (open_result_tx, open_result_rx) = oneshot::channel(); let (tx, rx) = mpsc::unbounded(); core.spawn(move || Self::worker_entrypoint(path, append, rx, open_result_tx)) .await; core.unwrap(open_result_rx.await)?; Ok(Self { core, tx, pos: 0 }) } /// entrypoint of worker task that runs in context of monoio async fn worker_entrypoint( path: PathBuf, append: bool, mut rx: mpsc::UnboundedReceiver, open_result_tx: oneshot::Sender>, ) { let result = OpenOptions::new() .write(true) .create(true) .append(append) .truncate(!append) .open(path) .await; // [`monoio::fs::File`] is non-Send, hence it is kept within // worker thread let file = match result { Ok(file) => { let Ok(()) = open_result_tx.send(Ok(())) else { // MonoiofsWriter::new is cancelled, exit worker task return; }; file } Err(e) => { // discard the result if send failed due to MonoiofsWriter::new // cancelled since we are going to exit anyway let _ = open_result_tx.send(Err(new_std_io_error(e))); return; } }; // wait for write or close request and send back result to main thread loop { let Some(req) = rx.next().await else { // MonoiofsWriter is dropped, exit worker task break; }; match req { WriterRequest::Write { pos, buf, tx } => { let (result, _) = file.write_all_at(buf, pos).await; // discard the result if send failed due to // MonoiofsWriter::write cancelled let _ = tx.send(result.map_err(new_std_io_error)); } WriterRequest::Stat { tx } => { let result = file.metadata().await; let _ = tx.send(result.map_err(new_std_io_error)); } WriterRequest::Close { tx } => { let result = file.sync_all().await; // discard the result if send failed due to // MonoiofsWriter::close cancelled let _ = tx.send(result.map_err(new_std_io_error)); // file is closed in background and result is useless let _ = file.close().await; break; } } } } } impl oio::Write for MonoiofsWriter { /// Send write request to worker thread and wait for result. Actual /// write happens in [`MonoiofsWriter::worker_entrypoint`] running /// on worker thread. async fn write(&mut self, mut bs: Buffer) -> Result<()> { while bs.has_remaining() { let buf = bs.current(); let n = buf.len(); let (tx, rx) = oneshot::channel(); self.core.unwrap( self.tx .send(WriterRequest::Write { pos: self.pos, buf, tx, }) .await, ); self.core.unwrap(rx.await)?; self.pos += n as u64; bs.advance(n); } Ok(()) } /// Send close request to worker thread and wait for result. Actual /// close happens in [`MonoiofsWriter::worker_entrypoint`] running /// on worker thread. async fn close(&mut self) -> Result { let (tx, rx) = oneshot::channel(); self.core .unwrap(self.tx.send(WriterRequest::Stat { tx }).await); let file_meta = self.core.unwrap(rx.await)?; let (tx, rx) = oneshot::channel(); self.core .unwrap(self.tx.send(WriterRequest::Close { tx }).await); self.core.unwrap(rx.await)?; let mode = if file_meta.is_dir() { EntryMode::DIR } else if file_meta.is_file() { EntryMode::FILE } else { EntryMode::Unknown }; let meta = Metadata::new(mode) .with_content_length(file_meta.len()) .with_last_modified( file_meta .modified() .map(DateTime::from) .map_err(new_std_io_error)?, ); Ok(meta) } async fn abort(&mut self) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "Monoiofs doesn't support abort", )) } } opendal-0.52.0/src/services/mysql/backend.rs000064400000000000000000000165361046102023000170610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::str::FromStr; use sqlx::mysql::MySqlConnectOptions; use sqlx::MySqlPool; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::MysqlConfig; use crate::*; impl Configurator for MysqlConfig { type Builder = MysqlBuilder; fn into_builder(self) -> Self::Builder { MysqlBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct MysqlBuilder { config: MysqlConfig, } impl Debug for MysqlBuilder { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("MysqlBuilder"); d.field("config", &self.config).finish() } } impl MysqlBuilder { /// Set the connection_string of the mysql service. /// /// This connection string is used to connect to the mysql service. There are url based formats: /// /// ## Url /// /// This format resembles the url format of the mysql client. The format is: `[scheme://][user[:[password]]@]host[:port][/schema][?attribute1=value1&attribute2=value2...` /// /// - `mysql://user@localhost` /// - `mysql://user:password@localhost` /// - `mysql://user:password@localhost:3306` /// - `mysql://user:password@localhost:3306/db` /// /// For more information, please refer to . pub fn connection_string(mut self, v: &str) -> Self { if !v.is_empty() { self.config.connection_string = Some(v.to_string()); } self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the table name of the mysql service to read/write. pub fn table(mut self, table: &str) -> Self { if !table.is_empty() { self.config.table = Some(table.to_string()); } self } /// Set the key field name of the mysql service to read/write. /// /// Default to `key` if not specified. pub fn key_field(mut self, key_field: &str) -> Self { if !key_field.is_empty() { self.config.key_field = Some(key_field.to_string()); } self } /// Set the value field name of the mysql service to read/write. /// /// Default to `value` if not specified. pub fn value_field(mut self, value_field: &str) -> Self { if !value_field.is_empty() { self.config.value_field = Some(value_field.to_string()); } self } } impl Builder for MysqlBuilder { const SCHEME: Scheme = Scheme::Mysql; type Config = MysqlConfig; fn build(self) -> Result { let conn = match self.config.connection_string { Some(v) => v, None => { return Err( Error::new(ErrorKind::ConfigInvalid, "connection_string is empty") .with_context("service", Scheme::Mysql), ) } }; let config = MySqlConnectOptions::from_str(&conn).map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "connection_string is invalid") .with_context("service", Scheme::Mysql) .set_source(err) })?; let table = match self.config.table { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "table is empty") .with_context("service", Scheme::Mysql)) } }; let key_field = self.config.key_field.unwrap_or_else(|| "key".to_string()); let value_field = self .config .value_field .unwrap_or_else(|| "value".to_string()); let root = normalize_root(self.config.root.unwrap_or_else(|| "/".to_string()).as_str()); Ok(MySqlBackend::new(Adapter { pool: OnceCell::new(), config, table, key_field, value_field, }) .with_normalized_root(root)) } } /// Backend for mysql service pub type MySqlBackend = kv::Backend; #[derive(Debug, Clone)] pub struct Adapter { pool: OnceCell, config: MySqlConnectOptions, table: String, key_field: String, value_field: String, } impl Adapter { async fn get_client(&self) -> Result<&MySqlPool> { self.pool .get_or_try_init(|| async { let pool = MySqlPool::connect_with(self.config.clone()) .await .map_err(parse_mysql_error)?; Ok(pool) }) .await } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Mysql, &self.table, Capability { read: true, write: true, delete: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let pool = self.get_client().await?; let value: Option> = sqlx::query_scalar(&format!( "SELECT `{}` FROM `{}` WHERE `{}` = ? LIMIT 1", self.value_field, self.table, self.key_field )) .bind(path) .fetch_optional(pool) .await .map_err(parse_mysql_error)?; Ok(value.map(Buffer::from)) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let pool = self.get_client().await?; sqlx::query(&format!( r#"INSERT INTO `{}` (`{}`, `{}`) VALUES (?, ?) ON DUPLICATE KEY UPDATE `{}` = VALUES({})"#, self.table, self.key_field, self.value_field, self.value_field, self.value_field )) .bind(path) .bind(value.to_vec()) .execute(pool) .await .map_err(parse_mysql_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let pool = self.get_client().await?; sqlx::query(&format!( "DELETE FROM `{}` WHERE `{}` = ?", self.table, self.key_field )) .bind(path) .execute(pool) .await .map_err(parse_mysql_error)?; Ok(()) } } fn parse_mysql_error(err: sqlx::Error) -> Error { Error::new(ErrorKind::Unexpected, "unhandled error from mysql").set_source(err) } opendal-0.52.0/src/services/mysql/config.rs000064400000000000000000000046511046102023000167320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Mysql services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct MysqlConfig { /// This connection string is used to connect to the mysql service. There are url based formats. /// /// The format of connect string resembles the url format of the mysql client. /// The format is: `[scheme://][user[:[password]]@]host[:port][/schema][?attribute1=value1&attribute2=value2...` /// /// - `mysql://user@localhost` /// - `mysql://user:password@localhost` /// - `mysql://user:password@localhost:3306` /// - `mysql://user:password@localhost:3306/db` /// /// For more information, please refer to . pub connection_string: Option, /// The table name for mysql. pub table: Option, /// The key field name for mysql. pub key_field: Option, /// The value field name for mysql. pub value_field: Option, /// The root for mysql. pub root: Option, } impl Debug for MysqlConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("MysqlConfig"); if self.connection_string.is_some() { d.field("connection_string", &""); } d.field("root", &self.root) .field("table", &self.table) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .finish() } } opendal-0.52.0/src/services/mysql/docs.md000064400000000000000000000020771046102023000163710ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `connection_string`: Set the connection string of mysql server - `table`: Set the table of mysql - `key_field`: Set the key field of mysql - `value_field`: Set the value field of mysql ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Mysql; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Mysql::default() .root("/") .connection_string("mysql://you_username:your_password@127.0.0.1:5432/your_database") .table("your_table") // key field type in the table should be compatible with Rust's &str like text .key_field("key") // value field type in the table should be compatible with Rust's Vec like bytea .value_field("value"); let op = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/mysql/mod.rs000064400000000000000000000017071046102023000162430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-mysql")] mod backend; #[cfg(feature = "services-mysql")] pub use backend::MysqlBuilder as Mysql; mod config; pub use config::MysqlConfig; opendal-0.52.0/src/services/nebula_graph/backend.rs000064400000000000000000000323141046102023000203330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; #[cfg(feature = "tests")] use std::time::Duration; use std::vec; use base64::engine::general_purpose::STANDARD as BASE64; use base64::engine::Engine as _; use bb8::{PooledConnection, RunError}; use rust_nebula::{ graph::GraphQuery, HostAddress, SingleConnSessionConf, SingleConnSessionManager, }; use snowflaked::sync::Generator; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::NebulaGraphConfig; use crate::*; static GENERATOR: Generator = Generator::new(0); impl Configurator for NebulaGraphConfig { type Builder = NebulaGraphBuilder; fn into_builder(self) -> Self::Builder { NebulaGraphBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct NebulaGraphBuilder { config: NebulaGraphConfig, } impl Debug for NebulaGraphBuilder { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("MysqlBuilder"); d.field("config", &self.config).finish() } } impl NebulaGraphBuilder { /// Set the host addr of nebulagraph's graphd server pub fn host(&mut self, host: &str) -> &mut Self { if !host.is_empty() { self.config.host = Some(host.to_string()); } self } /// Set the host port of nebulagraph's graphd server pub fn port(&mut self, port: u16) -> &mut Self { self.config.port = Some(port); self } /// Set the username of nebulagraph's graphd server pub fn username(&mut self, username: &str) -> &mut Self { if !username.is_empty() { self.config.username = Some(username.to_string()); } self } /// Set the password of nebulagraph's graphd server pub fn password(&mut self, password: &str) -> &mut Self { if !password.is_empty() { self.config.password = Some(password.to_string()); } self } /// Set the space name of nebulagraph's graphd server pub fn space(&mut self, space: &str) -> &mut Self { if !space.is_empty() { self.config.space = Some(space.to_string()); } self } /// Set the tag name of nebulagraph's graphd server pub fn tag(&mut self, tag: &str) -> &mut Self { if !tag.is_empty() { self.config.tag = Some(tag.to_string()); } self } /// Set the key field name of the NebulaGraph service to read/write. /// /// Default to `key` if not specified. pub fn key_field(&mut self, key_field: &str) -> &mut Self { if !key_field.is_empty() { self.config.key_field = Some(key_field.to_string()); } self } /// Set the value field name of the NebulaGraph service to read/write. /// /// Default to `value` if not specified. pub fn value_field(&mut self, value_field: &str) -> &mut Self { if !value_field.is_empty() { self.config.value_field = Some(value_field.to_string()); } self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(&mut self, root: &str) -> &mut Self { if !root.is_empty() { self.config.root = Some(root.to_string()); } self } } impl Builder for NebulaGraphBuilder { const SCHEME: Scheme = Scheme::NebulaGraph; type Config = NebulaGraphConfig; fn build(self) -> Result { let host = match self.config.host.clone() { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "host is empty") .with_context("service", Scheme::NebulaGraph)) } }; let port = match self.config.port { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "port is empty") .with_context("service", Scheme::NebulaGraph)) } }; let username = match self.config.username.clone() { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "username is empty") .with_context("service", Scheme::NebulaGraph)) } }; let password = match self.config.password.clone() { Some(v) => v, None => "".to_string(), }; let space = match self.config.space.clone() { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "space is empty") .with_context("service", Scheme::NebulaGraph)) } }; let tag = match self.config.tag.clone() { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "tag is empty") .with_context("service", Scheme::NebulaGraph)) } }; let key_field = match self.config.key_field.clone() { Some(v) => v, None => "key".to_string(), }; let value_field = match self.config.value_field.clone() { Some(v) => v, None => "value".to_string(), }; let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); let mut session_config = SingleConnSessionConf::new( vec![HostAddress::new(&host, port)], username, password, Some(space), ); // NebulaGraph use fbthrift for communication. fbthrift's max_buffer_size is default 4 KB, // which is too small to store something. // So we could set max_buffer_size to 10 MB so that NebulaGraph can store files with filesize < 1 MB at least. session_config.set_buf_size(1024 * 1024); session_config.set_max_buf_size(64 * 1024 * 1024); session_config.set_max_parse_response_bytes_count(254); Ok(NebulaGraphBackend::new(Adapter { session_pool: OnceCell::new(), session_config, tag, key_field, value_field, }) .with_root(root.as_str())) } } /// Backend for NebulaGraph service pub type NebulaGraphBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { session_pool: OnceCell>, session_config: SingleConnSessionConf, tag: String, key_field: String, value_field: String, } impl Debug for Adapter { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("Adapter") .field("session_config", &self.session_config) .field("tag", &self.tag) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .finish() } } impl Adapter { async fn get_session(&self) -> Result> { let session_pool = self .session_pool .get_or_try_init(|| async { bb8::Pool::builder() .max_size(64) .build(SingleConnSessionManager::new(self.session_config.clone())) .await }) .await .map_err(|err| Error::new(ErrorKind::Unexpected, format!("{}", err)).set_temporary())?; session_pool.get().await.map_err(|err| match err { RunError::User(err) => { Error::new(ErrorKind::Unexpected, format!("{}", err)).set_temporary() } RunError::TimedOut => { Error::new(ErrorKind::Unexpected, "connection request: timeout").set_temporary() } }) } } impl kv::Adapter for Adapter { type Scanner = kv::ScanStdIter>>; fn info(&self) -> kv::Info { kv::Info::new( Scheme::NebulaGraph, &self.session_config.space.clone().unwrap(), Capability { read: true, write: true, write_total_max_size: Some(1024 * 1024), write_can_empty: true, delete: true, list: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let path = path.replace("'", "\\'").replace('"', "\\\""); let query = format!( "LOOKUP ON {} WHERE {}.{} == '{}' YIELD properties(vertex).{} AS {};", self.tag, self.tag, self.key_field, path, self.value_field, self.value_field ); let mut sess = self.get_session().await?; let result = sess .query(&query) .await .map_err(parse_nebulagraph_session_error)?; if result.is_empty() { Ok(None) } else { let row = result .get_row_values_by_index(0) .map_err(parse_nebulagraph_dataset_error)?; let value = row .get_value_by_col_name(&self.value_field) .map_err(parse_nebulagraph_dataset_error)?; let base64_str = value.as_string().map_err(parse_nebulagraph_dataset_error)?; let value_str = BASE64.decode(base64_str).map_err(|err| { Error::new(ErrorKind::Unexpected, "unhandled error from nebulagraph") .set_source(err) })?; let buf = Buffer::from(value_str); Ok(Some(buf)) } } async fn set(&self, path: &str, value: Buffer) -> Result<()> { #[cfg(feature = "tests")] let path_copy = path; self.delete(path).await?; let path = path.replace("'", "\\'").replace('"', "\\\""); let file = value.to_vec(); let file = BASE64.encode(&file); let snowflake_id: u64 = GENERATOR.generate(); let query = format!( "INSERT VERTEX {} VALUES {}:('{}', '{}');", self.tag, snowflake_id, path, file ); let mut sess = self.get_session().await?; sess.execute(&query) .await .map_err(parse_nebulagraph_session_error)?; // To pass tests, we should confirm NebulaGraph has inserted data successfully #[cfg(feature = "tests")] loop { let v = self.get(path_copy).await.unwrap(); if v.is_none() { std::thread::sleep(Duration::from_millis(1000)); } else { break; } } Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let path = path.replace("'", "\\'").replace('"', "\\\""); let query = format!( "LOOKUP ON {} WHERE {}.{} == '{}' YIELD id(vertex) AS id | DELETE VERTEX $-.id;", self.tag, self.tag, self.key_field, path ); let mut sess = self.get_session().await?; sess.execute(&query) .await .map_err(parse_nebulagraph_session_error)?; Ok(()) } async fn scan(&self, path: &str) -> Result { let path = path.replace("'", "\\'").replace('"', "\\\""); let query = format!( "LOOKUP ON {} WHERE {}.{} STARTS WITH '{}' YIELD properties(vertex).{} AS {};", self.tag, self.tag, self.key_field, path, self.key_field, self.key_field ); let mut sess = self.get_session().await?; let result = sess .query(&query) .await .map_err(parse_nebulagraph_session_error)?; let mut res_vec = vec![]; for row_i in 0..result.get_row_size() { let row = result .get_row_values_by_index(row_i) .map_err(parse_nebulagraph_dataset_error)?; let value = row .get_value_by_col_name(&self.key_field) .map_err(parse_nebulagraph_dataset_error)?; let sub_path = value.as_string().map_err(parse_nebulagraph_dataset_error)?; res_vec.push(Ok(sub_path)); } Ok(kv::ScanStdIter::new(res_vec.into_iter())) } } fn parse_nebulagraph_session_error(err: rust_nebula::SingleConnSessionError) -> Error { Error::new(ErrorKind::Unexpected, "unhandled error from nebulagraph").set_source(err) } fn parse_nebulagraph_dataset_error(err: rust_nebula::DataSetError) -> Error { Error::new(ErrorKind::Unexpected, "unhandled error from nebulagraph").set_source(err) } opendal-0.52.0/src/services/nebula_graph/config.rs000064400000000000000000000045461046102023000202170ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// Config for Mysql services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct NebulaGraphConfig { /// The host addr of nebulagraph's graphd server pub host: Option, /// The host port of nebulagraph's graphd server pub port: Option, /// The username of nebulagraph's graphd server pub username: Option, /// The password of nebulagraph's graphd server pub password: Option, /// The space name of nebulagraph's graphd server pub space: Option, /// The tag name of nebulagraph's graphd server pub tag: Option, /// The key field name of the NebulaGraph service to read/write. pub key_field: Option, /// The value field name of the NebulaGraph service to read/write. pub value_field: Option, /// The root for NebulaGraph pub root: Option, } impl Debug for NebulaGraphConfig { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("NebulaGraphConfig"); d.field("host", &self.host) .field("port", &self.port) .field("username", &self.username) .field("password", &self.password) .field("space", &self.space) .field("tag", &self.tag) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .field("root", &self.root) .finish() } } opendal-0.52.0/src/services/nebula_graph/docs.md000064400000000000000000000024621046102023000176510ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [x] list - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `host`: Set the host address of NebulaGraph's graphd server - `port`: Set the port of NebulaGraph's graphd server - `username`: Set the username of NebulaGraph's graphd server - `password`: Set the password of NebulaGraph's graphd server - `space`: Set the passspaceword of NebulaGraph - `tag`: Set the tag of NebulaGraph - `key_field`: Set the key_field of NebulaGraph - `value_field`: Set the value_field of NebulaGraph ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::NebulaGraph; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = NebulaGraph::default(); builder.root("/"); builder.host("127.0.0.1"); builder.port(9669); builder.space("your_space"); builder.tag("your_tag"); // key field type in the table should be compatible with Rust's &str like text builder.key_field("key"); // value field type in the table should be compatible with Rust's Vec like bytea builder.value_field("value"); let op = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/nebula_graph/mod.rs000064400000000000000000000017471046102023000175310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-nebula-graph")] mod backend; #[cfg(feature = "services-nebula-graph")] pub use backend::NebulaGraphBuilder as NebulaGraph; mod config; pub use config::NebulaGraphConfig; opendal-0.52.0/src/services/obs/backend.rs000064400000000000000000000340531046102023000164710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use http::Uri; use log::debug; use reqsign::HuaweicloudObsConfig; use reqsign::HuaweicloudObsCredentialLoader; use reqsign::HuaweicloudObsSigner; use super::core::{constants, ObsCore}; use super::delete::ObsDeleter; use super::error::parse_error; use super::lister::ObsLister; use super::writer::ObsWriter; use super::writer::ObsWriters; use crate::raw::*; use crate::services::ObsConfig; use crate::*; impl Configurator for ObsConfig { type Builder = ObsBuilder; fn into_builder(self) -> Self::Builder { ObsBuilder { config: self, http_client: None, } } } /// Huawei-Cloud Object Storage Service (OBS) support #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct ObsBuilder { config: ObsConfig, http_client: Option, } impl Debug for ObsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("ObsBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl ObsBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set endpoint of this backend. /// /// Both huaweicloud default domain and user domain endpoints are allowed. /// Please DO NOT add the bucket name to the endpoint. /// /// - `https://obs.cn-north-4.myhuaweicloud.com` /// - `obs.cn-north-4.myhuaweicloud.com` (https by default) /// - `https://custom.obs.com` (port should not be set) pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()); } self } /// Set access_key_id of this backend. /// - If it is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn access_key_id(mut self, access_key_id: &str) -> Self { if !access_key_id.is_empty() { self.config.access_key_id = Some(access_key_id.to_string()); } self } /// Set secret_access_key of this backend. /// - If it is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn secret_access_key(mut self, secret_access_key: &str) -> Self { if !secret_access_key.is_empty() { self.config.secret_access_key = Some(secret_access_key.to_string()); } self } /// Set bucket of this backend. /// The param is required. pub fn bucket(mut self, bucket: &str) -> Self { if !bucket.is_empty() { self.config.bucket = Some(bucket.to_string()); } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for ObsBuilder { const SCHEME: Scheme = Scheme::Obs; type Config = ObsConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let bucket = match &self.config.bucket { Some(bucket) => Ok(bucket.to_string()), None => Err( Error::new(ErrorKind::ConfigInvalid, "The bucket is misconfigured") .with_context("service", Scheme::Obs), ), }?; debug!("backend use bucket {}", &bucket); let uri = match &self.config.endpoint { Some(endpoint) => endpoint.parse::().map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "endpoint is invalid") .with_context("service", Scheme::Obs) .set_source(err) }), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_context("service", Scheme::Obs)), }?; let scheme = match uri.scheme_str() { Some(scheme) => scheme.to_string(), None => "https".to_string(), }; let (endpoint, is_obs_default) = { let host = uri.host().unwrap_or_default().to_string(); if host.starts_with("obs.") && (host.ends_with(".myhuaweicloud.com") || host.ends_with(".huawei.com")) { (format!("{bucket}.{host}"), true) } else { (host, false) } }; debug!("backend use endpoint {}", &endpoint); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Obs) })? }; let mut cfg = HuaweicloudObsConfig::default(); // Load cfg from env first. cfg = cfg.from_env(); if let Some(v) = self.config.access_key_id { cfg.access_key_id = Some(v); } if let Some(v) = self.config.secret_access_key { cfg.secret_access_key = Some(v); } let loader = HuaweicloudObsCredentialLoader::new(cfg); // Set the bucket name in CanonicalizedResource. // 1. If the bucket is bound to a user domain name, use the user domain name as the bucket name, // for example, `/obs.ccc.com/object`. `obs.ccc.com` is the user domain name bound to the bucket. // 2. If you do not access OBS using a user domain name, this field is in the format of `/bucket/object`. // // Please refer to this doc for more details: // https://support.huaweicloud.com/intl/en-us/api-obs/obs_04_0010.html let signer = HuaweicloudObsSigner::new({ if is_obs_default { &bucket } else { &endpoint } }); debug!("backend build finished"); Ok(ObsBackend { core: Arc::new(ObsCore { bucket, root, endpoint: format!("{}://{}", &scheme, &endpoint), signer, loader, client, }), }) } } /// Backend for Huaweicloud OBS services. #[derive(Debug, Clone)] pub struct ObsBackend { core: Arc, } impl Access for ObsBackend { type Reader = HttpBody; type Writer = ObsWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Obs) .set_root(&self.core.root) .set_name(&self.core.bucket) .set_native_capability(Capability { stat: true, stat_with_if_match: true, stat_with_if_none_match: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, stat_has_user_metadata: true, read: true, read_with_if_match: true, read_with_if_none_match: true, write: true, write_can_empty: true, write_can_append: true, write_can_multi: true, write_with_content_type: true, write_with_cache_control: true, // The min multipart size of OBS is 5 MiB. // // ref: write_multi_min_size: Some(5 * 1024 * 1024), // The max multipart size of OBS is 5 GiB. // // ref: write_multi_max_size: if cfg!(target_pointer_width = "64") { Some(5 * 1024 * 1024 * 1024) } else { Some(usize::MAX) }, write_with_user_metadata: true, delete: true, copy: true, list: true, list_with_recursive: true, list_has_content_length: true, presign: true, presign_stat: true, presign_read: true, presign_write: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, args: OpStat) -> Result { let resp = self.core.obs_head_object(path, &args).await?; let headers = resp.headers(); let status = resp.status(); // The response is very similar to azblob. match status { StatusCode::OK => { let mut meta = parse_into_metadata(path, headers)?; let user_meta = headers .iter() .filter_map(|(name, _)| { name.as_str() .strip_prefix(constants::X_OBS_META_PREFIX) .and_then(|stripped_key| { parse_header_to_str(headers, name) .unwrap_or(None) .map(|val| (stripped_key.to_string(), val.to_string())) }) }) .collect::>(); if !user_meta.is_empty() { meta.with_user_metadata(user_meta); } Ok(RpStat::new(meta)) } StatusCode::NOT_FOUND if path.ends_with('/') => { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.obs_get_object(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = ObsWriter::new(self.core.clone(), path, args.clone()); let w = if args.append() { ObsWriters::Two(oio::AppendWriter::new(writer)) } else { ObsWriters::One(oio::MultipartWriter::new( writer, args.executor().cloned(), args.concurrent(), )) }; Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(ObsDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = ObsLister::new(self.core.clone(), path, args.recursive(), args.limit()); Ok((RpList::default(), oio::PageLister::new(l))) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.obs_copy_object(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn presign(&self, path: &str, args: OpPresign) -> Result { let mut req = match args.operation() { PresignOperation::Stat(v) => self.core.obs_head_object_request(path, v)?, PresignOperation::Read(v) => { self.core .obs_get_object_request(path, BytesRange::default(), v)? } PresignOperation::Write(v) => { self.core .obs_put_object_request(path, None, v, Buffer::new())? } }; self.core.sign_query(&mut req, args.expire()).await?; // We don't need this request anymore, consume it directly. let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } } opendal-0.52.0/src/services/obs/config.rs000064400000000000000000000034041046102023000163430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Huawei-Cloud Object Storage Service (OBS) support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct ObsConfig { /// Root for obs. pub root: Option, /// Endpoint for obs. pub endpoint: Option, /// Access key id for obs. pub access_key_id: Option, /// Secret access key for obs. pub secret_access_key: Option, /// Bucket for obs. pub bucket: Option, } impl Debug for ObsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("ObsConfig") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("access_key_id", &"") .field("secret_access_key", &"") .field("bucket", &self.bucket) .finish() } } opendal-0.52.0/src/services/obs/core.rs000064400000000000000000000410071046102023000160270ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::time::Duration; use bytes::Bytes; use http::header::CACHE_CONTROL; use http::header::CONTENT_DISPOSITION; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::header::IF_MATCH; use http::header::IF_NONE_MATCH; use http::Request; use http::Response; use reqsign::HuaweicloudObsCredential; use reqsign::HuaweicloudObsCredentialLoader; use reqsign::HuaweicloudObsSigner; use serde::Deserialize; use serde::Serialize; use crate::raw::*; use crate::*; pub mod constants { pub const X_OBS_META_PREFIX: &str = "x-obs-meta-"; } pub struct ObsCore { pub bucket: String, pub root: String, pub endpoint: String, pub signer: HuaweicloudObsSigner, pub loader: HuaweicloudObsCredentialLoader, pub client: HttpClient, } impl Debug for ObsCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .finish_non_exhaustive() } } impl ObsCore { async fn load_credential(&self) -> Result> { let cred = self .loader .load() .await .map_err(new_request_credential_error)?; if let Some(cred) = cred { Ok(Some(cred)) } else { Ok(None) } } pub async fn sign(&self, req: &mut Request) -> Result<()> { let cred = if let Some(cred) = self.load_credential().await? { cred } else { return Ok(()); }; self.signer.sign(req, &cred).map_err(new_request_sign_error) } pub async fn sign_query(&self, req: &mut Request, duration: Duration) -> Result<()> { let cred = if let Some(cred) = self.load_credential().await? { cred } else { return Ok(()); }; self.signer .sign_query(req, duration, &cred) .map_err(new_request_sign_error) } #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } } impl ObsCore { pub async fn obs_get_object( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let mut req = self.obs_get_object_request(path, range, args)?; self.sign(&mut req).await?; self.client.fetch(req).await } pub fn obs_get_object_request( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let mut req = Request::get(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if range.is_full() { req = req.header(http::header::RANGE, range.to_header()) } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub fn obs_put_object_request( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let mut req = Request::put(&url); if let Some(size) = size { req = req.header(CONTENT_LENGTH, size) } if let Some(cache_control) = args.cache_control() { req = req.header(CACHE_CONTROL, cache_control) } if let Some(mime) = args.content_type() { req = req.header(CONTENT_TYPE, mime) } // Set user metadata headers. if let Some(user_metadata) = args.user_metadata() { for (key, value) in user_metadata { req = req.header(format!("{}{}", constants::X_OBS_META_PREFIX, key), value) } } let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn obs_head_object(&self, path: &str, args: &OpStat) -> Result> { let mut req = self.obs_head_object_request(path, args)?; self.sign(&mut req).await?; self.send(req).await } pub fn obs_head_object_request(&self, path: &str, args: &OpStat) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); // The header 'Origin' is optional for API calling, the doc has mistake, confirmed with customer service of huaweicloud. // https://support.huaweicloud.com/intl/en-us/api-obs/obs_04_0084.html let mut req = Request::head(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub async fn obs_delete_object(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let req = Request::delete(&url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub fn obs_append_object_request( &self, path: &str, position: u64, size: u64, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?append&position={}", self.endpoint, percent_encode_path(&p), position ); let mut req = Request::post(&url); req = req.header(CONTENT_LENGTH, size); if let Some(mime) = args.content_type() { req = req.header(CONTENT_TYPE, mime); } if let Some(pos) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, pos); } if let Some(cache_control) = args.cache_control() { req = req.header(CACHE_CONTROL, cache_control) } let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn obs_copy_object(&self, from: &str, to: &str) -> Result> { let source = build_abs_path(&self.root, from); let target = build_abs_path(&self.root, to); let source = format!("/{}/{}", self.bucket, percent_encode_path(&source)); let url = format!("{}/{}", self.endpoint, percent_encode_path(&target)); let mut req = Request::put(&url) .header("x-obs-copy-source", &source) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn obs_list_objects( &self, path: &str, next_marker: &str, delimiter: &str, limit: Option, ) -> Result> { let p = build_abs_path(&self.root, path); let mut queries = vec![]; if !path.is_empty() { queries.push(format!("prefix={}", percent_encode_path(&p))); } if !delimiter.is_empty() { queries.push(format!("delimiter={delimiter}")); } if let Some(limit) = limit { queries.push(format!("max-keys={limit}")); } if !next_marker.is_empty() { queries.push(format!("marker={next_marker}")); } let url = if queries.is_empty() { self.endpoint.to_string() } else { format!("{}?{}", self.endpoint, queries.join("&")) }; let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn obs_initiate_multipart_upload( &self, path: &str, content_type: Option<&str>, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}?uploads", self.endpoint, percent_encode_path(&p)); let mut req = Request::post(&url); if let Some(mime) = content_type { req = req.header(CONTENT_TYPE, mime) } let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn obs_upload_part_request( &self, path: &str, upload_id: &str, part_number: usize, size: Option, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?partNumber={}&uploadId={}", self.endpoint, percent_encode_path(&p), part_number, percent_encode_path(upload_id) ); let mut req = Request::put(&url); if let Some(size) = size { req = req.header(CONTENT_LENGTH, size); } // Set body let mut req = req.body(body).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn obs_complete_multipart_upload( &self, path: &str, upload_id: &str, parts: Vec, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?uploadId={}", self.endpoint, percent_encode_path(&p), percent_encode_path(upload_id) ); let req = Request::post(&url); let content = quick_xml::se::to_string(&CompleteMultipartUploadRequest { part: parts.to_vec(), }) .map_err(new_xml_deserialize_error)?; // Make sure content length has been set to avoid post with chunked encoding. let req = req.header(CONTENT_LENGTH, content.len()); // Set content-type to `application/xml` to avoid mixed with form post. let req = req.header(CONTENT_TYPE, "application/xml"); let mut req = req .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } /// Abort an on-going multipart upload. pub async fn obs_abort_multipart_upload( &self, path: &str, upload_id: &str, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?uploadId={}", self.endpoint, percent_encode_path(&p), percent_encode_path(upload_id) ); let mut req = Request::delete(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } } /// Result of CreateMultipartUpload #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct InitiateMultipartUploadResult { pub upload_id: String, } /// Request of CompleteMultipartUploadRequest #[derive(Default, Debug, Serialize)] #[serde(default, rename = "CompleteMultipartUpload", rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequest { pub part: Vec, } #[derive(Clone, Default, Debug, Serialize)] #[serde(default, rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequestPart { #[serde(rename = "PartNumber")] pub part_number: usize, /// /// /// quick-xml will do escape on `"` which leads to our serialized output is /// not the same as aws s3's example. /// /// Ideally, we could use `serialize_with` to address this (buf failed) /// /// ```ignore /// #[derive(Default, Debug, Serialize)] /// #[serde(default, rename_all = "PascalCase")] /// struct CompleteMultipartUploadRequestPart { /// #[serde(rename = "PartNumber")] /// part_number: usize, /// #[serde(rename = "ETag", serialize_with = "partial_escape")] /// etag: String, /// } /// /// fn partial_escape(s: &str, ser: S) -> Result /// where /// S: serde::Serializer, /// { /// ser.serialize_str(&String::from_utf8_lossy( /// &quick_xml::escape::partial_escape(s.as_bytes()), /// )) /// } /// ``` /// /// ref: #[serde(rename = "ETag")] pub etag: String, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectsOutput { pub name: String, pub prefix: String, pub contents: Vec, pub common_prefixes: Vec, pub marker: String, pub next_marker: Option, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct CommonPrefix { pub prefix: String, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectsOutputContent { pub key: String, pub size: u64, } #[cfg(test)] mod tests { use bytes::Buf; use super::*; #[test] fn test_parse_xml() { let bs = bytes::Bytes::from( r#" examplebucket obj obj002 obj004 1000 false obj002 2015-07-01T02:11:19.775Z "a72e382246ac83e86bd203389849e71d" 9 b4bf1b36d9ca43d984fbcb9491b6fce9 STANDARD obj003 2015-07-01T02:11:19.775Z "a72e382246ac83e86bd203389849e71d" 10 b4bf1b36d9ca43d984fbcb9491b6fce9 STANDARD hello world "#, ); let out: ListObjectsOutput = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert_eq!(out.name, "examplebucket".to_string()); assert_eq!(out.prefix, "obj".to_string()); assert_eq!(out.marker, "obj002".to_string()); assert_eq!(out.next_marker, Some("obj004".to_string()),); assert_eq!( out.contents .iter() .map(|v| v.key.clone()) .collect::>(), ["obj002", "obj003"], ); assert_eq!( out.contents.iter().map(|v| v.size).collect::>(), [9, 10], ); assert_eq!( out.common_prefixes .iter() .map(|v| v.prefix.clone()) .collect::>(), ["hello", "world"], ) } } opendal-0.52.0/src/services/obs/delete.rs000064400000000000000000000026711046102023000163450ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct ObsDeleter { core: Arc, } impl ObsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for ObsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.obs_delete_object(&path).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT | StatusCode::ACCEPTED | StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/obs/docs.md000064400000000000000000000024131046102023000160010ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [x] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `bucket`: Set the container name for backend - `endpoint`: Customizable endpoint setting - `access_key_id`: Set the access_key_id for backend. - `secret_access_key`: Set the secret_access_key for backend. You can refer to [`ObsBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Obs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Obs::default() // set the storage bucket for OpenDAL .bucket("test") .endpoint("obs.cn-north-1.myhuaweicloud.com") // Set the access_key_id and secret_access_key. // // OpenDAL will try load credential from the env. // If credential not set and no valid credential in env, OpenDAL will // send request without signing like anonymous user. .access_key_id("access_key_id") .secret_access_key("secret_access_key"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/obs/error.rs000064400000000000000000000065411046102023000162340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use http::StatusCode; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// ObsError is the error returned by obs service. #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] struct ObsError { code: String, message: String, resource: String, request_id: String, host_id: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::NOT_MODIFIED => { (ErrorKind::ConditionNotMatch, false) } StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), // OBS could return `520 Origin Error` errors which should be retried. v if v.as_u16() == 520 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = match de::from_reader::<_, ObsError>(bs.clone().reader()) { Ok(obs_error) => format!("{obs_error:?}"), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod tests { use super::*; #[test] fn test_parse_error() { let bs = bytes::Bytes::from( r#" NoSuchKey The resource you requested does not exist /example-bucket/object 001B21A61C6C0000013402C4616D5285 RkRCRDJENDc5MzdGQkQ4OUY3MTI4NTQ3NDk2Mjg0M0FBQUFBQUFBYmJiYmJiYmJD "#, ); let out: ObsError = de::from_reader(bs.reader()).expect("must success"); println!("{out:?}"); assert_eq!(out.code, "NoSuchKey"); assert_eq!(out.message, "The resource you requested does not exist"); assert_eq!(out.resource, "/example-bucket/object"); assert_eq!(out.request_id, "001B21A61C6C0000013402C4616D5285"); assert_eq!( out.host_id, "RkRCRDJENDc5MzdGQkQ4OUY3MTI4NTQ3NDk2Mjg0M0FBQUFBQUFBYmJiYmJiYmJD" ); } } opendal-0.52.0/src/services/obs/lister.rs000064400000000000000000000056641046102023000164120ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use quick_xml::de; use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::EntryMode; use crate::Metadata; use crate::Result; pub struct ObsLister { core: Arc, path: String, delimiter: &'static str, limit: Option, } impl ObsLister { pub fn new(core: Arc, path: &str, recursive: bool, limit: Option) -> Self { let delimiter = if recursive { "" } else { "/" }; Self { core, path: path.to_string(), delimiter, limit, } } } impl oio::PageList for ObsLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .obs_list_objects(&self.path, &ctx.token, self.delimiter, self.limit) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let output: ListObjectsOutput = de::from_reader(bs.reader()).map_err(new_xml_deserialize_error)?; // Try our best to check whether this list is done. // // - Check `next_marker` ctx.done = match output.next_marker.as_ref() { None => true, Some(next_marker) => next_marker.is_empty(), }; ctx.token = output.next_marker.clone().unwrap_or_default(); let common_prefixes = output.common_prefixes; for prefix in common_prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix.prefix), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de); } for object in output.contents { let path = build_rel_path(&self.core.root, &object.key); if path == self.path || path.is_empty() { continue; } let meta = Metadata::new(EntryMode::from_path(&path)).with_content_length(object.size); let de = oio::Entry::with(path, meta); ctx.entries.push_back(de); } Ok(()) } } opendal-0.52.0/src/services/obs/mod.rs000064400000000000000000000022341046102023000156550ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-obs")] mod core; #[cfg(feature = "services-obs")] mod delete; #[cfg(feature = "services-obs")] mod error; #[cfg(feature = "services-obs")] mod lister; #[cfg(feature = "services-obs")] mod writer; #[cfg(feature = "services-obs")] mod backend; #[cfg(feature = "services-obs")] pub use backend::ObsBuilder as Obs; mod config; pub use config::ObsConfig; opendal-0.52.0/src/services/obs/writer.rs000064400000000000000000000134501046102023000164140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::*; use super::error::parse_error; use crate::raw::oio::MultipartPart; use crate::raw::*; use crate::*; pub type ObsWriters = TwoWays, oio::AppendWriter>; pub struct ObsWriter { core: Arc, op: OpWrite, path: String, } impl ObsWriter { pub fn new(core: Arc, path: &str, op: OpWrite) -> Self { ObsWriter { core, path: path.to_string(), op, } } } impl oio::MultipartWrite for ObsWriter { async fn write_once(&self, size: u64, body: Buffer) -> Result { let mut req = self .core .obs_put_object_request(&self.path, Some(size), &self.op, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn initiate_part(&self) -> Result { let resp = self .core .obs_initiate_multipart_upload(&self.path, self.op.content_type()) .await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let result: InitiateMultipartUploadResult = quick_xml::de::from_reader(bytes::Buf::reader(bs)) .map_err(new_xml_deserialize_error)?; Ok(result.upload_id) } _ => Err(parse_error(resp)), } } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result { // Obs service requires part number must between [1..=10000] let part_number = part_number + 1; let resp = self .core .obs_upload_part_request(&self.path, upload_id, part_number, Some(size), body) .await?; let status = resp.status(); match status { StatusCode::OK => { let etag = parse_etag(resp.headers())? .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "ETag not present in returning response", ) })? .to_string(); Ok(MultipartPart { part_number, etag, checksum: None, }) } _ => Err(parse_error(resp)), } } async fn complete_part(&self, upload_id: &str, parts: &[MultipartPart]) -> Result { let parts = parts .iter() .map(|p| CompleteMultipartUploadRequestPart { part_number: p.part_number, etag: p.etag.clone(), }) .collect(); let resp = self .core .obs_complete_multipart_upload(&self.path, upload_id, parts) .await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn abort_part(&self, upload_id: &str) -> Result<()> { let resp = self .core .obs_abort_multipart_upload(&self.path, upload_id) .await?; match resp.status() { // Obs returns code 204 No Content if abort succeeds. // Reference: https://support.huaweicloud.com/intl/en-us/api-obs/obs_04_0103.html StatusCode::NO_CONTENT => Ok(()), _ => Err(parse_error(resp)), } } } impl oio::AppendWrite for ObsWriter { async fn offset(&self) -> Result { let resp = self .core .obs_head_object(&self.path, &OpStat::default()) .await?; let status = resp.status(); match status { StatusCode::OK => { let content_length = parse_content_length(resp.headers())?.ok_or_else(|| { Error::new( ErrorKind::Unexpected, "Content-Length not present in returning response", ) })?; Ok(content_length) } StatusCode::NOT_FOUND => Ok(0), _ => Err(parse_error(resp)), } } async fn append(&self, offset: u64, size: u64, body: Buffer) -> Result { let mut req = self .core .obs_append_object_request(&self.path, offset, size, &self.op, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/onedrive/backend.rs000064400000000000000000000276171046102023000175310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::sync::Arc; use bytes::Buf; use bytes::Bytes; use http::header; use http::Request; use http::Response; use http::StatusCode; use super::delete::OnedriveDeleter; use super::error::parse_error; use super::graph_model::CreateDirPayload; use super::graph_model::ItemType; use super::graph_model::OneDriveUploadSessionCreationRequestBody; use super::graph_model::OnedriveGetItemBody; use super::lister::OnedriveLister; use super::writer::OneDriveWriter; use crate::raw::*; use crate::*; #[derive(Clone)] pub struct OnedriveBackend { root: String, access_token: String, client: HttpClient, } impl OnedriveBackend { pub(crate) fn new(root: String, access_token: String, http_client: HttpClient) -> Self { Self { root, access_token, client: http_client, } } } impl Debug for OnedriveBackend { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("OneDriveBackend"); de.field("root", &self.root); de.field("access_token", &""); de.finish() } } impl Access for OnedriveBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut ma = AccessorInfo::default(); ma.set_scheme(Scheme::Onedrive) .set_root(&self.root) .set_native_capability(Capability { read: true, write: true, stat: true, stat_has_etag: true, stat_has_last_modified: true, stat_has_content_length: true, delete: true, create_dir: true, list: true, shared: true, ..Default::default() }); ma.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let path = build_rooted_abs_path(&self.root, path); let path_before_last_slash = get_parent(&path); let encoded_path = percent_encode_path(path_before_last_slash); let uri = format!( "https://graph.microsoft.com/v1.0/me/drive/root:{}:/children", encoded_path ); let folder_name = get_basename(&path); let folder_name = folder_name.strip_suffix('/').unwrap_or(folder_name); let body = CreateDirPayload::new(folder_name.to_string()); let response = self.onedrive_create_dir(&uri, body).await?; let status = response.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(RpCreateDir::default()), _ => Err(parse_error(response)), } } async fn stat(&self, path: &str, _: OpStat) -> Result { // Stat root always returns a DIR. if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let resp = self.onedrive_get_stat(path).await?; let status = resp.status(); if status.is_success() { let bytes = resp.into_body(); let decoded_response: OnedriveGetItemBody = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; let entry_mode: EntryMode = match decoded_response.item_type { ItemType::Folder { .. } => EntryMode::DIR, ItemType::File { .. } => EntryMode::FILE, }; let mut meta = Metadata::new(entry_mode); meta.set_etag(&decoded_response.e_tag); let last_modified = decoded_response.last_modified_date_time; let date_utc_last_modified = parse_datetime_from_rfc3339(&last_modified)?; meta.set_last_modified(date_utc_last_modified); meta.set_content_length(decoded_response.size); Ok(RpStat::new(meta)) } else { match status { StatusCode::NOT_FOUND if path.ends_with('/') => { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } _ => Err(parse_error(resp)), } } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.onedrive_get_content(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let path = build_rooted_abs_path(&self.root, path); Ok(( RpWrite::default(), oio::OneShotWriter::new(OneDriveWriter::new(self.clone(), args, path)), )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(OnedriveDeleter::new(Arc::new(self.clone()))), )) } async fn list(&self, path: &str, _op_list: OpList) -> Result<(RpList, Self::Lister)> { let l = OnedriveLister::new(self.root.clone(), path.into(), self.clone()); Ok((RpList::default(), oio::PageLister::new(l))) } } impl OnedriveBackend { pub(crate) const BASE_URL: &'static str = "https://graph.microsoft.com/v1.0/me"; async fn onedrive_get_stat(&self, path: &str) -> Result> { let path = build_rooted_abs_path(&self.root, path); let url: String = format!( "https://graph.microsoft.com/v1.0/me/drive/root:{}{}", percent_encode_path(&path), "" ); let mut req = Request::get(&url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub(crate) async fn onedrive_get_next_list_page(&self, url: &str) -> Result> { let mut req = Request::get(url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn onedrive_get_content( &self, path: &str, range: BytesRange, ) -> Result> { let path = build_rooted_abs_path(&self.root, path); let url: String = format!( "https://graph.microsoft.com/v1.0/me/drive/root:{}{}", percent_encode_path(&path), ":/content" ); let mut req = Request::get(&url).header(header::RANGE, range.to_header()); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn onedrive_upload_simple( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let url = format!( "https://graph.microsoft.com/v1.0/me/drive/root:{}:/content", percent_encode_path(path) ); let mut req = Request::put(&url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); if let Some(size) = size { req = req.header(header::CONTENT_LENGTH, size) } if let Some(mime) = args.content_type() { req = req.header(header::CONTENT_TYPE, mime) } let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub(crate) async fn onedrive_chunked_upload( &self, url: &str, args: &OpWrite, offset: usize, chunk_end: usize, total_len: usize, body: Buffer, ) -> Result> { let mut req = Request::put(url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); let range = format!("bytes {}-{}/{}", offset, chunk_end, total_len); req = req.header("Content-Range".to_string(), range); let size = chunk_end - offset + 1; req = req.header(header::CONTENT_LENGTH, size.to_string()); if let Some(mime) = args.content_type() { req = req.header(header::CONTENT_TYPE, mime) } let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub(crate) async fn onedrive_create_upload_session( &self, url: &str, body: OneDriveUploadSessionCreationRequestBody, ) -> Result> { let mut req = Request::post(url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); req = req.header(header::CONTENT_TYPE, "application/json"); let body_bytes = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let asyn_body = Buffer::from(Bytes::from(body_bytes)); let req = req.body(asyn_body).map_err(new_request_build_error)?; self.client.send(req).await } async fn onedrive_create_dir( &self, url: &str, body: CreateDirPayload, ) -> Result> { let mut req = Request::post(url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); req = req.header(header::CONTENT_TYPE, "application/json"); let body_bytes = serde_json::to_vec(&body).map_err(new_json_serialize_error)?; let async_body = Buffer::from(bytes::Bytes::from(body_bytes)); let req = req.body(async_body).map_err(new_request_build_error)?; self.client.send(req).await } pub(crate) async fn onedrive_delete(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://graph.microsoft.com/v1.0/me/drive/root:/{}", percent_encode_path(&path) ); let mut req = Request::delete(&url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } } opendal-0.52.0/src/services/onedrive/builder.rs000064400000000000000000000064351046102023000175630ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use log::debug; use super::backend::OnedriveBackend; use crate::raw::normalize_root; use crate::raw::Access; use crate::raw::HttpClient; use crate::services::OnedriveConfig; use crate::Scheme; use crate::*; impl Configurator for OnedriveConfig { type Builder = OnedriveBuilder; fn into_builder(self) -> Self::Builder { OnedriveBuilder { config: self, http_client: None, } } } /// [OneDrive](https://onedrive.com) backend support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct OnedriveBuilder { config: OnedriveConfig, http_client: Option, } impl Debug for OnedriveBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("config", &self.config) .finish() } } impl OnedriveBuilder { /// set the bearer access token for OneDrive /// /// default: no access token, which leads to failure pub fn access_token(mut self, access_token: &str) -> Self { self.config.access_token = Some(access_token.to_string()); self } /// Set root path of OneDrive folder. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, http_client: HttpClient) -> Self { self.http_client = Some(http_client); self } } impl Builder for OnedriveBuilder { const SCHEME: Scheme = Scheme::Onedrive; type Config = OnedriveConfig; fn build(self) -> Result { let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Onedrive) })? }; match self.config.access_token.clone() { Some(access_token) => Ok(OnedriveBackend::new(root, access_token, client)), None => Err(Error::new(ErrorKind::ConfigInvalid, "access_token not set")), } } } opendal-0.52.0/src/services/onedrive/config.rs000064400000000000000000000026641046102023000174020ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for [OneDrive](https://onedrive.com) backend support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct OnedriveConfig { /// bearer access token for OneDrive pub access_token: Option, /// root path of OneDrive folder. pub root: Option, } impl Debug for OnedriveConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("OnedriveConfig") .field("root", &self.root) .finish_non_exhaustive() } } opendal-0.52.0/src/services/onedrive/delete.rs000064400000000000000000000031421046102023000173670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::backend::OnedriveBackend; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; /// Delete operation /// Documentation: https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_delete?view=odsp-graph-online pub struct OnedriveDeleter { core: Arc, } impl OnedriveDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for OnedriveDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.onedrive_delete(&path).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT | StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/onedrive/docs.md000064400000000000000000000013621046102023000170330ustar 00000000000000## Capabilities This service can be used to: - [x] read - [x] write - [x] list - [ ] copy - [ ] rename - [ ] ~~presign~~ - [ ] blocking ## Notes Currently, only OneDrive Personal is supported. ## Configuration - `access_token`: set the access_token for Graph API - `root`: Set the work directory for backend You can refer to [`OnedriveBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Onedrive; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Onedrive::default() .access_token("xxx") .root("/path/to/root"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } opendal-0.52.0/src/services/onedrive/error.rs000064400000000000000000000032371046102023000172630ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use crate::raw::*; use crate::*; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = String::from_utf8_lossy(&bs); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/onedrive/graph_model.rs000064400000000000000000000215011046102023000204050ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use serde::Deserialize; use serde::Serialize; #[derive(Debug, Serialize, Deserialize)] pub struct GraphApiOnedriveListResponse { #[serde(rename = "@odata.nextLink")] pub next_link: Option, pub value: Vec, } /// DriveItem representation /// https://learn.microsoft.com/en-us/onedrive/developer/rest-api/resources/list?view=odsp-graph-online#json-representation #[derive(Debug, Serialize, Deserialize)] pub struct OneDriveItem { pub name: String, #[serde(rename = "parentReference")] pub parent_reference: ParentReference, #[serde(flatten)] pub item_type: ItemType, } #[derive(Debug, Serialize, Deserialize)] pub struct ParentReference { pub path: String, } #[derive(Debug, Serialize, Deserialize, PartialEq, Clone)] #[serde(untagged)] pub enum ItemType { Folder { folder: Folder, #[serde(rename = "specialFolder")] special_folder: Option>, }, File { file: File, }, } #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct OnedriveGetItemBody { #[serde(rename = "cTag")] pub(crate) c_tag: String, #[serde(rename = "eTag")] pub(crate) e_tag: String, id: String, #[serde(rename = "lastModifiedDateTime")] pub(crate) last_modified_date_time: String, pub(crate) name: String, pub(crate) size: u64, #[serde(flatten)] pub(crate) item_type: ItemType, } #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct File { #[serde(rename = "mimeType")] mime_type: String, } #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct Folder { #[serde(rename = "childCount")] child_count: i64, } #[derive(Debug, Clone, Serialize, Deserialize)] pub struct CreateDirPayload { // folder: String, #[serde(rename = "@microsoft.graph.conflictBehavior")] conflict_behavior: String, name: String, folder: EmptyStruct, } impl CreateDirPayload { pub fn new(name: String) -> Self { Self { conflict_behavior: "replace".to_string(), name, folder: EmptyStruct {}, } } } #[derive(Debug, Clone, Serialize, Deserialize)] struct EmptyStruct {} #[derive(Debug, Clone, Serialize, Deserialize)] struct FileUploadItem { #[serde(rename = "@odata.type")] odata_type: String, #[serde(rename = "@microsoft.graph.conflictBehavior")] microsoft_graph_conflict_behavior: String, name: String, } #[derive(Debug, Clone, Serialize, Deserialize)] pub struct OneDriveUploadSessionCreationResponseBody { #[serde(rename = "uploadUrl")] pub upload_url: String, #[serde(rename = "expirationDateTime")] pub expiration_date_time: String, } #[derive(Debug, Clone, Serialize, Deserialize)] pub struct OneDriveUploadSessionCreationRequestBody { item: FileUploadItem, } impl OneDriveUploadSessionCreationRequestBody { pub fn new(path: String) -> Self { OneDriveUploadSessionCreationRequestBody { item: FileUploadItem { odata_type: "microsoft.graph.driveItemUploadableProperties".to_string(), microsoft_graph_conflict_behavior: "replace".to_string(), name: path, }, } } } #[test] fn test_parse_one_drive_json() { let data = r#"{ "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('user_id')/drive/root/children", "value": [ { "createdDateTime": "2020-01-01T00:00:00Z", "cTag": "cTag", "eTag": "eTag", "id": "id", "lastModifiedDateTime": "2020-01-01T00:00:00Z", "name": "name", "size": 0, "webUrl": "webUrl", "reactions": { "like": 0 }, "parentReference": { "driveId": "driveId", "driveType": "driveType", "id": "id", "path": "/drive/root:" }, "fileSystemInfo": { "createdDateTime": "2020-01-01T00:00:00Z", "lastModifiedDateTime": "2020-01-01T00:00:00Z" }, "folder": { "childCount": 0 }, "specialFolder": { "name": "name" } }, { "createdDateTime": "2018-12-30T05:32:55.46Z", "cTag": "sample", "eTag": "sample", "id": "ID!102", "lastModifiedDateTime": "2018-12-30T05:33:23.557Z", "name": "Getting started with OneDrive.pdf", "size": 1025867, "reactions": { "commentCount": 0 }, "createdBy": { "user": { "displayName": "Foo bar", "id": "ID" } }, "lastModifiedBy": { "user": { "displayName": "Foo bar", "id": "32217fc1154aec3d" } }, "parentReference": { "driveId": "32217fc1154aec3d", "driveType": "personal", "id": "32217FC1154AEC3D!101", "path": "/drive/root:" }, "file": { "mimeType": "application/pdf" }, "fileSystemInfo": { "createdDateTime": "2018-12-30T05:32:55.46Z", "lastModifiedDateTime": "2018-12-30T05:32:55.46Z" } } ] }"#; let response: GraphApiOnedriveListResponse = serde_json::from_str(data).unwrap(); assert_eq!(response.value.len(), 2); let item = &response.value[0]; assert_eq!(item.name, "name"); } #[test] fn test_parse_folder_single() { let response_json = r#" { "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('great.cat%40outlook.com')/drive/root/children", "value": [ { "createdDateTime": "2023-02-01T00:51:02.803Z", "cTag": "sample", "eTag": "sample", "id": "ID!3003", "lastModifiedDateTime": "2023-02-01T00:51:10.703Z", "name": "misc", "size": 1084627, "webUrl": "sample", "reactions": { "commentCount": 0 }, "createdBy": { "application": { "displayName": "OneDrive", "id": "481710a4" }, "user": { "displayName": "Foo bar", "id": "01" } }, "lastModifiedBy": { "application": { "displayName": "OneDrive", "id": "481710a4" }, "user": { "displayName": "Foo bar", "id": "02" } }, "parentReference": { "driveId": "ID", "driveType": "personal", "id": "ID!101", "path": "/drive/root:" }, "fileSystemInfo": { "createdDateTime": "2023-02-01T00:51:02.803Z", "lastModifiedDateTime": "2023-02-01T00:51:02.803Z" }, "folder": { "childCount": 9, "view": { "viewType": "thumbnails", "sortBy": "name", "sortOrder": "ascending" } } } ] }"#; let response: GraphApiOnedriveListResponse = serde_json::from_str(response_json).unwrap(); assert_eq!(response.value.len(), 1); let item = &response.value[0]; if let ItemType::Folder { folder, .. } = &item.item_type { assert_eq!(folder.child_count, serde_json::Value::Number(9.into())); } else { panic!("item_type is not folder"); } } opendal-0.52.0/src/services/onedrive/lister.rs000064400000000000000000000077451046102023000174440ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use super::backend::OnedriveBackend; use super::error::parse_error; use super::graph_model::GraphApiOnedriveListResponse; use super::graph_model::ItemType; use crate::raw::oio; use crate::raw::*; use crate::*; pub struct OnedriveLister { root: String, path: String, backend: OnedriveBackend, } impl OnedriveLister { const DRIVE_ROOT_PREFIX: &'static str = "/drive/root:"; pub(crate) fn new(root: String, path: String, backend: OnedriveBackend) -> Self { Self { root, path, backend, } } } impl oio::PageList for OnedriveLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let request_url = if ctx.token.is_empty() { let path = build_rooted_abs_path(&self.root, &self.path); let url: String = if path == "." || path == "/" { "https://graph.microsoft.com/v1.0/me/drive/root/children".to_string() } else { // According to OneDrive API examples, the path should not end with a slash. // Reference: let path = path.strip_suffix('/').unwrap_or(""); format!( "https://graph.microsoft.com/v1.0/me/drive/root:{}:/children", percent_encode_path(path), ) }; url } else { ctx.token.clone() }; let resp = self .backend .onedrive_get_next_list_page(&request_url) .await?; let status_code = resp.status(); if !status_code.is_success() { if status_code == http::StatusCode::NOT_FOUND { ctx.done = true; return Ok(()); } let error = parse_error(resp); return Err(error); } let bytes = resp.into_body(); let decoded_response: GraphApiOnedriveListResponse = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; if let Some(next_link) = decoded_response.next_link { ctx.token = next_link; } else { ctx.done = true; } for drive_item in decoded_response.value { let name = drive_item.name; let parent_path = drive_item.parent_reference.path; let parent_path = parent_path .strip_prefix(Self::DRIVE_ROOT_PREFIX) .unwrap_or(""); let path = format!("{}/{}", parent_path, name); let normalized_path = build_rel_path(&self.root, &path); let entry: oio::Entry = match drive_item.item_type { ItemType::Folder { .. } => { let normalized_path = format!("{}/", normalized_path); oio::Entry::new(&normalized_path, Metadata::new(EntryMode::DIR)) } ItemType::File { .. } => { oio::Entry::new(&normalized_path, Metadata::new(EntryMode::FILE)) } }; ctx.entries.push_back(entry) } Ok(()) } } opendal-0.52.0/src/services/onedrive/mod.rs000064400000000000000000000024101046102023000167010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-onedrive")] mod backend; #[cfg(feature = "services-onedrive")] mod delete; #[cfg(feature = "services-onedrive")] mod error; #[cfg(feature = "services-onedrive")] mod graph_model; #[cfg(feature = "services-onedrive")] mod lister; #[cfg(feature = "services-onedrive")] mod writer; #[cfg(feature = "services-onedrive")] mod builder; #[cfg(feature = "services-onedrive")] pub use builder::OnedriveBuilder as Onedrive; mod config; pub use config::OnedriveConfig; opendal-0.52.0/src/services/onedrive/writer.rs000064400000000000000000000132761046102023000174520ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use bytes::Bytes; use http::StatusCode; use super::backend::OnedriveBackend; use super::error::parse_error; use super::graph_model::OneDriveUploadSessionCreationRequestBody; use super::graph_model::OneDriveUploadSessionCreationResponseBody; use crate::raw::*; use crate::*; pub struct OneDriveWriter { backend: OnedriveBackend, op: OpWrite, path: String, } impl OneDriveWriter { const MAX_SIMPLE_SIZE: usize = 4 * 1024 * 1024; // If your app splits a file into multiple byte ranges, the size of each byte range MUST be a multiple of 320 KiB (327,680 bytes). Using a fragment size that does not divide evenly by 320 KiB will result in errors committing some files. // https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession?view=odsp-graph-online#upload-bytes-to-the-upload-session const CHUNK_SIZE_FACTOR: usize = 327_680; pub fn new(backend: OnedriveBackend, op: OpWrite, path: String) -> Self { OneDriveWriter { backend, op, path } } } impl oio::OneShotWrite for OneDriveWriter { async fn write_once(&self, bs: Buffer) -> Result { let size = bs.len(); if size <= Self::MAX_SIMPLE_SIZE { self.write_simple(bs).await?; } else { self.write_chunked(bs.to_bytes()).await?; } Ok(Metadata::default()) } } impl OneDriveWriter { async fn write_simple(&self, bs: Buffer) -> Result<()> { let resp = self .backend .onedrive_upload_simple(&self.path, Some(bs.len()), &self.op, bs) .await?; let status = resp.status(); match status { // Typical response code: 201 Created // Reference: https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_put_content?view=odsp-graph-online#response StatusCode::CREATED | StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } pub(crate) async fn write_chunked(&self, total_bytes: Bytes) -> Result<()> { // Upload large files via sessions: https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession?view=odsp-graph-online#upload-bytes-to-the-upload-session // 1. Create an upload session // 2. Upload the bytes of each chunk // 3. Commit the session let session_response = self.create_upload_session().await?; let mut offset = 0; let iter = total_bytes.chunks(OneDriveWriter::CHUNK_SIZE_FACTOR); for chunk in iter { let mut end = offset + OneDriveWriter::CHUNK_SIZE_FACTOR; if end > total_bytes.len() { end = total_bytes.len(); } let total_len = total_bytes.len(); let chunk_end = end - 1; let resp = self .backend .onedrive_chunked_upload( &session_response.upload_url, &OpWrite::default(), offset, chunk_end, total_len, Buffer::from(Bytes::copy_from_slice(chunk)), ) .await?; let status = resp.status(); match status { // Typical response code: 202 Accepted // Reference: https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_put_content?view=odsp-graph-online#response StatusCode::ACCEPTED | StatusCode::CREATED | StatusCode::OK => {} _ => return Err(parse_error(resp)), } offset += OneDriveWriter::CHUNK_SIZE_FACTOR; } Ok(()) } async fn create_upload_session(&self) -> Result { let file_name_from_path = self.path.split('/').next_back().ok_or_else(|| { Error::new( ErrorKind::Unexpected, "connection string must have AccountName", ) })?; let url = format!( "{}/drive/root:{}:/createUploadSession", OnedriveBackend::BASE_URL, percent_encode_path(&self.path) ); let body = OneDriveUploadSessionCreationRequestBody::new(file_name_from_path.to_string()); let resp = self .backend .onedrive_create_upload_session(&url, body) .await?; let status = resp.status(); match status { // Reference: https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession?view=odsp-graph-online#response StatusCode::OK => { let bs = resp.into_body(); let result: OneDriveUploadSessionCreationResponseBody = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; Ok(result) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/oss/backend.rs000064400000000000000000000525471046102023000165220ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use http::Uri; use log::debug; use reqsign::AliyunConfig; use reqsign::AliyunLoader; use reqsign::AliyunOssSigner; use super::core::*; use super::delete::OssDeleter; use super::error::parse_error; use super::lister::{OssLister, OssListers, OssObjectVersionsLister}; use super::writer::OssWriter; use super::writer::OssWriters; use crate::raw::*; use crate::services::OssConfig; use crate::*; const DEFAULT_BATCH_MAX_OPERATIONS: usize = 1000; impl Configurator for OssConfig { type Builder = OssBuilder; fn into_builder(self) -> Self::Builder { OssBuilder { config: self, http_client: None, } } } /// Aliyun Object Storage Service (OSS) support #[doc = include_str!("docs.md")] #[derive(Default)] pub struct OssBuilder { config: OssConfig, http_client: Option, } impl Debug for OssBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("OssBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl OssBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set bucket name of this backend. pub fn bucket(mut self, bucket: &str) -> Self { self.config.bucket = bucket.to_string(); self } /// Set endpoint of this backend. pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()) } self } /// Set bucket versioning status for this backend pub fn enable_versioning(mut self, enabled: bool) -> Self { self.config.enable_versioning = enabled; self } /// Set an endpoint for generating presigned urls. /// /// You can offer a public endpoint like to return a presinged url for /// public accessors, along with an internal endpoint like /// to access objects in a faster path. /// /// - If presign_endpoint is set, we will use presign_endpoint on generating presigned urls. /// - if not, we will use endpoint as default. pub fn presign_endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.presign_endpoint = Some(endpoint.trim_end_matches('/').to_string()) } self } /// Set access_key_id of this backend. /// /// - If access_key_id is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn access_key_id(mut self, v: &str) -> Self { if !v.is_empty() { self.config.access_key_id = Some(v.to_string()) } self } /// Set access_key_secret of this backend. /// /// - If access_key_secret is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn access_key_secret(mut self, v: &str) -> Self { if !v.is_empty() { self.config.access_key_secret = Some(v.to_string()) } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } /// preprocess the endpoint option fn parse_endpoint(&self, endpoint: &Option, bucket: &str) -> Result<(String, String)> { let (endpoint, host) = match endpoint.clone() { Some(ep) => { let uri = ep.parse::().map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "endpoint is invalid") .with_context("service", Scheme::Oss) .with_context("endpoint", &ep) .set_source(err) })?; let host = uri.host().ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "endpoint host is empty") .with_context("service", Scheme::Oss) .with_context("endpoint", &ep) })?; let full_host = if let Some(port) = uri.port_u16() { format!("{bucket}.{host}:{port}") } else { format!("{bucket}.{host}") }; let endpoint = match uri.scheme_str() { Some(scheme_str) => match scheme_str { "http" | "https" => format!("{scheme_str}://{full_host}"), _ => { return Err(Error::new( ErrorKind::ConfigInvalid, "endpoint protocol is invalid", ) .with_context("service", Scheme::Oss)); } }, None => format!("https://{full_host}"), }; (endpoint, full_host) } None => { return Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_context("service", Scheme::Oss)); } }; Ok((endpoint, host)) } /// Set server_side_encryption for this backend. /// /// Available values: `AES256`, `KMS`. /// /// Reference: /// Brief explanation: /// There are two server-side encryption methods available: /// SSE-AES256: /// 1. Configure the bucket encryption mode as OSS-managed and specify the encryption algorithm as AES256. /// 2. Include the `x-oss-server-side-encryption` parameter in the request and set its value to AES256. /// SSE-KMS: /// 1. To use this service, you need to first enable KMS. /// 2. Configure the bucket encryption mode as KMS, and specify the specific CMK ID for BYOK (Bring Your Own Key) /// or not specify the specific CMK ID for OSS-managed KMS key. /// 3. Include the `x-oss-server-side-encryption` parameter in the request and set its value to KMS. /// 4. If a specific CMK ID is specified, include the `x-oss-server-side-encryption-key-id` parameter in the request, and set its value to the specified CMK ID. pub fn server_side_encryption(mut self, v: &str) -> Self { if !v.is_empty() { self.config.server_side_encryption = Some(v.to_string()) } self } /// Set server_side_encryption_key_id for this backend. /// /// # Notes /// /// This option only takes effect when server_side_encryption equals to KMS. pub fn server_side_encryption_key_id(mut self, v: &str) -> Self { if !v.is_empty() { self.config.server_side_encryption_key_id = Some(v.to_string()) } self } /// Set maximum batch operations of this backend. #[deprecated( since = "0.52.0", note = "Please use `delete_max_size` instead of `batch_max_operations`" )] pub fn batch_max_operations(mut self, delete_max_size: usize) -> Self { self.config.delete_max_size = Some(delete_max_size); self } /// Set maximum delete operations of this backend. pub fn delete_max_size(mut self, delete_max_size: usize) -> Self { self.config.delete_max_size = Some(delete_max_size); self } /// Allow anonymous will allow opendal to send request without signing /// when credential is not loaded. pub fn allow_anonymous(mut self) -> Self { self.config.allow_anonymous = true; self } /// Set role_arn for this backend. /// /// If `role_arn` is set, we will use already known config as source /// credential to assume role with `role_arn`. pub fn role_arn(mut self, role_arn: &str) -> Self { if !role_arn.is_empty() { self.config.role_arn = Some(role_arn.to_string()) } self } /// Set role_session_name for this backend. pub fn role_session_name(mut self, role_session_name: &str) -> Self { if !role_session_name.is_empty() { self.config.role_session_name = Some(role_session_name.to_string()) } self } /// Set oidc_provider_arn for this backend. pub fn oidc_provider_arn(mut self, oidc_provider_arn: &str) -> Self { if !oidc_provider_arn.is_empty() { self.config.oidc_provider_arn = Some(oidc_provider_arn.to_string()) } self } /// Set oidc_token_file for this backend. pub fn oidc_token_file(mut self, oidc_token_file: &str) -> Self { if !oidc_token_file.is_empty() { self.config.oidc_token_file = Some(oidc_token_file.to_string()) } self } /// Set sts_endpoint for this backend. pub fn sts_endpoint(mut self, sts_endpoint: &str) -> Self { if !sts_endpoint.is_empty() { self.config.sts_endpoint = Some(sts_endpoint.to_string()) } self } } impl Builder for OssBuilder { const SCHEME: Scheme = Scheme::Oss; type Config = OssConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle endpoint, region and bucket name. let bucket = match self.config.bucket.is_empty() { false => Ok(&self.config.bucket), true => Err( Error::new(ErrorKind::ConfigInvalid, "The bucket is misconfigured") .with_context("service", Scheme::Oss), ), }?; // Retrieve endpoint and host by parsing the endpoint option and bucket. If presign_endpoint is not // set, take endpoint as default presign_endpoint. let (endpoint, host) = self.parse_endpoint(&self.config.endpoint, bucket)?; debug!("backend use bucket {}, endpoint: {}", &bucket, &endpoint); let presign_endpoint = if self.config.presign_endpoint.is_some() { self.parse_endpoint(&self.config.presign_endpoint, bucket)? .0 } else { endpoint.clone() }; debug!("backend use presign_endpoint: {}", &presign_endpoint); let server_side_encryption = match &self.config.server_side_encryption { None => None, Some(v) => Some( build_header_value(v) .map_err(|err| err.with_context("key", "server_side_encryption"))?, ), }; let server_side_encryption_key_id = match &self.config.server_side_encryption_key_id { None => None, Some(v) => Some( build_header_value(v) .map_err(|err| err.with_context("key", "server_side_encryption_key_id"))?, ), }; let mut cfg = AliyunConfig::default(); // Load cfg from env first. cfg = cfg.from_env(); if let Some(v) = self.config.access_key_id { cfg.access_key_id = Some(v); } if let Some(v) = self.config.access_key_secret { cfg.access_key_secret = Some(v); } if let Some(v) = self.config.role_arn { cfg.role_arn = Some(v); } // override default role_session_name if set if let Some(v) = self.config.role_session_name { cfg.role_session_name = v; } if let Some(v) = self.config.oidc_provider_arn { cfg.oidc_provider_arn = Some(v); } if let Some(v) = self.config.oidc_token_file { cfg.oidc_token_file = Some(v); } if let Some(v) = self.config.sts_endpoint { cfg.sts_endpoint = Some(v); } let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Oss) })? }; let loader = AliyunLoader::new(GLOBAL_REQWEST_CLIENT.clone(), cfg); let signer = AliyunOssSigner::new(bucket); let delete_max_size = self .config .delete_max_size .unwrap_or(DEFAULT_BATCH_MAX_OPERATIONS); Ok(OssBackend { core: Arc::new(OssCore { root, bucket: bucket.to_owned(), endpoint, host, presign_endpoint, allow_anonymous: self.config.allow_anonymous, enable_versioning: self.config.enable_versioning, signer, loader, client, server_side_encryption, server_side_encryption_key_id, delete_max_size, }), }) } } #[derive(Debug, Clone)] /// Aliyun Object Storage Service backend pub struct OssBackend { core: Arc, } impl Access for OssBackend { type Reader = HttpBody; type Writer = OssWriters; type Lister = OssListers; type Deleter = oio::BatchDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Oss) .set_root(&self.core.root) .set_name(&self.core.bucket) .set_native_capability(Capability { stat: true, stat_with_if_match: true, stat_with_if_none_match: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_with_version: self.core.enable_versioning, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, stat_has_user_metadata: true, stat_has_version: true, read: true, read_with_if_match: true, read_with_if_none_match: true, read_with_version: self.core.enable_versioning, read_with_if_modified_since: true, read_with_if_unmodified_since: true, write: true, write_can_empty: true, write_can_append: true, write_can_multi: true, write_with_cache_control: true, write_with_content_type: true, write_with_content_disposition: true, // TODO: set this to false while version has been enabled. write_with_if_not_exists: !self.core.enable_versioning, // The min multipart size of OSS is 100 KiB. // // ref: write_multi_min_size: Some(100 * 1024), // The max multipart size of OSS is 5 GiB. // // ref: write_multi_max_size: if cfg!(target_pointer_width = "64") { Some(5 * 1024 * 1024 * 1024) } else { Some(usize::MAX) }, write_with_user_metadata: true, delete: true, delete_with_version: self.core.enable_versioning, delete_max_size: Some(self.core.delete_max_size), copy: true, list: true, list_with_limit: true, list_with_start_after: true, list_with_recursive: true, list_has_etag: true, list_has_content_md5: true, list_with_versions: self.core.enable_versioning, list_with_deleted: self.core.enable_versioning, list_has_content_length: true, list_has_last_modified: true, presign: true, presign_stat: true, presign_read: true, presign_write: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, args: OpStat) -> Result { let resp = self.core.oss_head_object(path, &args).await?; let status = resp.status(); match status { StatusCode::OK => { let headers = resp.headers(); let mut meta = self.core.parse_metadata(path, resp.headers())?; if let Some(v) = parse_header_to_str(headers, "x-oss-version-id")? { meta.set_version(v); } Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.oss_get_object(path, &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = OssWriter::new(self.core.clone(), path, args.clone()); let w = if args.append() { OssWriters::Two(oio::AppendWriter::new(writer)) } else { OssWriters::One(oio::MultipartWriter::new( writer, args.executor().cloned(), args.concurrent(), )) }; Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::BatchDeleter::new(OssDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = if args.versions() || args.deleted() { TwoWays::Two(oio::PageLister::new(OssObjectVersionsLister::new( self.core.clone(), path, args, ))) } else { TwoWays::One(oio::PageLister::new(OssLister::new( self.core.clone(), path, args.recursive(), args.limit(), args.start_after(), ))) }; Ok((RpList::default(), l)) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.oss_copy_object(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn presign(&self, path: &str, args: OpPresign) -> Result { // We will not send this request out, just for signing. let mut req = match args.operation() { PresignOperation::Stat(v) => self.core.oss_head_object_request(path, true, v)?, PresignOperation::Read(v) => self.core.oss_get_object_request(path, true, v)?, PresignOperation::Write(v) => { self.core .oss_put_object_request(path, None, v, Buffer::new(), true)? } }; self.core.sign_query(&mut req, args.expire()).await?; // We don't need this request anymore, consume it directly. let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } } opendal-0.52.0/src/services/oss/config.rs000064400000000000000000000064111046102023000163650ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Aliyun Object Storage Service (OSS) support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct OssConfig { /// Root for oss. pub root: Option, /// Endpoint for oss. pub endpoint: Option, /// Presign endpoint for oss. pub presign_endpoint: Option, /// Bucket for oss. pub bucket: String, /// is bucket versioning enabled for this bucket pub enable_versioning: bool, // OSS features /// Server side encryption for oss. pub server_side_encryption: Option, /// Server side encryption key id for oss. pub server_side_encryption_key_id: Option, /// Allow anonymous for oss. pub allow_anonymous: bool, // authenticate options /// Access key id for oss. pub access_key_id: Option, /// Access key secret for oss. pub access_key_secret: Option, /// The size of max batch operations. #[deprecated( since = "0.52.0", note = "Please use `delete_max_size` instead of `batch_max_operations`" )] pub batch_max_operations: Option, /// The size of max delete operations. pub delete_max_size: Option, /// If `role_arn` is set, we will use already known config as source /// credential to assume role with `role_arn`. pub role_arn: Option, /// role_session_name for this backend. pub role_session_name: Option, /// `oidc_provider_arn` will be loaded from /// /// - this field if it's `is_some` /// - env value: [`ALIBABA_CLOUD_OIDC_PROVIDER_ARN`] pub oidc_provider_arn: Option, /// `oidc_token_file` will be loaded from /// /// - this field if it's `is_some` /// - env value: [`ALIBABA_CLOUD_OIDC_TOKEN_FILE`] pub oidc_token_file: Option, /// `sts_endpoint` will be loaded from /// /// - this field if it's `is_some` /// - env value: [`ALIBABA_CLOUD_STS_ENDPOINT`] pub sts_endpoint: Option, } impl Debug for OssConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("Builder"); d.field("root", &self.root) .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .field("allow_anonymous", &self.allow_anonymous); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/oss/core.rs000064400000000000000000001030761046102023000160550ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::Write; use std::time::Duration; use bytes::Bytes; use constants::X_OSS_META_PREFIX; use http::header::CACHE_CONTROL; use http::header::CONTENT_DISPOSITION; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::header::IF_MATCH; use http::header::IF_MODIFIED_SINCE; use http::header::IF_NONE_MATCH; use http::header::IF_UNMODIFIED_SINCE; use http::header::RANGE; use http::HeaderMap; use http::HeaderName; use http::HeaderValue; use http::Request; use http::Response; use reqsign::AliyunCredential; use reqsign::AliyunLoader; use reqsign::AliyunOssSigner; use serde::Deserialize; use serde::Serialize; use crate::raw::*; use crate::services::oss::core::constants::X_OSS_FORBID_OVERWRITE; use crate::*; pub mod constants { pub const X_OSS_SERVER_SIDE_ENCRYPTION: &str = "x-oss-server-side-encryption"; pub const X_OSS_SERVER_SIDE_ENCRYPTION_KEY_ID: &str = "x-oss-server-side-encryption-key-id"; pub const X_OSS_FORBID_OVERWRITE: &str = "x-oss-forbid-overwrite"; pub const RESPONSE_CONTENT_DISPOSITION: &str = "response-content-disposition"; pub const OSS_QUERY_VERSION_ID: &str = "versionId"; pub const X_OSS_META_PREFIX: &str = "x-oss-meta-"; } pub struct OssCore { pub root: String, pub bucket: String, /// buffered host string /// /// format: . pub host: String, pub endpoint: String, pub presign_endpoint: String, pub allow_anonymous: bool, pub enable_versioning: bool, pub server_side_encryption: Option, pub server_side_encryption_key_id: Option, pub client: HttpClient, pub loader: AliyunLoader, pub signer: AliyunOssSigner, pub delete_max_size: usize, } impl Debug for OssCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .field("host", &self.host) .finish_non_exhaustive() } } impl OssCore { async fn load_credential(&self) -> Result> { let cred = self .loader .load() .await .map_err(new_request_credential_error)?; if let Some(cred) = cred { Ok(Some(cred)) } else if self.allow_anonymous { // If allow_anonymous has been set, we will not sign the request. Ok(None) } else { // Mark this error as temporary since it could be caused by Aliyun STS. Err(Error::new( ErrorKind::PermissionDenied, "no valid credential found, please check configuration or try again", ) .set_temporary()) } } pub async fn sign(&self, req: &mut Request) -> Result<()> { let cred = if let Some(cred) = self.load_credential().await? { cred } else { return Ok(()); }; self.signer.sign(req, &cred).map_err(new_request_sign_error) } pub async fn sign_query(&self, req: &mut Request, duration: Duration) -> Result<()> { let cred = if let Some(cred) = self.load_credential().await? { cred } else { return Ok(()); }; self.signer .sign_query(req, duration, &cred) .map_err(new_request_sign_error) } #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } /// Set sse headers /// # Note /// According to the OSS documentation, only PutObject, CopyObject, and InitiateMultipartUpload may require to be set. pub fn insert_sse_headers(&self, mut req: http::request::Builder) -> http::request::Builder { if let Some(v) = &self.server_side_encryption { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_OSS_SERVER_SIDE_ENCRYPTION), v, ) } if let Some(v) = &self.server_side_encryption_key_id { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_OSS_SERVER_SIDE_ENCRYPTION_KEY_ID), v, ) } req } fn insert_metadata_headers( &self, mut req: http::request::Builder, size: Option, args: &OpWrite, ) -> Result { req = req.header(CONTENT_LENGTH, size.unwrap_or_default()); if let Some(mime) = args.content_type() { req = req.header(CONTENT_TYPE, mime); } if let Some(pos) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, pos); } if let Some(cache_control) = args.cache_control() { req = req.header(CACHE_CONTROL, cache_control); } // TODO: disable if not exists while version has been enabled. // // Specifies whether the object that is uploaded by calling the PutObject operation // overwrites the existing object that has the same name. When versioning is enabled // or suspended for the bucket to which you want to upload the object, the // x-oss-forbid-overwrite header does not take effect. In this case, the object that // is uploaded by calling the PutObject operation overwrites the existing object that // has the same name. // // ref: https://www.alibabacloud.com/help/en/oss/developer-reference/putobject?spm=a2c63.p38356.0.0.39ef75e93o0Xtz if args.if_not_exists() { req = req.header(X_OSS_FORBID_OVERWRITE, "true"); } if let Some(user_metadata) = args.user_metadata() { for (key, value) in user_metadata { // before insert user defined metadata header, add prefix to the header name if !self.check_user_metadata_key(key) { return Err(Error::new( ErrorKind::Unsupported, "the format of the user metadata key is invalid, please refer the document", )); } req = req.header(format!("{X_OSS_META_PREFIX}{key}"), value) } } Ok(req) } // According to https://help.aliyun.com/zh/oss/developer-reference/putobject // there are some limits in user defined metadata key fn check_user_metadata_key(&self, key: &str) -> bool { key.chars().all(|c| c.is_ascii_alphanumeric() || c == '-') } /// parse_metadata will parse http headers(including standards http headers /// and user defined metadata header) into Metadata. /// /// # Arguments /// /// * `user_metadata_prefix` is the prefix of user defined metadata key /// /// # Notes /// /// before return the user defined metadata, we'll strip the user_metadata_prefix from the key pub fn parse_metadata(&self, path: &str, headers: &HeaderMap) -> Result { let mut m = parse_into_metadata(path, headers)?; let user_meta = parse_prefixed_headers(headers, X_OSS_META_PREFIX); if !user_meta.is_empty() { m.with_user_metadata(user_meta); } Ok(m) } } impl OssCore { #[allow(clippy::too_many_arguments)] pub fn oss_put_object_request( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, is_presign: bool, ) -> Result> { let p = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(is_presign); let url = format!("{}/{}", endpoint, percent_encode_path(&p)); let mut req = Request::put(&url); req = self.insert_metadata_headers(req, size, args)?; // set sse headers req = self.insert_sse_headers(req); let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub fn oss_append_object_request( &self, path: &str, position: u64, size: u64, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(false); let url = format!( "{}/{}?append&position={}", endpoint, percent_encode_path(&p), position ); let mut req = Request::post(&url); req = self.insert_metadata_headers(req, Some(size), args)?; // set sse headers req = self.insert_sse_headers(req); let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub fn oss_get_object_request( &self, path: &str, is_presign: bool, args: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(is_presign); let range = args.range(); let mut url = format!("{}/{}", endpoint, percent_encode_path(&p)); // Add query arguments to the URL based on response overrides let mut query_args = Vec::new(); if let Some(override_content_disposition) = args.override_content_disposition() { query_args.push(format!( "{}={}", constants::RESPONSE_CONTENT_DISPOSITION, percent_encode_path(override_content_disposition) )) } if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::OSS_QUERY_VERSION_ID, percent_encode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let mut req = Request::get(&url); req = req.header(CONTENT_TYPE, "application/octet-stream"); if !range.is_full() { req = req.header(RANGE, range.to_header()); // Adding `x-oss-range-behavior` header to use standard behavior. // ref: https://help.aliyun.com/document_detail/39571.html req = req.header("x-oss-range-behavior", "standard"); } if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match) } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(if_modified_since) = args.if_modified_since() { req = req.header( IF_MODIFIED_SINCE, format_datetime_into_http_date(if_modified_since), ); } if let Some(if_unmodified_since) = args.if_unmodified_since() { req = req.header( IF_UNMODIFIED_SINCE, format_datetime_into_http_date(if_unmodified_since), ); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } fn oss_delete_object_request(&self, path: &str, args: &OpDelete) -> Result> { let p = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(false); let mut url = format!("{}/{}", endpoint, percent_encode_path(&p)); let mut query_args = Vec::new(); if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::OSS_QUERY_VERSION_ID, percent_encode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let req = Request::delete(&url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub fn oss_head_object_request( &self, path: &str, is_presign: bool, args: &OpStat, ) -> Result> { let p = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(is_presign); let mut url = format!("{}/{}", endpoint, percent_encode_path(&p)); let mut query_args = Vec::new(); if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::OSS_QUERY_VERSION_ID, percent_encode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let mut req = Request::head(&url); if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match) } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub fn oss_list_object_request( &self, path: &str, token: &str, delimiter: &str, limit: Option, start_after: Option, ) -> Result> { let p = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(false); let mut url = format!("{}/?list-type=2", endpoint); write!(url, "&delimiter={delimiter}").expect("write into string must succeed"); // prefix if !p.is_empty() { write!(url, "&prefix={}", percent_encode_path(&p)) .expect("write into string must succeed"); } // max-key if let Some(limit) = limit { write!(url, "&max-keys={limit}").expect("write into string must succeed"); } // continuation_token if !token.is_empty() { write!(url, "&continuation-token={}", percent_encode_path(token)) .expect("write into string must succeed"); } // start-after if let Some(start_after) = start_after { let start_after = build_abs_path(&self.root, &start_after); write!(url, "&start-after={}", percent_encode_path(&start_after)) .expect("write into string must succeed"); } let req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; Ok(req) } pub async fn oss_get_object(&self, path: &str, args: &OpRead) -> Result> { let mut req = self.oss_get_object_request(path, false, args)?; self.sign(&mut req).await?; self.client.fetch(req).await } pub async fn oss_head_object(&self, path: &str, args: &OpStat) -> Result> { let mut req = self.oss_head_object_request(path, false, args)?; self.sign(&mut req).await?; self.send(req).await } pub async fn oss_copy_object(&self, from: &str, to: &str) -> Result> { let source = build_abs_path(&self.root, from); let target = build_abs_path(&self.root, to); let url = format!( "{}/{}", self.get_endpoint(false), percent_encode_path(&target) ); let source = format!("/{}/{}", self.bucket, percent_encode_path(&source)); let mut req = Request::put(&url); req = self.insert_sse_headers(req); req = req.header("x-oss-copy-source", source); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn oss_list_object( &self, path: &str, token: &str, delimiter: &str, limit: Option, start_after: Option, ) -> Result> { let mut req = self.oss_list_object_request(path, token, delimiter, limit, start_after)?; self.sign(&mut req).await?; self.send(req).await } pub async fn oss_list_object_versions( &self, prefix: &str, delimiter: &str, limit: Option, key_marker: &str, version_id_marker: &str, ) -> Result> { let p = build_abs_path(&self.root, prefix); let mut url = format!("{}?versions", self.endpoint); if !p.is_empty() { write!(url, "&prefix={}", percent_encode_path(p.as_str())) .expect("write into string must succeed"); } if !delimiter.is_empty() { write!(url, "&delimiter={}", delimiter).expect("write into string must succeed"); } if let Some(limit) = limit { write!(url, "&max-keys={}", limit).expect("write into string must succeed"); } if !key_marker.is_empty() { write!(url, "&key-marker={}", percent_encode_path(key_marker)) .expect("write into string must succeed"); } if !version_id_marker.is_empty() { write!( url, "&version-id-marker={}", percent_encode_path(version_id_marker) ) .expect("write into string must succeed"); } let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn oss_delete_object(&self, path: &str, args: &OpDelete) -> Result> { let mut req = self.oss_delete_object_request(path, args)?; self.sign(&mut req).await?; self.send(req).await } pub async fn oss_delete_objects(&self, paths: Vec) -> Result> { let url = format!("{}/?delete", self.endpoint); let req = Request::post(&url); let content = quick_xml::se::to_string(&DeleteObjectsRequest { object: paths .into_iter() .map(|path| DeleteObjectsRequestObject { key: build_abs_path(&self.root, &path), }) .collect(), }) .map_err(new_xml_deserialize_error)?; // Make sure content length has been set to avoid post with chunked encoding. let req = req.header(CONTENT_LENGTH, content.len()); // Set content-type to `application/xml` to avoid mixed with form post. let req = req.header(CONTENT_TYPE, "application/xml"); // Set content-md5 as required by API. let req = req.header("CONTENT-MD5", format_content_md5(content.as_bytes())); let mut req = req .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } fn get_endpoint(&self, is_presign: bool) -> &str { if is_presign { &self.presign_endpoint } else { &self.endpoint } } pub async fn oss_initiate_upload( &self, path: &str, content_type: Option<&str>, content_disposition: Option<&str>, cache_control: Option<&str>, is_presign: bool, ) -> Result> { let path = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(is_presign); let url = format!("{}/{}?uploads", endpoint, percent_encode_path(&path)); let mut req = Request::post(&url); if let Some(mime) = content_type { req = req.header(CONTENT_TYPE, mime); } if let Some(disposition) = content_disposition { req = req.header(CONTENT_DISPOSITION, disposition); } if let Some(cache_control) = cache_control { req = req.header(CACHE_CONTROL, cache_control); } req = self.insert_sse_headers(req); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } /// Creates a request to upload a part pub async fn oss_upload_part_request( &self, path: &str, upload_id: &str, part_number: usize, is_presign: bool, size: u64, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(is_presign); let url = format!( "{}/{}?partNumber={}&uploadId={}", endpoint, percent_encode_path(&p), part_number, percent_encode_path(upload_id) ); let mut req = Request::put(&url); req = req.header(CONTENT_LENGTH, size); let mut req = req.body(body).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn oss_complete_multipart_upload_request( &self, path: &str, upload_id: &str, is_presign: bool, parts: Vec, ) -> Result> { let p = build_abs_path(&self.root, path); let endpoint = self.get_endpoint(is_presign); let url = format!( "{}/{}?uploadId={}", endpoint, percent_encode_path(&p), percent_encode_path(upload_id) ); let req = Request::post(&url); let content = quick_xml::se::to_string(&CompleteMultipartUploadRequest { part: parts.to_vec(), }) .map_err(new_xml_deserialize_error)?; // Make sure content length has been set to avoid post with chunked encoding. let req = req.header(CONTENT_LENGTH, content.len()); // Set content-type to `application/xml` to avoid mixed with form post. let req = req.header(CONTENT_TYPE, "application/xml"); let mut req = req .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } /// Abort an ongoing multipart upload. /// reference docs https://www.alibabacloud.com/help/zh/oss/developer-reference/abortmultipartupload pub async fn oss_abort_multipart_upload( &self, path: &str, upload_id: &str, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?uploadId={}", self.endpoint, percent_encode_path(&p), percent_encode_path(upload_id) ); let mut req = Request::delete(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } } /// Request of DeleteObjects. #[derive(Default, Debug, Serialize)] #[serde(default, rename = "Delete", rename_all = "PascalCase")] pub struct DeleteObjectsRequest { pub object: Vec, } #[derive(Default, Debug, Serialize)] #[serde(rename_all = "PascalCase")] pub struct DeleteObjectsRequestObject { pub key: String, } /// Result of DeleteObjects. #[derive(Default, Debug, Deserialize)] #[serde(default, rename = "DeleteResult", rename_all = "PascalCase")] pub struct DeleteObjectsResult { pub deleted: Vec, } #[derive(Default, Debug, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct DeleteObjectsResultDeleted { pub key: String, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct DeleteObjectsResultError { pub code: String, pub key: String, pub message: String, } #[derive(Default, Debug, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct InitiateMultipartUploadResult { #[cfg(test)] pub bucket: String, #[cfg(test)] pub key: String, pub upload_id: String, } #[derive(Clone, Default, Debug, Serialize)] #[serde(default, rename_all = "PascalCase")] pub struct MultipartUploadPart { #[serde(rename = "PartNumber")] pub part_number: usize, #[serde(rename = "ETag")] pub etag: String, } #[derive(Default, Debug, Serialize)] #[serde(default, rename = "CompleteMultipartUpload", rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequest { pub part: Vec, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectsOutput { pub prefix: String, pub max_keys: u64, pub encoding_type: String, pub is_truncated: bool, pub common_prefixes: Vec, pub contents: Vec, pub key_count: u64, pub next_continuation_token: Option, } #[derive(Default, Debug, Deserialize, PartialEq, Eq)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectsOutputContent { pub key: String, pub last_modified: String, #[serde(rename = "ETag")] pub etag: String, pub size: u64, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct CommonPrefix { pub prefix: String, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct OutputCommonPrefix { pub prefix: String, } /// Output of ListObjectVersions #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectVersionsOutput { pub is_truncated: Option, pub next_key_marker: Option, pub next_version_id_marker: Option, pub common_prefixes: Vec, pub version: Vec, pub delete_marker: Vec, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct ListObjectVersionsOutputVersion { pub key: String, pub version_id: String, pub is_latest: bool, pub size: u64, pub last_modified: String, #[serde(rename = "ETag")] pub etag: Option, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct ListObjectVersionsOutputDeleteMarker { pub key: String, pub version_id: String, pub is_latest: bool, pub last_modified: String, } #[cfg(test)] mod tests { use bytes::Buf; use bytes::Bytes; use super::*; /// This example is from https://www.alibabacloud.com/help/zh/object-storage-service/latest/deletemultipleobjects #[test] fn test_serialize_delete_objects_request() { let req = DeleteObjectsRequest { object: vec![ DeleteObjectsRequestObject { key: "multipart.data".to_string(), }, DeleteObjectsRequestObject { key: "test.jpg".to_string(), }, DeleteObjectsRequestObject { key: "demo.jpg".to_string(), }, ], }; let actual = quick_xml::se::to_string(&req).expect("must succeed"); pretty_assertions::assert_eq!( actual, r#" multipart.data test.jpg demo.jpg "# // Cleanup space and new line .replace([' ', '\n'], "") ) } /// This example is from https://www.alibabacloud.com/help/zh/object-storage-service/latest/deletemultipleobjects #[test] fn test_deserialize_delete_objects_result() { let bs = Bytes::from( r#" multipart.data test.jpg demo.jpg "#, ); let out: DeleteObjectsResult = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert_eq!(out.deleted.len(), 3); assert_eq!(out.deleted[0].key, "multipart.data"); assert_eq!(out.deleted[1].key, "test.jpg"); assert_eq!(out.deleted[2].key, "demo.jpg"); } #[test] fn test_deserialize_initiate_multipart_upload_response() { let bs = Bytes::from( r#" oss-example multipart.data 0004B9894A22E5B1888A1E29F823**** "#, ); let out: InitiateMultipartUploadResult = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert_eq!("0004B9894A22E5B1888A1E29F823****", out.upload_id); assert_eq!("multipart.data", out.key); assert_eq!("oss-example", out.bucket); } #[test] fn test_serialize_complete_multipart_upload_request() { let req = CompleteMultipartUploadRequest { part: vec![ MultipartUploadPart { part_number: 1, etag: "\"3349DC700140D7F86A0784842780****\"".to_string(), }, MultipartUploadPart { part_number: 5, etag: "\"8EFDA8BE206636A695359836FE0A****\"".to_string(), }, MultipartUploadPart { part_number: 8, etag: "\"8C315065167132444177411FDA14****\"".to_string(), }, ], }; // quick_xml::se::to_string() let mut serialized = String::new(); let mut serializer = quick_xml::se::Serializer::new(&mut serialized); serializer.indent(' ', 4); req.serialize(serializer).unwrap(); pretty_assertions::assert_eq!( serialized, r#" 1 "3349DC700140D7F86A0784842780****" 5 "8EFDA8BE206636A695359836FE0A****" 8 "8C315065167132444177411FDA14****" "# ) } #[test] fn test_parse_list_output() { let bs = bytes::Bytes::from( r#" examplebucket b 3 url true CgJiYw-- b/c 2020-05-18T05:45:54.000Z "35A27C2B9EAEEB6F48FD7FB5861D****" 25 STANDARD 1686240967192623 1686240967192623 ba 2020-05-18T11:17:58.000Z "35A27C2B9EAEEB6F48FD7FB5861D****" 25 STANDARD 1686240967192623 1686240967192623 bc 2020-05-18T05:45:59.000Z "35A27C2B9EAEEB6F48FD7FB5861D****" 25 STANDARD 1686240967192623 1686240967192623 3 "#, ); let out: ListObjectsOutput = quick_xml::de::from_reader(bs.reader()).expect("must_success"); assert!(out.is_truncated); assert_eq!(out.next_continuation_token, Some("CgJiYw--".to_string())); assert!(out.common_prefixes.is_empty()); assert_eq!( out.contents, vec![ ListObjectsOutputContent { key: "b/c".to_string(), last_modified: "2020-05-18T05:45:54.000Z".to_string(), etag: "\"35A27C2B9EAEEB6F48FD7FB5861D****\"".to_string(), size: 25, }, ListObjectsOutputContent { key: "ba".to_string(), last_modified: "2020-05-18T11:17:58.000Z".to_string(), etag: "\"35A27C2B9EAEEB6F48FD7FB5861D****\"".to_string(), size: 25, }, ListObjectsOutputContent { key: "bc".to_string(), last_modified: "2020-05-18T05:45:59.000Z".to_string(), etag: "\"35A27C2B9EAEEB6F48FD7FB5861D****\"".to_string(), size: 25, } ] ) } } opendal-0.52.0/src/services/oss/delete.rs000064400000000000000000000064371046102023000163720ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::oio::BatchDeleteResult; use crate::raw::*; use crate::*; use bytes::Buf; use http::StatusCode; use std::collections::HashSet; use std::sync::Arc; pub struct OssDeleter { core: Arc, } impl OssDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::BatchDelete for OssDeleter { async fn delete_once(&self, path: String, args: OpDelete) -> Result<()> { let resp = self.core.oss_delete_object(&path, &args).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT | StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } async fn delete_batch(&self, batch: Vec<(String, OpDelete)>) -> Result { // Sadly, OSS will not return failed keys, so we will build // a set to calculate the failed keys. let mut keys = HashSet::new(); let paths = batch .into_iter() .map(|(p, _)| { keys.insert(p.clone()); p }) .collect(); let resp = self.core.oss_delete_objects(paths).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let result: DeleteObjectsResult = quick_xml::de::from_reader(bs.reader()).map_err(new_xml_deserialize_error)?; if result.deleted.is_empty() { return Err(Error::new( ErrorKind::Unexpected, "oss delete this key failed for reason we don't know", )); } let mut batched_result = BatchDeleteResult { succeeded: Vec::with_capacity(result.deleted.len()), failed: Vec::with_capacity(keys.len() - result.deleted.len()), }; for i in result.deleted { let path = build_rel_path(&self.core.root, &i.key); keys.remove(&path); batched_result.succeeded.push((path, OpDelete::default())); } // TODO: we should handle those errors with code. for i in keys { batched_result.failed.push(( i, OpDelete::default(), Error::new( ErrorKind::Unexpected, "oss delete this key failed for reason we don't know", ), )); } Ok(batched_result) } } opendal-0.52.0/src/services/oss/docs.md000064400000000000000000000035231046102023000160250ustar 00000000000000 # Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] append - [x] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [x] presign - [ ] blocking # Configuration - `root`: Set the work dir for backend. - `bucket`: Set the container name for backend. - `endpoint`: Set the endpoint for backend. - `presign_endpoint`: Set the endpoint for presign. - `access_key_id`: Set the access_key_id for backend. - `access_key_secret`: Set the access_key_secret for backend. - `role_arn`: Set the role of backend. - `oidc_token`: Set the oidc_token for backend. - `allow_anonymous`: Set the backend access OSS in anonymous way. Refer to [`OssBuilder`]'s public API docs for more information. # Example ## Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Oss; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create OSS backend builder. let mut builder = Oss::default() // Set the root for oss, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/path/to/dir") // Set the bucket name, this is required. .bucket("test") // Set the endpoint. // // For example: // - "https://oss-ap-northeast-1.aliyuncs.com" // - "https://oss-hangzhou.aliyuncs.com" .endpoint("https://oss-cn-beijing.aliyuncs.com") // Set the access_key_id and access_key_secret. // // OpenDAL will try load credential from the env. // If credential not set and no valid credential in env, OpenDAL will // send request without signing like anonymous user. .access_key_id("access_key_id") .access_key_secret("access_key_secret"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/oss/error.rs000064400000000000000000000064431046102023000162560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use http::StatusCode; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// OssError is the error returned by oss service. #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] struct OssError { code: String, message: String, request_id: String, host_id: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::NOT_MODIFIED | StatusCode::CONFLICT => { (ErrorKind::ConditionNotMatch, false) } StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = match de::from_reader::<_, OssError>(bs.clone().reader()) { Ok(oss_err) => format!("{oss_err:?}"), Err(_) => String::from_utf8_lossy(&bs).into_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod tests { use super::*; /// Error response example is from https://www.alibabacloud.com/help/en/object-storage-service/latest/error-responses #[test] fn test_parse_error() { let bs = bytes::Bytes::from( r#" AccessDenied Query-string authentication requires the Signature, Expires and OSSAccessKeyId parameters 1D842BC54255**** oss-cn-hangzhou.aliyuncs.com "#, ); let out: OssError = de::from_reader(bs.reader()).expect("must success"); println!("{out:?}"); assert_eq!(out.code, "AccessDenied"); assert_eq!(out.message, "Query-string authentication requires the Signature, Expires and OSSAccessKeyId parameters"); assert_eq!(out.request_id, "1D842BC54255****"); assert_eq!(out.host_id, "oss-cn-hangzhou.aliyuncs.com"); } } opendal-0.52.0/src/services/oss/lister.rs000064400000000000000000000205011046102023000164160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use quick_xml::de; use super::core::*; use super::error::parse_error; use crate::raw::oio::PageContext; use crate::raw::*; use crate::*; pub type OssListers = TwoWays, oio::PageLister>; pub struct OssLister { core: Arc, path: String, delimiter: &'static str, limit: Option, /// Filter results to objects whose names are lexicographically /// **equal to or after** startOffset start_after: Option, } impl OssLister { pub fn new( core: Arc, path: &str, recursive: bool, limit: Option, start_after: Option<&str>, ) -> Self { let delimiter = if recursive { "" } else { "/" }; Self { core, path: path.to_string(), delimiter, limit, start_after: start_after.map(String::from), } } } impl oio::PageList for OssLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .oss_list_object( &self.path, &ctx.token, self.delimiter, self.limit, if ctx.token.is_empty() { self.start_after.clone() } else { None }, ) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let output: ListObjectsOutput = de::from_reader(bs.reader()) .map_err(|e| Error::new(ErrorKind::Unexpected, "deserialize xml").set_source(e))?; ctx.done = !output.is_truncated; ctx.token = output.next_continuation_token.unwrap_or_default(); for prefix in output.common_prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix.prefix), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de); } for object in output.contents { let mut path = build_rel_path(&self.core.root, &object.key); if path.is_empty() { path = "/".to_string(); } if self.start_after.as_ref() == Some(&path) { continue; } let mut meta = Metadata::new(EntryMode::from_path(&path)); meta.set_is_current(true); meta.set_etag(&object.etag); meta.set_content_md5(object.etag.trim_matches('"')); meta.set_content_length(object.size); meta.set_last_modified(parse_datetime_from_rfc3339(object.last_modified.as_str())?); let de = oio::Entry::with(path, meta); ctx.entries.push_back(de); } Ok(()) } } /// refer: https://help.aliyun.com/zh/oss/developer-reference/listobjectversions?spm=a2c4g.11186623.help-menu-31815.d_3_1_1_5_5_2.53f67237GJlMPw&scm=20140722.H_112467._.OR_help-T_cn~zh-V_1 pub struct OssObjectVersionsLister { core: Arc, prefix: String, args: OpList, delimiter: &'static str, abs_start_after: Option, } impl OssObjectVersionsLister { pub fn new(core: Arc, path: &str, args: OpList) -> Self { let delimiter = if args.recursive() { "" } else { "/" }; let abs_start_after = args .start_after() .map(|start_after| build_abs_path(&core.root, start_after)); Self { core, prefix: path.to_string(), args, delimiter, abs_start_after, } } } impl oio::PageList for OssObjectVersionsLister { async fn next_page(&self, ctx: &mut PageContext) -> Result<()> { let markers = ctx.token.rsplit_once(" "); let (key_marker, version_id_marker) = if let Some(data) = markers { data } else if let Some(start_after) = &self.abs_start_after { (start_after.as_str(), "") } else { ("", "") }; let resp = self .core .oss_list_object_versions( &self.prefix, self.delimiter, self.args.limit(), key_marker, version_id_marker, ) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let body = resp.into_body(); let output: ListObjectVersionsOutput = de::from_reader(body.reader()) .map_err(new_xml_deserialize_error) // Allow Cos list to retry on XML deserialization errors. // // This is because the Cos list API may return incomplete XML data under high load. // We are confident that our XML decoding logic is correct. When this error occurs, // we allow retries to obtain the correct data. .map_err(Error::set_temporary)?; ctx.done = if let Some(is_truncated) = output.is_truncated { !is_truncated } else { false }; ctx.token = format!( "{} {}", output.next_key_marker.unwrap_or_default(), output.next_version_id_marker.unwrap_or_default() ); for prefix in output.common_prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix.prefix), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de); } for version_object in output.version { // `list` must be additive, so we need to include the latest version object // even if `versions` is not enabled. // // Here we skip all non-latest version objects if `versions` is not enabled. if !(self.args.versions() || version_object.is_latest) { continue; } let mut path = build_rel_path(&self.core.root, &version_object.key); if path.is_empty() { path = "/".to_owned(); } let mut meta = Metadata::new(EntryMode::from_path(&path)); meta.set_version(&version_object.version_id); meta.set_is_current(version_object.is_latest); meta.set_content_length(version_object.size); meta.set_last_modified(parse_datetime_from_rfc3339( version_object.last_modified.as_str(), )?); if let Some(etag) = version_object.etag { meta.set_etag(&etag); meta.set_content_md5(etag.trim_matches('"')); } let entry = oio::Entry::new(&path, meta); ctx.entries.push_back(entry); } if self.args.deleted() { for delete_marker in output.delete_marker { let mut path = build_rel_path(&self.core.root, &delete_marker.key); if path.is_empty() { path = "/".to_owned(); } let mut meta = Metadata::new(EntryMode::FILE); meta.set_version(&delete_marker.version_id); meta.set_is_deleted(true); meta.set_is_current(delete_marker.is_latest); meta.set_last_modified(parse_datetime_from_rfc3339( delete_marker.last_modified.as_str(), )?); let entry = oio::Entry::new(&path, meta); ctx.entries.push_back(entry); } } Ok(()) } } opendal-0.52.0/src/services/oss/mod.rs000064400000000000000000000022341046102023000156760ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-oss")] mod core; #[cfg(feature = "services-oss")] mod delete; #[cfg(feature = "services-oss")] mod error; #[cfg(feature = "services-oss")] mod lister; #[cfg(feature = "services-oss")] mod writer; #[cfg(feature = "services-oss")] mod backend; #[cfg(feature = "services-oss")] pub use backend::OssBuilder as Oss; mod config; pub use config::OssConfig; opendal-0.52.0/src/services/oss/writer.rs000064400000000000000000000135341046102023000164400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; pub type OssWriters = TwoWays, oio::AppendWriter>; pub struct OssWriter { core: Arc, op: OpWrite, path: String, } impl OssWriter { pub fn new(core: Arc, path: &str, op: OpWrite) -> Self { OssWriter { core, path: path.to_string(), op, } } } impl oio::MultipartWrite for OssWriter { async fn write_once(&self, size: u64, body: Buffer) -> Result { let mut req = self.core .oss_put_object_request(&self.path, Some(size), &self.op, body, false)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn initiate_part(&self) -> Result { let resp = self .core .oss_initiate_upload( &self.path, self.op.content_type(), self.op.content_disposition(), self.op.cache_control(), false, ) .await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let result: InitiateMultipartUploadResult = quick_xml::de::from_reader(bytes::Buf::reader(bs)) .map_err(new_xml_deserialize_error)?; Ok(result.upload_id) } _ => Err(parse_error(resp)), } } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result { // OSS requires part number must between [1..=10000] let part_number = part_number + 1; let resp = self .core .oss_upload_part_request(&self.path, upload_id, part_number, false, size, body) .await?; let status = resp.status(); match status { StatusCode::OK => { let etag = parse_etag(resp.headers())? .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "ETag not present in returning response", ) })? .to_string(); Ok(oio::MultipartPart { part_number, etag, checksum: None, }) } _ => Err(parse_error(resp)), } } async fn complete_part( &self, upload_id: &str, parts: &[oio::MultipartPart], ) -> Result { let parts = parts .iter() .map(|p| MultipartUploadPart { part_number: p.part_number, etag: p.etag.clone(), }) .collect(); let resp = self .core .oss_complete_multipart_upload_request(&self.path, upload_id, false, parts) .await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn abort_part(&self, upload_id: &str) -> Result<()> { let resp = self .core .oss_abort_multipart_upload(&self.path, upload_id) .await?; match resp.status() { // OSS returns code 204 if abort succeeds. StatusCode::NO_CONTENT => Ok(()), _ => Err(parse_error(resp)), } } } impl oio::AppendWrite for OssWriter { async fn offset(&self) -> Result { let resp = self .core .oss_head_object(&self.path, &OpStat::new()) .await?; let status = resp.status(); match status { StatusCode::OK => { let content_length = parse_content_length(resp.headers())?.ok_or_else(|| { Error::new( ErrorKind::Unexpected, "Content-Length not present in returning response", ) })?; Ok(content_length) } StatusCode::NOT_FOUND => Ok(0), _ => Err(parse_error(resp)), } } async fn append(&self, offset: u64, size: u64, body: Buffer) -> Result { let mut req = self .core .oss_append_object_request(&self.path, offset, size, &self.op, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/pcloud/backend.rs000064400000000000000000000260511046102023000171730ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::Response; use http::StatusCode; use log::debug; use super::core::*; use super::delete::PcloudDeleter; use super::error::parse_error; use super::error::PcloudError; use super::lister::PcloudLister; use super::writer::PcloudWriter; use super::writer::PcloudWriters; use crate::raw::*; use crate::services::PcloudConfig; use crate::*; impl Configurator for PcloudConfig { type Builder = PcloudBuilder; fn into_builder(self) -> Self::Builder { PcloudBuilder { config: self, http_client: None, } } } /// [pCloud](https://www.pcloud.com/) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct PcloudBuilder { config: PcloudConfig, http_client: Option, } impl Debug for PcloudBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("PcloudBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl PcloudBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Pcloud endpoint. /// for United States and for Europe /// ref to [doc.pcloud.com](https://docs.pcloud.com/) /// /// It is required. e.g. `https://api.pcloud.com` pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = endpoint.to_string(); self } /// Pcloud username. /// /// It is required. your pCloud login email, e.g. `example@gmail.com` pub fn username(mut self, username: &str) -> Self { self.config.username = if username.is_empty() { None } else { Some(username.to_string()) }; self } /// Pcloud password. /// /// It is required. your pCloud login password, e.g. `password` pub fn password(mut self, password: &str) -> Self { self.config.password = if password.is_empty() { None } else { Some(password.to_string()) }; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for PcloudBuilder { const SCHEME: Scheme = Scheme::Pcloud; type Config = PcloudConfig; /// Builds the backend and returns the result of PcloudBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle endpoint. if self.config.endpoint.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Pcloud)); } debug!("backend use endpoint {}", &self.config.endpoint); let username = match &self.config.username { Some(username) => Ok(username.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "username is empty") .with_operation("Builder::build") .with_context("service", Scheme::Pcloud)), }?; let password = match &self.config.password { Some(password) => Ok(password.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty") .with_operation("Builder::build") .with_context("service", Scheme::Pcloud)), }?; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Pcloud) })? }; Ok(PcloudBackend { core: Arc::new(PcloudCore { root, endpoint: self.config.endpoint.clone(), username, password, client, }), }) } } /// Backend for Pcloud services. #[derive(Debug, Clone)] pub struct PcloudBackend { core: Arc, } impl Access for PcloudBackend { type Reader = HttpBody; type Writer = PcloudWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Pcloud) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, create_dir: true, read: true, write: true, delete: true, rename: true, copy: true, list: true, list_has_content_length: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { self.core.ensure_dir_exists(path).await?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _args: OpStat) -> Result { let resp = self.core.stat(path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: StatResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let result = resp.result; if result == 2010 || result == 2055 || result == 2002 { return Err(Error::new(ErrorKind::NotFound, format!("{resp:?}"))); } if result != 0 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } if let Some(md) = resp.metadata { let md = parse_stat_metadata(md); return md.map(RpStat::new); } Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let link = self.core.get_file_link(path).await?; let resp = self.core.download(&link, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, _args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = PcloudWriter::new(self.core.clone(), path.to_string()); let w = oio::OneShotWriter::new(writer); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(PcloudDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, _args: OpList) -> Result<(RpList, Self::Lister)> { let l = PcloudLister::new(self.core.clone(), path); Ok((RpList::default(), oio::PageLister::new(l))) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { self.core.ensure_dir_exists(to).await?; let resp = if from.ends_with('/') { self.core.copy_folder(from, to).await? } else { self.core.copy_file(from, to).await? }; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: PcloudError = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let result = resp.result; if result == 2009 || result == 2010 || result == 2055 || result == 2002 { return Err(Error::new(ErrorKind::NotFound, format!("{resp:?}"))); } if result != 0 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } Ok(RpCopy::default()) } _ => Err(parse_error(resp)), } } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { self.core.ensure_dir_exists(to).await?; let resp = if from.ends_with('/') { self.core.rename_folder(from, to).await? } else { self.core.rename_file(from, to).await? }; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: PcloudError = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let result = resp.result; if result == 2009 || result == 2010 || result == 2055 || result == 2002 { return Err(Error::new(ErrorKind::NotFound, format!("{resp:?}"))); } if result != 0 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } Ok(RpRename::default()) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/pcloud/config.rs000064400000000000000000000031771046102023000170550ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Pcloud services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct PcloudConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, ///pCloud endpoint address. pub endpoint: String, /// pCloud username. pub username: Option, /// pCloud password. pub password: Option, } impl Debug for PcloudConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Config"); ds.field("root", &self.root); ds.field("endpoint", &self.endpoint); ds.field("username", &self.username); ds.finish() } } opendal-0.52.0/src/services/pcloud/core.rs000064400000000000000000000304221046102023000165310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use bytes::Buf; use http::header; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use super::error::parse_error; use super::error::PcloudError; use crate::raw::*; use crate::*; #[derive(Clone)] pub struct PcloudCore { /// The root of this core. pub root: String, /// The endpoint of this backend. pub endpoint: String, /// The username id of this backend. pub username: String, /// The password of this backend. pub password: String, pub client: HttpClient, } impl Debug for PcloudCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("username", &self.username) .finish_non_exhaustive() } } impl PcloudCore { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } } impl PcloudCore { pub async fn get_file_link(&self, path: &str) -> Result { let path = build_abs_path(&self.root, path); let url = format!( "{}/getfilelink?path=/{}&username={}&password={}", self.endpoint, percent_encode_path(&path), self.username, self.password ); let req = Request::get(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: GetFileLinkResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let result = resp.result; if result == 2010 || result == 2055 || result == 2002 { return Err(Error::new(ErrorKind::NotFound, format!("{resp:?}"))); } if result != 0 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } if let Some(hosts) = resp.hosts { if let Some(path) = resp.path { if !hosts.is_empty() { return Ok(format!("https://{}{}", hosts[0], path)); } } } Err(Error::new(ErrorKind::Unexpected, "hosts is empty")) } _ => Err(parse_error(resp)), } } pub async fn download(&self, url: &str, range: BytesRange) -> Result> { let req = Request::get(url); // set body let req = req .header(header::RANGE, range.to_header()) .body(Buffer::new()) .map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn ensure_dir_exists(&self, path: &str) -> Result<()> { let path = build_abs_path(&self.root, path); let paths = path.split('/').collect::>(); for i in 0..paths.len() - 1 { let path = paths[..i + 1].join("/"); let resp = self.create_folder_if_not_exists(&path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: PcloudError = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let result = resp.result; if result == 2010 || result == 2055 || result == 2002 { return Err(Error::new(ErrorKind::NotFound, format!("{resp:?}"))); } if result != 0 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } if result != 0 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } } _ => return Err(parse_error(resp)), } } Ok(()) } pub async fn create_folder_if_not_exists(&self, path: &str) -> Result> { let url = format!( "{}/createfolderifnotexists?path=/{}&username={}&password={}", self.endpoint, percent_encode_path(path), self.username, self.password ); let req = Request::post(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn rename_file(&self, from: &str, to: &str) -> Result> { let from = build_abs_path(&self.root, from); let to = build_abs_path(&self.root, to); let url = format!( "{}/renamefile?path=/{}&topath=/{}&username={}&password={}", self.endpoint, percent_encode_path(&from), percent_encode_path(&to), self.username, self.password ); let req = Request::post(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn rename_folder(&self, from: &str, to: &str) -> Result> { let from = build_abs_path(&self.root, from); let to = build_abs_path(&self.root, to); let url = format!( "{}/renamefolder?path=/{}&topath=/{}&username={}&password={}", self.endpoint, percent_encode_path(&from), percent_encode_path(&to), self.username, self.password ); let req = Request::post(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn delete_folder(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "{}/deletefolder?path=/{}&username={}&password={}", self.endpoint, percent_encode_path(&path), self.username, self.password ); let req = Request::post(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn delete_file(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "{}/deletefile?path=/{}&username={}&password={}", self.endpoint, percent_encode_path(&path), self.username, self.password ); let req = Request::post(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn copy_file(&self, from: &str, to: &str) -> Result> { let from = build_abs_path(&self.root, from); let to = build_abs_path(&self.root, to); let url = format!( "{}/copyfile?path=/{}&topath=/{}&username={}&password={}", self.endpoint, percent_encode_path(&from), percent_encode_path(&to), self.username, self.password ); let req = Request::post(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn copy_folder(&self, from: &str, to: &str) -> Result> { let from = build_abs_path(&self.root, from); let to = build_abs_path(&self.root, to); let url = format!( "{}/copyfolder?path=/{}&topath=/{}&username={}&password={}", self.endpoint, percent_encode_path(&from), percent_encode_path(&to), self.username, self.password ); let req = Request::post(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn stat(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let path = path.trim_end_matches('/'); let url = format!( "{}/stat?path=/{}&username={}&password={}", self.endpoint, percent_encode_path(path), self.username, self.password ); let req = Request::post(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn upload_file(&self, path: &str, bs: Buffer) -> Result> { let path = build_abs_path(&self.root, path); let (name, path) = (get_basename(&path), get_parent(&path).trim_end_matches('/')); let url = format!( "{}/uploadfile?path=/{}&filename={}&username={}&password={}", self.endpoint, percent_encode_path(path), percent_encode_path(name), self.username, self.password ); let req = Request::put(url); // set body let req = req.body(bs).map_err(new_request_build_error)?; self.send(req).await } pub async fn list_folder(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let path = normalize_root(&path); let path = path.trim_end_matches('/'); let url = format!( "{}/listfolder?path={}&username={}&password={}", self.endpoint, percent_encode_path(path), self.username, self.password ); let req = Request::get(url); // set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } } pub(super) fn parse_stat_metadata(content: StatMetadata) -> Result { let mut md = if content.isfolder { Metadata::new(EntryMode::DIR) } else { Metadata::new(EntryMode::FILE) }; if let Some(size) = content.size { md.set_content_length(size); } md.set_last_modified(parse_datetime_from_rfc2822(&content.modified)?); Ok(md) } pub(super) fn parse_list_metadata(content: ListMetadata) -> Result { let mut md = if content.isfolder { Metadata::new(EntryMode::DIR) } else { Metadata::new(EntryMode::FILE) }; if let Some(size) = content.size { md.set_content_length(size); } md.set_last_modified(parse_datetime_from_rfc2822(&content.modified)?); Ok(md) } #[derive(Debug, Deserialize)] pub struct GetFileLinkResponse { pub result: u64, pub path: Option, pub hosts: Option>, } #[derive(Debug, Deserialize)] pub struct StatResponse { pub result: u64, pub metadata: Option, } #[derive(Debug, Deserialize)] pub struct StatMetadata { pub modified: String, pub isfolder: bool, pub size: Option, } #[derive(Debug, Deserialize)] pub struct ListFolderResponse { pub result: u64, pub metadata: Option, } #[derive(Debug, Deserialize)] pub struct ListMetadata { pub path: String, pub modified: String, pub isfolder: bool, pub size: Option, pub contents: Option>, } opendal-0.52.0/src/services/pcloud/delete.rs000064400000000000000000000040321046102023000170410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::{parse_error, PcloudError}; use crate::raw::*; use crate::*; use bytes::Buf; use http::StatusCode; use std::sync::Arc; pub struct PcloudDeleter { core: Arc, } impl PcloudDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for PcloudDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = if path.ends_with('/') { self.core.delete_folder(&path).await? } else { self.core.delete_file(&path).await? }; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: PcloudError = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let result = resp.result; // pCloud returns 2005 or 2009 if the file or folder is not found if result != 0 && result != 2005 && result != 2009 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } Ok(()) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/pcloud/docs.md000064400000000000000000000017711046102023000165120ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `endpoint`: Pcloud bucket name - `username` Pcloud username - `password` Pcloud password You can refer to [`PcloudBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Pcloud; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Pcloud::default() // set the storage bucket for OpenDAL .root("/") // set the bucket for OpenDAL .endpoint("[https](https://api.pcloud.com)") // set the username for OpenDAL .username("opendal@gmail.com") // set the password name for OpenDAL .password("opendal"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/pcloud/error.rs000064400000000000000000000047071046102023000167410ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use http::Response; use serde::Deserialize; use crate::raw::*; use crate::*; /// PcloudError is the error returned by Pcloud service. #[derive(Default, Deserialize)] pub(super) struct PcloudError { pub result: u32, pub error: Option, } impl Debug for PcloudError { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("PcloudError") .field("result", &self.result) .field("error", &self.error) .finish_non_exhaustive() } } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let message = String::from_utf8_lossy(&bs).into_owned(); let mut err = Error::new(ErrorKind::Unexpected, message); err = with_error_response_context(err, parts); err } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[tokio::test] async fn test_parse_error() { let err_res = vec![( r#" Invalid link This link was generated for another IP address. Try previous step again. "#, ErrorKind::Unexpected, StatusCode::GONE, )]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/pcloud/lister.rs000064400000000000000000000057071046102023000171130ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use super::core::*; use super::error::parse_error; use crate::raw::oio::Entry; use crate::raw::*; use crate::*; pub struct PcloudLister { core: Arc, path: String, } impl PcloudLister { pub(super) fn new(core: Arc, path: &str) -> Self { PcloudLister { core, path: path.to_string(), } } } impl oio::PageList for PcloudLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self.core.list_folder(&self.path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: ListFolderResponse = serde_json::from_reader(bs.clone().reader()) .map_err(new_json_deserialize_error)?; let result = resp.result; if result == 2005 { ctx.done = true; return Ok(()); } if result != 0 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } if let Some(metadata) = resp.metadata { if let Some(contents) = metadata.contents { for content in contents { let path = if content.isfolder { format!("{}/", content.path.clone()) } else { content.path.clone() }; let md = parse_list_metadata(content)?; let path = build_rel_path(&self.core.root, &path); ctx.entries.push_back(Entry::new(&path, md)) } } ctx.done = true; return Ok(()); } Err(Error::new( ErrorKind::Unexpected, String::from_utf8_lossy(&bs.to_bytes()), )) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/pcloud/mod.rs000064400000000000000000000022721046102023000163620ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-pcloud")] mod core; #[cfg(feature = "services-pcloud")] mod delete; #[cfg(feature = "services-pcloud")] mod error; #[cfg(feature = "services-pcloud")] mod lister; #[cfg(feature = "services-pcloud")] mod writer; #[cfg(feature = "services-pcloud")] mod backend; #[cfg(feature = "services-pcloud")] pub use backend::PcloudBuilder as Pcloud; mod config; pub use config::PcloudConfig; opendal-0.52.0/src/services/pcloud/writer.rs000064400000000000000000000037741046102023000171270ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use super::core::PcloudCore; use super::error::parse_error; use super::error::PcloudError; use crate::raw::*; use crate::*; pub type PcloudWriters = oio::OneShotWriter; pub struct PcloudWriter { core: Arc, path: String, } impl PcloudWriter { pub fn new(core: Arc, path: String) -> Self { PcloudWriter { core, path } } } impl oio::OneShotWrite for PcloudWriter { async fn write_once(&self, bs: Buffer) -> Result { self.core.ensure_dir_exists(&self.path).await?; let resp = self.core.upload_file(&self.path, bs).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: PcloudError = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let result = resp.result; if result != 0 { return Err(Error::new(ErrorKind::Unexpected, format!("{resp:?}"))); } Ok(Metadata::default()) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/persy/backend.rs000064400000000000000000000177131046102023000170540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::str; use persy; use tokio::task; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::PersyConfig; use crate::Builder; use crate::Error; use crate::ErrorKind; use crate::Scheme; use crate::*; impl Configurator for PersyConfig { type Builder = PersyBuilder; fn into_builder(self) -> Self::Builder { PersyBuilder { config: self } } } /// persy service support. #[doc = include_str!("docs.md")] #[derive(Default, Debug)] pub struct PersyBuilder { config: PersyConfig, } impl PersyBuilder { /// Set the path to the persy data directory. Will create if not exists. pub fn datafile(mut self, path: &str) -> Self { self.config.datafile = Some(path.into()); self } /// Set the name of the persy segment. Will create if not exists. pub fn segment(mut self, path: &str) -> Self { self.config.segment = Some(path.into()); self } /// Set the name of the persy index. Will create if not exists. pub fn index(mut self, path: &str) -> Self { self.config.index = Some(path.into()); self } } impl Builder for PersyBuilder { const SCHEME: Scheme = Scheme::Persy; type Config = PersyConfig; fn build(self) -> Result { let datafile_path = self.config.datafile.ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "datafile is required but not set") .with_context("service", Scheme::Persy) })?; let segment_name = self.config.segment.ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "segment is required but not set") .with_context("service", Scheme::Persy) })?; let segment = segment_name.clone(); let index_name = self.config.index.ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "index is required but not set") .with_context("service", Scheme::Persy) })?; let index = index_name.clone(); let persy = persy::OpenOptions::new() .create(true) .prepare_with(move |p| init(p, &segment_name, &index_name)) .open(&datafile_path) .map_err(|e| { Error::new(ErrorKind::ConfigInvalid, "open db") .with_context("service", Scheme::Persy) .with_context("datafile", datafile_path.clone()) .set_source(e) })?; // This function will only be called on database creation fn init( persy: &persy::Persy, segment_name: &str, index_name: &str, ) -> Result<(), Box> { let mut tx = persy.begin()?; if !tx.exists_segment(segment_name)? { tx.create_segment(segment_name)?; } if !tx.exists_index(index_name)? { tx.create_index::(index_name, persy::ValueMode::Replace)?; } let prepared = tx.prepare()?; prepared.commit()?; Ok(()) } Ok(PersyBackend::new(Adapter { datafile: datafile_path, segment, index, persy, })) } } /// Backend for persy services. pub type PersyBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { datafile: String, segment: String, index: String, persy: persy::Persy, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.field("path", &self.datafile); ds.field("segment", &self.segment); ds.field("index", &self.index); ds.finish() } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Persy, &self.datafile, Capability { read: true, write: true, delete: true, blocking: true, shared: false, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_get(cloned_path.as_str())) .await .map_err(new_task_join_error) .and_then(|inner_result| inner_result) } fn blocking_get(&self, path: &str) -> Result> { let mut read_id = self .persy .get::(&self.index, &path.to_string()) .map_err(parse_error)?; if let Some(id) = read_id.next() { let value = self.persy.read(&self.segment, &id).map_err(parse_error)?; return Ok(value.map(Buffer::from)); } Ok(None) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let cloned_path = path.to_string(); let cloned_self = self.clone(); task::spawn_blocking(move || cloned_self.blocking_set(cloned_path.as_str(), value)) .await .map_err(new_task_join_error) .and_then(|inner_result| inner_result) } fn blocking_set(&self, path: &str, value: Buffer) -> Result<()> { let mut tx = self.persy.begin().map_err(parse_error)?; let id = tx .insert(&self.segment, &value.to_vec()) .map_err(parse_error)?; tx.put::(&self.index, path.to_string(), id) .map_err(parse_error)?; let prepared = tx.prepare().map_err(parse_error)?; prepared.commit().map_err(parse_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let cloned_path = path.to_string(); let cloned_self = self.clone(); task::spawn_blocking(move || cloned_self.blocking_delete(cloned_path.as_str())) .await .map_err(new_task_join_error) .and_then(|inner_result| inner_result) } fn blocking_delete(&self, path: &str) -> Result<()> { let mut delete_id = self .persy .get::(&self.index, &path.to_string()) .map_err(parse_error)?; if let Some(id) = delete_id.next() { // Begin a transaction. let mut tx = self.persy.begin().map_err(parse_error)?; // Delete the record. tx.delete(&self.segment, &id).map_err(parse_error)?; // Remove the index. tx.remove::(&self.index, path.to_string(), Some(id)) .map_err(parse_error)?; // Commit the tx. let prepared = tx.prepare().map_err(parse_error)?; prepared.commit().map_err(parse_error)?; } Ok(()) } } fn parse_error>(err: persy::PE) -> Error { let err: persy::PersyError = err.persy_error(); let kind = match err { persy::PersyError::RecordNotFound(_) => ErrorKind::NotFound, _ => ErrorKind::Unexpected, }; Error::new(kind, "error from persy").set_source(err) } opendal-0.52.0/src/services/persy/config.rs000064400000000000000000000024061046102023000167230ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use serde::Deserialize; use serde::Serialize; /// Config for persy service support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct PersyConfig { /// That path to the persy data file. The directory in the path must already exist. pub datafile: Option, /// That name of the persy segment. pub segment: Option, /// That name of the persy index. pub index: Option, } opendal-0.52.0/src/services/persy/docs.md000064400000000000000000000014561046102023000163660ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] list - [ ] ~~presign~~ - [x] blocking ## Configuration - `datafile`: Set the path to the persy data file. The directory in the path must already exist. - `segment`: Set the name of the persy segment. - `index`: Set the name of the persy index. You can refer to [`PersyBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Persy; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Persy::default() .datafile("./test.persy") .segment("data") .index("index"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/persy/mod.rs000064400000000000000000000017071046102023000162400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-persy")] mod backend; #[cfg(feature = "services-persy")] pub use backend::PersyBuilder as Persy; mod config; pub use config::PersyConfig; opendal-0.52.0/src/services/postgresql/backend.rs000064400000000000000000000170431046102023000201110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::str::FromStr; use sqlx::postgres::PgConnectOptions; use sqlx::PgPool; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::PostgresqlConfig; use crate::*; impl Configurator for PostgresqlConfig { type Builder = PostgresqlBuilder; fn into_builder(self) -> Self::Builder { PostgresqlBuilder { config: self } } } /// [PostgreSQL](https://www.postgresql.org/) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct PostgresqlBuilder { config: PostgresqlConfig, } impl Debug for PostgresqlBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("PostgresqlBuilder"); d.field("config", &self.config); d.finish() } } impl PostgresqlBuilder { /// Set the connection url string of the postgresql service. /// /// The URL should be with a scheme of either `postgres://` or `postgresql://`. /// /// - `postgresql://user@localhost` /// - `postgresql://user:password@%2Fvar%2Flib%2Fpostgresql/mydb?connect_timeout=10` /// - `postgresql://user@host1:1234,host2,host3:5678?target_session_attrs=read-write` /// - `postgresql:///mydb?user=user&host=/var/lib/postgresql` /// /// For more information, please visit . pub fn connection_string(mut self, v: &str) -> Self { if !v.is_empty() { self.config.connection_string = Some(v.to_string()); } self } /// Set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the table name of the postgresql service to read/write. pub fn table(mut self, table: &str) -> Self { if !table.is_empty() { self.config.table = Some(table.to_string()); } self } /// Set the key field name of the postgresql service to read/write. /// /// Default to `key` if not specified. pub fn key_field(mut self, key_field: &str) -> Self { if !key_field.is_empty() { self.config.key_field = Some(key_field.to_string()); } self } /// Set the value field name of the postgresql service to read/write. /// /// Default to `value` if not specified. pub fn value_field(mut self, value_field: &str) -> Self { if !value_field.is_empty() { self.config.value_field = Some(value_field.to_string()); } self } } impl Builder for PostgresqlBuilder { const SCHEME: Scheme = Scheme::Postgresql; type Config = PostgresqlConfig; fn build(self) -> Result { let conn = match self.config.connection_string { Some(v) => v, None => { return Err( Error::new(ErrorKind::ConfigInvalid, "connection_string is empty") .with_context("service", Scheme::Postgresql), ) } }; let config = PgConnectOptions::from_str(&conn).map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "connection_string is invalid") .with_context("service", Scheme::Postgresql) .set_source(err) })?; let table = match self.config.table { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "table is empty") .with_context("service", Scheme::Postgresql)) } }; let key_field = self.config.key_field.unwrap_or_else(|| "key".to_string()); let value_field = self .config .value_field .unwrap_or_else(|| "value".to_string()); let root = normalize_root(self.config.root.unwrap_or_else(|| "/".to_string()).as_str()); Ok(PostgresqlBackend::new(Adapter { pool: OnceCell::new(), config, table, key_field, value_field, }) .with_normalized_root(root)) } } /// Backend for Postgresql service pub type PostgresqlBackend = kv::Backend; #[derive(Debug, Clone)] pub struct Adapter { pool: OnceCell, config: PgConnectOptions, table: String, key_field: String, value_field: String, } impl Adapter { async fn get_client(&self) -> Result<&PgPool> { self.pool .get_or_try_init(|| async { let pool = PgPool::connect_with(self.config.clone()) .await .map_err(parse_postgres_error)?; Ok(pool) }) .await } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Postgresql, &self.table, Capability { read: true, write: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let pool = self.get_client().await?; let value: Option> = sqlx::query_scalar(&format!( r#"SELECT "{}" FROM "{}" WHERE "{}" = $1 LIMIT 1"#, self.value_field, self.table, self.key_field )) .bind(path) .fetch_optional(pool) .await .map_err(parse_postgres_error)?; Ok(value.map(Buffer::from)) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let pool = self.get_client().await?; let table = &self.table; let key_field = &self.key_field; let value_field = &self.value_field; sqlx::query(&format!( r#"INSERT INTO "{table}" ("{key_field}", "{value_field}") VALUES ($1, $2) ON CONFLICT ("{key_field}") DO UPDATE SET "{value_field}" = EXCLUDED."{value_field}""#, )) .bind(path) .bind(value.to_vec()) .execute(pool) .await .map_err(parse_postgres_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let pool = self.get_client().await?; sqlx::query(&format!( "DELETE FROM {} WHERE {} = $1", self.table, self.key_field )) .bind(path) .execute(pool) .await .map_err(parse_postgres_error)?; Ok(()) } } fn parse_postgres_error(err: sqlx::Error) -> Error { Error::new(ErrorKind::Unexpected, "unhandled error from postgresql").set_source(err) } opendal-0.52.0/src/services/postgresql/config.rs000064400000000000000000000046441046102023000177720ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for PostgreSQL services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct PostgresqlConfig { /// Root of this backend. /// /// All operations will happen under this root. /// /// Default to `/` if not set. pub root: Option, /// The URL should be with a scheme of either `postgres://` or `postgresql://`. /// /// - `postgresql://user@localhost` /// - `postgresql://user:password@%2Fvar%2Flib%2Fpostgresql/mydb?connect_timeout=10` /// - `postgresql://user@host1:1234,host2,host3:5678?target_session_attrs=read-write` /// - `postgresql:///mydb?user=user&host=/var/lib/postgresql` /// /// For more information, please visit . pub connection_string: Option, /// the table of postgresql pub table: Option, /// the key field of postgresql pub key_field: Option, /// the value field of postgresql pub value_field: Option, } impl Debug for PostgresqlConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("PostgresqlConfig"); if self.connection_string.is_some() { d.field("connection_string", &""); } d.field("root", &self.root) .field("table", &self.table) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .finish() } } opendal-0.52.0/src/services/postgresql/docs.md000064400000000000000000000021401046102023000174160ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `connection_string`: Set the connection string of postgres server - `table`: Set the table of postgresql - `key_field`: Set the key field of postgresql - `value_field`: Set the value field of postgresql ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Postgresql; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Postgresql::default() .root("/") .connection_string("postgresql://you_username:your_password@127.0.0.1:5432/your_database") .table("your_table") // key field type in the table should be compatible with Rust's &str like text .key_field("key") // value field type in the table should be compatible with Rust's Vec like bytea .value_field("value"); let op = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/postgresql/mod.rs000064400000000000000000000017401046102023000172760ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-postgresql")] mod backend; #[cfg(feature = "services-postgresql")] pub use backend::PostgresqlBuilder as Postgresql; mod config; pub use config::PostgresqlConfig; opendal-0.52.0/src/services/redb/backend.rs000064400000000000000000000160551046102023000166240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use tokio::task; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::RedbConfig; use crate::Builder; use crate::Error; use crate::ErrorKind; use crate::Scheme; use crate::*; impl Configurator for RedbConfig { type Builder = RedbBuilder; fn into_builder(self) -> Self::Builder { RedbBuilder { config: self } } } /// Redb service support. #[doc = include_str!("docs.md")] #[derive(Default, Debug)] pub struct RedbBuilder { config: RedbConfig, } impl RedbBuilder { /// Set the path to the redb data directory. Will create if not exists. pub fn datadir(mut self, path: &str) -> Self { self.config.datadir = Some(path.into()); self } /// Set the table name for Redb. pub fn table(mut self, table: &str) -> Self { self.config.table = Some(table.into()); self } /// Set the root for Redb. pub fn root(mut self, path: &str) -> Self { self.config.root = Some(path.into()); self } } impl Builder for RedbBuilder { const SCHEME: Scheme = Scheme::Redb; type Config = RedbConfig; fn build(self) -> Result { let datadir_path = self.config.datadir.ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "datadir is required but not set") .with_context("service", Scheme::Redb) })?; let table_name = self.config.table.ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "table is required but not set") .with_context("service", Scheme::Redb) })?; let db = redb::Database::create(&datadir_path).map_err(parse_database_error)?; let db = Arc::new(db); Ok(RedbBackend::new(Adapter { datadir: datadir_path, table: table_name, db, }) .with_root(self.config.root.as_deref().unwrap_or_default())) } } /// Backend for Redb services. pub type RedbBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { datadir: String, table: String, db: Arc, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.field("path", &self.datadir); ds.finish() } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Redb, &self.datadir, Capability { read: true, write: true, blocking: true, shared: false, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_get(cloned_path.as_str())) .await .map_err(new_task_join_error) .and_then(|inner_result| inner_result) } fn blocking_get(&self, path: &str) -> Result> { let read_txn = self.db.begin_read().map_err(parse_transaction_error)?; let table_define: redb::TableDefinition<&str, &[u8]> = redb::TableDefinition::new(&self.table); let table = read_txn .open_table(table_define) .map_err(parse_table_error)?; let result = match table.get(path) { Ok(Some(v)) => Ok(Some(v.value().to_vec())), Ok(None) => Ok(None), Err(e) => Err(parse_storage_error(e)), }?; Ok(result.map(Buffer::from)) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_set(cloned_path.as_str(), value)) .await .map_err(new_task_join_error) .and_then(|inner_result| inner_result) } fn blocking_set(&self, path: &str, value: Buffer) -> Result<()> { let write_txn = self.db.begin_write().map_err(parse_transaction_error)?; let table_define: redb::TableDefinition<&str, &[u8]> = redb::TableDefinition::new(&self.table); { let mut table = write_txn .open_table(table_define) .map_err(parse_table_error)?; table .insert(path, &*value.to_vec()) .map_err(parse_storage_error)?; } write_txn.commit().map_err(parse_commit_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_delete(cloned_path.as_str())) .await .map_err(new_task_join_error) .and_then(|inner_result| inner_result) } fn blocking_delete(&self, path: &str) -> Result<()> { let write_txn = self.db.begin_write().map_err(parse_transaction_error)?; let table_define: redb::TableDefinition<&str, &[u8]> = redb::TableDefinition::new(&self.table); { let mut table = write_txn .open_table(table_define) .map_err(parse_table_error)?; table.remove(path).map_err(parse_storage_error)?; } write_txn.commit().map_err(parse_commit_error)?; Ok(()) } } fn parse_transaction_error(e: redb::TransactionError) -> Error { Error::new(ErrorKind::Unexpected, "error from redb").set_source(e) } fn parse_table_error(e: redb::TableError) -> Error { match e { redb::TableError::TableDoesNotExist(_) => { Error::new(ErrorKind::NotFound, "error from redb").set_source(e) } _ => Error::new(ErrorKind::Unexpected, "error from redb").set_source(e), } } fn parse_storage_error(e: redb::StorageError) -> Error { Error::new(ErrorKind::Unexpected, "error from redb").set_source(e) } fn parse_database_error(e: redb::DatabaseError) -> Error { Error::new(ErrorKind::Unexpected, "error from redb").set_source(e) } fn parse_commit_error(e: redb::CommitError) -> Error { Error::new(ErrorKind::Unexpected, "error from redb").set_source(e) } opendal-0.52.0/src/services/redb/config.rs000064400000000000000000000023251046102023000164750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// Config for redb service support. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct RedbConfig { /// path to the redb data directory. pub datadir: Option, /// The root for redb. pub root: Option, /// The table name for redb. pub table: Option, } opendal-0.52.0/src/services/redb/docs.md000064400000000000000000000012311046102023000161270ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [x] blocking ## Configuration - `datadir`: Set the path to the redb data directory You can refer to [`RedbBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Redb; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Redb::default() .datadir("/tmp/opendal/redb") .table("opendal-redb"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/redb/mod.rs000064400000000000000000000017021046102023000160050ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-redb")] mod backend; #[cfg(feature = "services-redb")] pub use backend::RedbBuilder as Redb; mod config; pub use config::RedbConfig; opendal-0.52.0/src/services/redis/backend.rs000064400000000000000000000266421046102023000170210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bb8::RunError; use http::Uri; use redis::cluster::ClusterClient; use redis::cluster::ClusterClientBuilder; use redis::Client; use redis::ConnectionAddr; use redis::ConnectionInfo; use redis::ProtocolVersion; use redis::RedisConnectionInfo; use std::fmt::Debug; use std::fmt::Formatter; use std::path::PathBuf; use std::time::Duration; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::RedisConfig; use crate::*; use super::core::*; const DEFAULT_REDIS_ENDPOINT: &str = "tcp://127.0.0.1:6379"; const DEFAULT_REDIS_PORT: u16 = 6379; impl Configurator for RedisConfig { type Builder = RedisBuilder; fn into_builder(self) -> Self::Builder { RedisBuilder { config: self } } } /// [Redis](https://redis.io/) services support. #[doc = include_str!("docs.md")] #[derive(Clone, Default)] pub struct RedisBuilder { config: RedisConfig, } impl Debug for RedisBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("RedisBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl RedisBuilder { /// set the network address of redis service. /// /// currently supported schemes: /// - no scheme: will be seen as "tcp" /// - "tcp" or "redis": unsecured redis connections /// - "rediss": secured redis connections /// - "unix" or "redis+unix": unix socket connection pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { self.config.endpoint = Some(endpoint.to_owned()); } self } /// set the network address of redis cluster service. /// This parameter is mutually exclusive with the endpoint parameter. /// /// currently supported schemes: /// - no scheme: will be seen as "tcp" /// - "tcp" or "redis": unsecured redis connections /// - "rediss": secured redis connections /// - "unix" or "redis+unix": unix socket connection pub fn cluster_endpoints(mut self, cluster_endpoints: &str) -> Self { if !cluster_endpoints.is_empty() { self.config.cluster_endpoints = Some(cluster_endpoints.to_owned()); } self } /// set the username for redis /// /// default: no username pub fn username(mut self, username: &str) -> Self { if !username.is_empty() { self.config.username = Some(username.to_owned()); } self } /// set the password for redis /// /// default: no password pub fn password(mut self, password: &str) -> Self { if !password.is_empty() { self.config.password = Some(password.to_owned()); } self } /// set the db used in redis /// /// default: 0 pub fn db(mut self, db: i64) -> Self { self.config.db = db; self } /// Set the default ttl for redis services. /// /// If set, we will specify `EX` for write operations. pub fn default_ttl(mut self, ttl: Duration) -> Self { self.config.default_ttl = Some(ttl); self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } } impl Builder for RedisBuilder { const SCHEME: Scheme = Scheme::Redis; type Config = RedisConfig; fn build(self) -> Result { let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); if let Some(endpoints) = self.config.cluster_endpoints.clone() { let mut cluster_endpoints: Vec = Vec::default(); for endpoint in endpoints.split(',') { cluster_endpoints.push(self.get_connection_info(endpoint.to_string())?); } let mut client_builder = ClusterClientBuilder::new(cluster_endpoints); if let Some(username) = &self.config.username { client_builder = client_builder.username(username.clone()); } if let Some(password) = &self.config.password { client_builder = client_builder.password(password.clone()); } let client = client_builder.build().map_err(format_redis_error)?; let conn = OnceCell::new(); Ok(RedisBackend::new(Adapter { addr: endpoints, client: None, cluster_client: Some(client), conn, default_ttl: self.config.default_ttl, }) .with_normalized_root(root)) } else { let endpoint = self .config .endpoint .clone() .unwrap_or_else(|| DEFAULT_REDIS_ENDPOINT.to_string()); let client = Client::open(self.get_connection_info(endpoint.clone())?).map_err(|e| { Error::new(ErrorKind::ConfigInvalid, "invalid or unsupported scheme") .with_context("service", Scheme::Redis) .with_context("endpoint", self.config.endpoint.as_ref().unwrap()) .with_context("db", self.config.db.to_string()) .set_source(e) })?; let conn = OnceCell::new(); Ok(RedisBackend::new(Adapter { addr: endpoint, client: Some(client), cluster_client: None, conn, default_ttl: self.config.default_ttl, }) .with_normalized_root(root)) } } } impl RedisBuilder { fn get_connection_info(&self, endpoint: String) -> Result { let ep_url = endpoint.parse::().map_err(|e| { Error::new(ErrorKind::ConfigInvalid, "endpoint is invalid") .with_context("service", Scheme::Redis) .with_context("endpoint", endpoint) .set_source(e) })?; let con_addr = match ep_url.scheme_str() { Some("tcp") | Some("redis") | None => { let host = ep_url .host() .map(|h| h.to_string()) .unwrap_or_else(|| "127.0.0.1".to_string()); let port = ep_url.port_u16().unwrap_or(DEFAULT_REDIS_PORT); ConnectionAddr::Tcp(host, port) } Some("rediss") => { let host = ep_url .host() .map(|h| h.to_string()) .unwrap_or_else(|| "127.0.0.1".to_string()); let port = ep_url.port_u16().unwrap_or(DEFAULT_REDIS_PORT); ConnectionAddr::TcpTls { host, port, insecure: false, tls_params: None, } } Some("unix") | Some("redis+unix") => { let path = PathBuf::from(ep_url.path()); ConnectionAddr::Unix(path) } Some(s) => { return Err( Error::new(ErrorKind::ConfigInvalid, "invalid or unsupported scheme") .with_context("service", Scheme::Redis) .with_context("scheme", s), ) } }; let redis_info = RedisConnectionInfo { db: self.config.db, username: self.config.username.clone(), password: self.config.password.clone(), protocol: ProtocolVersion::RESP2, }; Ok(ConnectionInfo { addr: con_addr, redis: redis_info, }) } } /// Backend for redis services. pub type RedisBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { addr: String, client: Option, cluster_client: Option, conn: OnceCell>, default_ttl: Option, } // implement `Debug` manually, or password may be leaked. impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.field("addr", &self.addr); ds.finish() } } impl Adapter { async fn conn(&self) -> Result> { let pool = self .conn .get_or_try_init(|| async { bb8::Pool::builder() .build(self.get_redis_connection_manager()) .await .map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "connect to redis failed") .set_source(err) }) }) .await?; pool.get().await.map_err(|err| match err { RunError::TimedOut => { Error::new(ErrorKind::Unexpected, "get connection from pool failed").set_temporary() } RunError::User(err) => err, }) } fn get_redis_connection_manager(&self) -> RedisConnectionManager { if let Some(_client) = self.client.clone() { RedisConnectionManager { client: self.client.clone(), cluster_client: None, } } else { RedisConnectionManager { client: None, cluster_client: self.cluster_client.clone(), } } } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Redis, self.addr.as_str(), Capability { read: true, write: true, shared: true, ..Default::default() }, ) } async fn get(&self, key: &str) -> Result> { let mut conn = self.conn().await?; let result = conn.get(key).await?; Ok(result) } async fn set(&self, key: &str, value: Buffer) -> Result<()> { let mut conn = self.conn().await?; let value = value.to_vec(); conn.set(key, value, self.default_ttl).await?; Ok(()) } async fn delete(&self, key: &str) -> Result<()> { let mut conn = self.conn().await?; conn.delete(key).await?; Ok(()) } async fn append(&self, key: &str, value: &[u8]) -> Result<()> { let mut conn = self.conn().await?; conn.append(key, value).await?; Ok(()) } } opendal-0.52.0/src/services/redis/config.rs000064400000000000000000000052541046102023000166730ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::time::Duration; use serde::Deserialize; use serde::Serialize; /// Config for Redis services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct RedisConfig { /// network address of the Redis service. Can be "tcp://127.0.0.1:6379", e.g. /// /// default is "tcp://127.0.0.1:6379" pub endpoint: Option, /// network address of the Redis cluster service. Can be "tcp://127.0.0.1:6379,tcp://127.0.0.1:6380,tcp://127.0.0.1:6381", e.g. /// /// default is None pub cluster_endpoints: Option, /// the username to connect redis service. /// /// default is None pub username: Option, /// the password for authentication /// /// default is None pub password: Option, /// the working directory of the Redis service. Can be "/path/to/dir" /// /// default is "/" pub root: Option, /// the number of DBs redis can take is unlimited /// /// default is db 0 pub db: i64, /// The default ttl for put operations. pub default_ttl: Option, } impl Debug for RedisConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("RedisConfig"); d.field("db", &self.db.to_string()); d.field("root", &self.root); if let Some(endpoint) = self.endpoint.clone() { d.field("endpoint", &endpoint); } if let Some(cluster_endpoints) = self.cluster_endpoints.clone() { d.field("cluster_endpoints", &cluster_endpoints); } if let Some(username) = self.username.clone() { d.field("username", &username); } if self.password.is_some() { d.field("password", &""); } d.finish_non_exhaustive() } } opendal-0.52.0/src/services/redis/core.rs000064400000000000000000000125021046102023000163500ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::Buffer; use crate::Error; use crate::ErrorKind; use redis::aio::ConnectionLike; use redis::aio::ConnectionManager; use redis::cluster::ClusterClient; use redis::cluster_async::ClusterConnection; use redis::from_redis_value; use redis::AsyncCommands; use redis::Client; use redis::RedisError; use std::time::Duration; #[derive(Clone)] pub enum RedisConnection { Normal(ConnectionManager), Cluster(ClusterConnection), } impl RedisConnection { pub async fn get(&mut self, key: &str) -> crate::Result> { let result: Option = match self { RedisConnection::Normal(ref mut conn) => { conn.get(key).await.map_err(format_redis_error) } RedisConnection::Cluster(ref mut conn) => { conn.get(key).await.map_err(format_redis_error) } }?; Ok(result.map(Buffer::from)) } pub async fn set( &mut self, key: &str, value: Vec, ttl: Option, ) -> crate::Result<()> { let value = value.to_vec(); if let Some(ttl) = ttl { match self { RedisConnection::Normal(ref mut conn) => conn .set_ex(key, value, ttl.as_secs()) .await .map_err(format_redis_error)?, RedisConnection::Cluster(ref mut conn) => conn .set_ex(key, value, ttl.as_secs()) .await .map_err(format_redis_error)?, } } else { match self { RedisConnection::Normal(ref mut conn) => { conn.set(key, value).await.map_err(format_redis_error)? } RedisConnection::Cluster(ref mut conn) => { conn.set(key, value).await.map_err(format_redis_error)? } } } Ok(()) } pub async fn delete(&mut self, key: &str) -> crate::Result<()> { match self { RedisConnection::Normal(ref mut conn) => { let _: () = conn.del(key).await.map_err(format_redis_error)?; } RedisConnection::Cluster(ref mut conn) => { let _: () = conn.del(key).await.map_err(format_redis_error)?; } } Ok(()) } pub async fn append(&mut self, key: &str, value: &[u8]) -> crate::Result<()> { match self { RedisConnection::Normal(ref mut conn) => { () = conn.append(key, value).await.map_err(format_redis_error)?; } RedisConnection::Cluster(ref mut conn) => { () = conn.append(key, value).await.map_err(format_redis_error)?; } } Ok(()) } } #[derive(Clone)] pub struct RedisConnectionManager { pub client: Option, pub cluster_client: Option, } #[async_trait::async_trait] impl bb8::ManageConnection for RedisConnectionManager { type Connection = RedisConnection; type Error = Error; async fn connect(&self) -> Result { if let Some(client) = self.client.clone() { ConnectionManager::new(client.clone()) .await .map_err(format_redis_error) .map(RedisConnection::Normal) } else { self.cluster_client .clone() .unwrap() .get_async_connection() .await .map_err(format_redis_error) .map(RedisConnection::Cluster) } } async fn is_valid(&self, conn: &mut Self::Connection) -> Result<(), Self::Error> { let pong_value = match conn { RedisConnection::Normal(ref mut conn) => conn .send_packed_command(&redis::cmd("PING")) .await .map_err(format_redis_error)?, RedisConnection::Cluster(ref mut conn) => conn .req_packed_command(&redis::cmd("PING")) .await .map_err(format_redis_error)?, }; let pong: String = from_redis_value(&pong_value).map_err(format_redis_error)?; if pong == "PONG" { Ok(()) } else { Err(Error::new(ErrorKind::Unexpected, "PING ERROR")) } } fn has_broken(&self, _: &mut Self::Connection) -> bool { false } } pub fn format_redis_error(e: RedisError) -> Error { Error::new(ErrorKind::Unexpected, e.category()) .set_source(e) .set_temporary() } opendal-0.52.0/src/services/redis/docs.md000064400000000000000000000017401046102023000163260ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `endpoint`: Set the network address of redis server - `cluster_endpoints`: Set the network address of redis cluster server. This parameter is mutually exclusive with the `endpoint` parameter. - `username`: Set the username of Redis - `password`: Set the password for authentication - `db`: Set the DB of redis You can refer to [`RedisBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Redis; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Redis::default(); // this will build a Operator accessing Redis which runs on tcp://localhost:6379 let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/redis/mod.rs000064400000000000000000000017631046102023000162060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-redis")] mod backend; #[cfg(feature = "services-redis")] pub use backend::RedisBuilder as Redis; mod config; #[cfg(feature = "services-redis")] mod core; pub use config::RedisConfig; opendal-0.52.0/src/services/rocksdb/backend.rs000064400000000000000000000135621046102023000173370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use rocksdb::DB; use tokio::task; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::RocksdbConfig; use crate::Result; use crate::*; impl Configurator for RocksdbConfig { type Builder = RocksdbBuilder; fn into_builder(self) -> Self::Builder { RocksdbBuilder { config: self } } } /// RocksDB service support. #[doc = include_str!("docs.md")] #[derive(Clone, Default)] pub struct RocksdbBuilder { config: RocksdbConfig, } impl RocksdbBuilder { /// Set the path to the rocksdb data directory. Will create if not exists. pub fn datadir(mut self, path: &str) -> Self { self.config.datadir = Some(path.into()); self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } } impl Builder for RocksdbBuilder { const SCHEME: Scheme = Scheme::Rocksdb; type Config = RocksdbConfig; fn build(self) -> Result { let path = self.config.datadir.ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "datadir is required but not set") .with_context("service", Scheme::Rocksdb) })?; let db = DB::open_default(&path).map_err(|e| { Error::new(ErrorKind::ConfigInvalid, "open default transaction db") .with_context("service", Scheme::Rocksdb) .with_context("datadir", path) .set_source(e) })?; let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); Ok(RocksdbBackend::new(Adapter { db: Arc::new(db) }).with_normalized_root(root)) } } /// Backend for rocksdb services. pub type RocksdbBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { db: Arc, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.field("path", &self.db.path()); ds.finish() } } impl kv::Adapter for Adapter { type Scanner = kv::Scanner; fn info(&self) -> kv::Info { kv::Info::new( Scheme::Rocksdb, &self.db.path().to_string_lossy(), Capability { read: true, write: true, list: true, blocking: true, shared: false, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_get(cloned_path.as_str())) .await .map_err(new_task_join_error)? } fn blocking_get(&self, path: &str) -> Result> { let result = self.db.get(path).map_err(parse_rocksdb_error)?; Ok(result.map(Buffer::from)) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_set(cloned_path.as_str(), value)) .await .map_err(new_task_join_error)? } fn blocking_set(&self, path: &str, value: Buffer) -> Result<()> { self.db .put(path, value.to_vec()) .map_err(parse_rocksdb_error) } async fn delete(&self, path: &str) -> Result<()> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_delete(cloned_path.as_str())) .await .map_err(new_task_join_error)? } fn blocking_delete(&self, path: &str) -> Result<()> { self.db.delete(path).map_err(parse_rocksdb_error) } async fn scan(&self, path: &str) -> Result { let cloned_self = self.clone(); let cloned_path = path.to_string(); let res = task::spawn_blocking(move || cloned_self.blocking_scan(cloned_path.as_str())) .await .map_err(new_task_join_error)??; Ok(Box::new(kv::ScanStdIter::new(res.into_iter().map(Ok)))) } /// TODO: we only need key here. fn blocking_scan(&self, path: &str) -> Result> { let it = self.db.prefix_iterator(path).map(|r| r.map(|(k, _)| k)); let mut res = Vec::default(); for key in it { let key = key.map_err(parse_rocksdb_error)?; let key = String::from_utf8_lossy(&key); if !key.starts_with(path) { break; } res.push(key.to_string()); } Ok(res) } } fn parse_rocksdb_error(e: rocksdb::Error) -> Error { Error::new(ErrorKind::Unexpected, "got rocksdb error").set_source(e) } opendal-0.52.0/src/services/rocksdb/config.rs000064400000000000000000000023421046102023000172070ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::Deserialize; use serde::Serialize; /// Config for Rocksdb Service. #[derive(Default, Debug, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct RocksdbConfig { /// The path to the rocksdb data directory. pub datadir: Option, /// the working directory of the service. Can be "/path/to/dir" /// /// default is "/" pub root: Option, } opendal-0.52.0/src/services/rocksdb/docs.md000064400000000000000000000023471046102023000166530ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [x] blocking ## Note OpenDAL will build rocksdb from source by default. To link with existing rocksdb lib, please set one of the following: - `ROCKSDB_LIB_DIR` to the dir that contains `librocksdb.so` - `ROCKSDB_STATIC` to the dir that contains `librocksdb.a` If the version of RocksDB is below 6.0, you may encounter compatibility issues. It is advisable to follow the steps provided in the [`INSTALL`](https://github.com/facebook/rocksdb/blob/main/INSTALL.md) file to build rocksdb, rather than relying on system libraries that may be outdated and incompatible. ## Configuration - `root`: Set the working directory of `OpenDAL` - `datadir`: Set the path to the rocksdb data directory You can refer to [`RocksdbBuilder`]'s docs for more information. ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Rocksdb; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Rocksdb::default() .datadir("/tmp/opendal/rocksdb"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/rocksdb/mod.rs000064400000000000000000000017211046102023000165210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-rocksdb")] mod backend; #[cfg(feature = "services-rocksdb")] pub use backend::RocksdbBuilder as Rocksdb; mod config; pub use config::RocksdbConfig; opendal-0.52.0/src/services/s3/backend.rs000064400000000000000000001225061046102023000162340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::fmt::Debug; use std::fmt::Formatter; use std::fmt::Write; use std::str::FromStr; use std::sync::atomic::AtomicBool; use std::sync::Arc; use base64::prelude::BASE64_STANDARD; use base64::Engine; use constants::X_AMZ_META_PREFIX; use http::Response; use http::StatusCode; use log::debug; use log::warn; use md5::Digest; use md5::Md5; use once_cell::sync::Lazy; use reqsign::AwsAssumeRoleLoader; use reqsign::AwsConfig; use reqsign::AwsCredentialLoad; use reqsign::AwsDefaultLoader; use reqsign::AwsV4Signer; use reqwest::Url; use super::core::*; use super::delete::S3Deleter; use super::error::parse_error; use super::lister::{S3Lister, S3Listers, S3ObjectVersionsLister}; use super::writer::S3Writer; use super::writer::S3Writers; use crate::raw::oio::PageLister; use crate::raw::*; use crate::services::S3Config; use crate::*; use constants::X_AMZ_VERSION_ID; /// Allow constructing correct region endpoint if user gives a global endpoint. static ENDPOINT_TEMPLATES: Lazy> = Lazy::new(|| { let mut m = HashMap::new(); // AWS S3 Service. m.insert( "https://s3.amazonaws.com", "https://s3.{region}.amazonaws.com", ); m }); const DEFAULT_BATCH_MAX_OPERATIONS: usize = 1000; impl Configurator for S3Config { type Builder = S3Builder; fn into_builder(self) -> Self::Builder { S3Builder { config: self, customized_credential_load: None, http_client: None, } } } /// Aws S3 and compatible services (including minio, digitalocean space, Tencent Cloud Object Storage(COS) and so on) support. /// For more information about s3-compatible services, refer to [Compatible Services](#compatible-services). #[doc = include_str!("docs.md")] #[doc = include_str!("compatible_services.md")] #[derive(Default)] pub struct S3Builder { config: S3Config, customized_credential_load: Option>, http_client: Option, } impl Debug for S3Builder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("S3Builder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl S3Builder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set bucket name of this backend. pub fn bucket(mut self, bucket: &str) -> Self { self.config.bucket = bucket.to_string(); self } /// Set endpoint of this backend. /// /// Endpoint must be full uri, e.g. /// /// - AWS S3: `https://s3.amazonaws.com` or `https://s3.{region}.amazonaws.com` /// - Cloudflare R2: `https://.r2.cloudflarestorage.com` /// - Aliyun OSS: `https://{region}.aliyuncs.com` /// - Tencent COS: `https://cos.{region}.myqcloud.com` /// - Minio: `http://127.0.0.1:9000` /// /// If user inputs endpoint without scheme like "s3.amazonaws.com", we /// will prepend "https://" before it. pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // Trim trailing `/` so that we can accept `http://127.0.0.1:9000/` self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()) } self } /// Region represent the signing region of this endpoint. This is required /// if you are using the default AWS S3 endpoint. /// /// If using a custom endpoint, /// - If region is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn region(mut self, region: &str) -> Self { if !region.is_empty() { self.config.region = Some(region.to_string()) } self } /// Set access_key_id of this backend. /// /// - If access_key_id is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn access_key_id(mut self, v: &str) -> Self { if !v.is_empty() { self.config.access_key_id = Some(v.to_string()) } self } /// Set secret_access_key of this backend. /// /// - If secret_access_key is set, we will take user's input first. /// - If not, we will try to load it from environment. pub fn secret_access_key(mut self, v: &str) -> Self { if !v.is_empty() { self.config.secret_access_key = Some(v.to_string()) } self } /// Set role_arn for this backend. /// /// If `role_arn` is set, we will use already known config as source /// credential to assume role with `role_arn`. pub fn role_arn(mut self, v: &str) -> Self { if !v.is_empty() { self.config.role_arn = Some(v.to_string()) } self } /// Set external_id for this backend. pub fn external_id(mut self, v: &str) -> Self { if !v.is_empty() { self.config.external_id = Some(v.to_string()) } self } /// Set role_session_name for this backend. pub fn role_session_name(mut self, v: &str) -> Self { if !v.is_empty() { self.config.role_session_name = Some(v.to_string()) } self } /// Set default storage_class for this backend. /// /// Available values: /// - `DEEP_ARCHIVE` /// - `GLACIER` /// - `GLACIER_IR` /// - `INTELLIGENT_TIERING` /// - `ONEZONE_IA` /// - `OUTPOSTS` /// - `REDUCED_REDUNDANCY` /// - `STANDARD` /// - `STANDARD_IA` pub fn default_storage_class(mut self, v: &str) -> Self { if !v.is_empty() { self.config.default_storage_class = Some(v.to_string()) } self } /// Set server_side_encryption for this backend. /// /// Available values: `AES256`, `aws:kms`. /// /// # Note /// /// This function is the low-level setting for SSE related features. /// /// SSE related options should be set carefully to make them works. /// Please use `server_side_encryption_with_*` helpers if even possible. pub fn server_side_encryption(mut self, v: &str) -> Self { if !v.is_empty() { self.config.server_side_encryption = Some(v.to_string()) } self } /// Set server_side_encryption_aws_kms_key_id for this backend /// /// - If `server_side_encryption` set to `aws:kms`, and `server_side_encryption_aws_kms_key_id` /// is not set, S3 will use aws managed kms key to encrypt data. /// - If `server_side_encryption` set to `aws:kms`, and `server_side_encryption_aws_kms_key_id` /// is a valid kms key id, S3 will use the provided kms key to encrypt data. /// - If the `server_side_encryption_aws_kms_key_id` is invalid or not found, an error will be /// returned. /// - If `server_side_encryption` is not `aws:kms`, setting `server_side_encryption_aws_kms_key_id` is a noop. /// /// # Note /// /// This function is the low-level setting for SSE related features. /// /// SSE related options should be set carefully to make them works. /// Please use `server_side_encryption_with_*` helpers if even possible. pub fn server_side_encryption_aws_kms_key_id(mut self, v: &str) -> Self { if !v.is_empty() { self.config.server_side_encryption_aws_kms_key_id = Some(v.to_string()) } self } /// Set server_side_encryption_customer_algorithm for this backend. /// /// Available values: `AES256`. /// /// # Note /// /// This function is the low-level setting for SSE related features. /// /// SSE related options should be set carefully to make them works. /// Please use `server_side_encryption_with_*` helpers if even possible. pub fn server_side_encryption_customer_algorithm(mut self, v: &str) -> Self { if !v.is_empty() { self.config.server_side_encryption_customer_algorithm = Some(v.to_string()) } self } /// Set server_side_encryption_customer_key for this backend. /// /// # Args /// /// `v`: base64 encoded key that matches algorithm specified in /// `server_side_encryption_customer_algorithm`. /// /// # Note /// /// This function is the low-level setting for SSE related features. /// /// SSE related options should be set carefully to make them works. /// Please use `server_side_encryption_with_*` helpers if even possible. pub fn server_side_encryption_customer_key(mut self, v: &str) -> Self { if !v.is_empty() { self.config.server_side_encryption_customer_key = Some(v.to_string()) } self } /// Set server_side_encryption_customer_key_md5 for this backend. /// /// # Args /// /// `v`: MD5 digest of key specified in `server_side_encryption_customer_key`. /// /// # Note /// /// This function is the low-level setting for SSE related features. /// /// SSE related options should be set carefully to make them works. /// Please use `server_side_encryption_with_*` helpers if even possible. pub fn server_side_encryption_customer_key_md5(mut self, v: &str) -> Self { if !v.is_empty() { self.config.server_side_encryption_customer_key_md5 = Some(v.to_string()) } self } /// Enable server side encryption with aws managed kms key /// /// As known as: SSE-KMS /// /// NOTE: This function should not be used along with other `server_side_encryption_with_` functions. pub fn server_side_encryption_with_aws_managed_kms_key(mut self) -> Self { self.config.server_side_encryption = Some("aws:kms".to_string()); self } /// Enable server side encryption with customer managed kms key /// /// As known as: SSE-KMS /// /// NOTE: This function should not be used along with other `server_side_encryption_with_` functions. pub fn server_side_encryption_with_customer_managed_kms_key( mut self, aws_kms_key_id: &str, ) -> Self { self.config.server_side_encryption = Some("aws:kms".to_string()); self.config.server_side_encryption_aws_kms_key_id = Some(aws_kms_key_id.to_string()); self } /// Enable server side encryption with s3 managed key /// /// As known as: SSE-S3 /// /// NOTE: This function should not be used along with other `server_side_encryption_with_` functions. pub fn server_side_encryption_with_s3_key(mut self) -> Self { self.config.server_side_encryption = Some("AES256".to_string()); self } /// Enable server side encryption with customer key. /// /// As known as: SSE-C /// /// NOTE: This function should not be used along with other `server_side_encryption_with_` functions. pub fn server_side_encryption_with_customer_key(mut self, algorithm: &str, key: &[u8]) -> Self { self.config.server_side_encryption_customer_algorithm = Some(algorithm.to_string()); self.config.server_side_encryption_customer_key = Some(BASE64_STANDARD.encode(key)); self.config.server_side_encryption_customer_key_md5 = Some(BASE64_STANDARD.encode(Md5::digest(key).as_slice())); self } /// Set temporary credential used in AWS S3 connections /// /// # Warning /// /// session token's lifetime is short and requires users to refresh in time. pub fn session_token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.session_token = Some(token.to_string()); } self } /// Set temporary credential used in AWS S3 connections #[deprecated(note = "Please use `session_token` instead")] pub fn security_token(self, token: &str) -> Self { self.session_token(token) } /// Disable config load so that opendal will not load config from /// environment. /// /// For examples: /// /// - envs like `AWS_ACCESS_KEY_ID` /// - files like `~/.aws/config` pub fn disable_config_load(mut self) -> Self { self.config.disable_config_load = true; self } /// Disable load credential from ec2 metadata. /// /// This option is used to disable the default behavior of opendal /// to load credential from ec2 metadata, a.k.a, IMDSv2 pub fn disable_ec2_metadata(mut self) -> Self { self.config.disable_ec2_metadata = true; self } /// Allow anonymous will allow opendal to send request without signing /// when credential is not loaded. pub fn allow_anonymous(mut self) -> Self { self.config.allow_anonymous = true; self } /// Enable virtual host style so that opendal will send API requests /// in virtual host style instead of path style. /// /// - By default, opendal will send API to `https://s3.us-east-1.amazonaws.com/bucket_name` /// - Enabled, opendal will send API to `https://bucket_name.s3.us-east-1.amazonaws.com` pub fn enable_virtual_host_style(mut self) -> Self { self.config.enable_virtual_host_style = true; self } /// Disable stat with override so that opendal will not send stat request with override queries. /// /// For example, R2 doesn't support stat with `response_content_type` query. pub fn disable_stat_with_override(mut self) -> Self { self.config.disable_stat_with_override = true; self } /// Adding a customized credential load for service. /// /// If customized_credential_load has been set, we will ignore all other /// credential load methods. pub fn customized_credential_load(mut self, cred: Box) -> Self { self.customized_credential_load = Some(cred); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } /// Set bucket versioning status for this backend pub fn enable_versioning(mut self, enabled: bool) -> Self { self.config.enable_versioning = enabled; self } /// Check if `bucket` is valid /// `bucket` must be not empty and if `enable_virtual_host_style` is true /// it couldn't contain dot(.) character fn is_bucket_valid(&self) -> bool { if self.config.bucket.is_empty() { return false; } // If enable virtual host style, `bucket` will reside in domain part, // for example `https://bucket_name.s3.us-east-1.amazonaws.com`, // so `bucket` with dot can't be recognized correctly for this format. if self.config.enable_virtual_host_style && self.config.bucket.contains('.') { return false; } true } /// Build endpoint with given region. fn build_endpoint(&self, region: &str) -> String { let bucket = { debug_assert!(self.is_bucket_valid(), "bucket must be valid"); self.config.bucket.as_str() }; let mut endpoint = match &self.config.endpoint { Some(endpoint) => { if endpoint.starts_with("http") { endpoint.to_string() } else { // Prefix https if endpoint doesn't start with scheme. format!("https://{endpoint}") } } None => "https://s3.amazonaws.com".to_string(), }; // If endpoint contains bucket name, we should trim them. endpoint = endpoint.replace(&format!("//{bucket}."), "//"); // Omit default ports if specified. if let Ok(url) = Url::from_str(&endpoint) { // Remove the trailing `/` of root path. endpoint = url.to_string().trim_end_matches('/').to_string(); } // Update with endpoint templates. endpoint = if let Some(template) = ENDPOINT_TEMPLATES.get(endpoint.as_str()) { template.replace("{region}", region) } else { // If we don't know where about this endpoint, just leave // them as it. endpoint.to_string() }; // Apply virtual host style. if self.config.enable_virtual_host_style { endpoint = endpoint.replace("//", &format!("//{bucket}.")) } else { write!(endpoint, "/{bucket}").expect("write into string must succeed"); }; endpoint } /// Set maximum batch operations of this backend. #[deprecated( since = "0.52.0", note = "Please use `delete_max_size` instead of `batch_max_operations`" )] pub fn batch_max_operations(mut self, batch_max_operations: usize) -> Self { self.config.delete_max_size = Some(batch_max_operations); self } /// Set maximum delete operations of this backend. pub fn delete_max_size(mut self, delete_max_size: usize) -> Self { self.config.delete_max_size = Some(delete_max_size); self } /// Set checksum algorithm of this backend. /// This is necessary when writing to AWS S3 Buckets with Object Lock enabled for example. /// /// Available options: /// - "crc32c" pub fn checksum_algorithm(mut self, checksum_algorithm: &str) -> Self { self.config.checksum_algorithm = Some(checksum_algorithm.to_string()); self } /// Disable write with if match so that opendal will not send write request with if match headers. pub fn disable_write_with_if_match(mut self) -> Self { self.config.disable_write_with_if_match = true; self } /// Detect region of S3 bucket. /// /// # Args /// /// - endpoint: the endpoint of S3 service /// - bucket: the bucket of S3 service /// /// # Return /// /// - `Some(region)` means we detect the region successfully /// - `None` means we can't detect the region or meeting errors. /// /// # Notes /// /// We will try to detect region by the following methods. /// /// - Match endpoint with given rules to get region /// - Cloudflare R2 /// - AWS S3 /// - Aliyun OSS /// - Send a `HEAD` request to endpoint with bucket name to get `x-amz-bucket-region`. /// /// # Examples /// /// ```no_run /// use opendal::services::S3; /// /// # async fn example() { /// let region: Option = S3::detect_region("https://s3.amazonaws.com", "example").await; /// # } /// ``` /// /// # Reference /// /// - [Amazon S3 HeadBucket API](https://docs.aws.amazon.com/zh_cn/AmazonS3/latest/API/API_HeadBucket.html) pub async fn detect_region(endpoint: &str, bucket: &str) -> Option { // Remove the possible trailing `/` in endpoint. let endpoint = endpoint.trim_end_matches('/'); // Make sure the endpoint contains the scheme. let mut endpoint = if endpoint.starts_with("http") { endpoint.to_string() } else { // Prefix https if endpoint doesn't start with scheme. format!("https://{}", endpoint) }; // Remove bucket name from endpoint. endpoint = endpoint.replace(&format!("//{bucket}."), "//"); let url = format!("{endpoint}/{bucket}"); debug!("detect region with url: {url}"); // Try to detect region by endpoint. // If this bucket is R2, we can return auto directly. // // Reference: if endpoint.ends_with("r2.cloudflarestorage.com") { return Some("auto".to_string()); } // If this bucket is AWS, we can try to match the endpoint. if let Some(v) = endpoint.strip_prefix("https://s3.") { if let Some(region) = v.strip_suffix(".amazonaws.com") { return Some(region.to_string()); } } // If this bucket is OSS, we can try to match the endpoint. // // - `oss-ap-southeast-1.aliyuncs.com` => `oss-ap-southeast-1` // - `oss-cn-hangzhou-internal.aliyuncs.com` => `oss-cn-hangzhou` if let Some(v) = endpoint.strip_prefix("https://") { if let Some(region) = v.strip_suffix(".aliyuncs.com") { return Some(region.to_string()); } if let Some(region) = v.strip_suffix("-internal.aliyuncs.com") { return Some(region.to_string()); } } // Try to detect region by HeadBucket. let req = http::Request::head(&url).body(Buffer::new()).ok()?; let client = HttpClient::new().ok()?; let res = client .send(req) .await .map_err(|err| warn!("detect region failed for: {err:?}")) .ok()?; debug!( "auto detect region got response: status {:?}, header: {:?}", res.status(), res.headers() ); // Get region from response header no matter status code. if let Some(header) = res.headers().get("x-amz-bucket-region") { if let Ok(regin) = header.to_str() { return Some(regin.to_string()); } } // Status code is 403 or 200 means we already visit the correct // region, we can use the default region directly. if res.status() == StatusCode::FORBIDDEN || res.status() == StatusCode::OK { return Some("us-east-1".to_string()); } None } } impl Builder for S3Builder { const SCHEME: Scheme = Scheme::S3; type Config = S3Config; fn build(mut self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle bucket name. let bucket = if self.is_bucket_valid() { Ok(&self.config.bucket) } else { Err( Error::new(ErrorKind::ConfigInvalid, "The bucket is misconfigured") .with_context("service", Scheme::S3), ) }?; debug!("backend use bucket {}", &bucket); let default_storage_class = match &self.config.default_storage_class { None => None, Some(v) => Some( build_header_value(v).map_err(|err| err.with_context("key", "storage_class"))?, ), }; let server_side_encryption = match &self.config.server_side_encryption { None => None, Some(v) => Some( build_header_value(v) .map_err(|err| err.with_context("key", "server_side_encryption"))?, ), }; let server_side_encryption_aws_kms_key_id = match &self.config.server_side_encryption_aws_kms_key_id { None => None, Some(v) => Some(build_header_value(v).map_err(|err| { err.with_context("key", "server_side_encryption_aws_kms_key_id") })?), }; let server_side_encryption_customer_algorithm = match &self.config.server_side_encryption_customer_algorithm { None => None, Some(v) => Some(build_header_value(v).map_err(|err| { err.with_context("key", "server_side_encryption_customer_algorithm") })?), }; let server_side_encryption_customer_key = match &self.config.server_side_encryption_customer_key { None => None, Some(v) => Some(build_header_value(v).map_err(|err| { err.with_context("key", "server_side_encryption_customer_key") })?), }; let server_side_encryption_customer_key_md5 = match &self.config.server_side_encryption_customer_key_md5 { None => None, Some(v) => Some(build_header_value(v).map_err(|err| { err.with_context("key", "server_side_encryption_customer_key_md5") })?), }; let checksum_algorithm = match self.config.checksum_algorithm.as_deref() { Some("crc32c") => Some(ChecksumAlgorithm::Crc32c), None => None, v => { return Err(Error::new( ErrorKind::ConfigInvalid, format!("{:?} is not a supported checksum_algorithm.", v), )) } }; // This is our current config. let mut cfg = AwsConfig::default(); if !self.config.disable_config_load { #[cfg(not(target_arch = "wasm32"))] { cfg = cfg.from_profile(); cfg = cfg.from_env(); } } if let Some(ref v) = self.config.region { cfg.region = Some(v.to_string()); } if cfg.region.is_none() { return Err(Error::new( ErrorKind::ConfigInvalid, "region is missing. Please find it by S3::detect_region() or set them in env.", ) .with_operation("Builder::build") .with_context("service", Scheme::S3)); } let region = cfg.region.to_owned().unwrap(); debug!("backend use region: {region}"); // Retain the user's endpoint if it exists; otherwise, try loading it from the environment. self.config.endpoint = self.config.endpoint.or_else(|| cfg.endpoint_url.clone()); // Building endpoint. let endpoint = self.build_endpoint(®ion); debug!("backend use endpoint: {endpoint}"); // Setting all value from user input if available. if let Some(v) = self.config.access_key_id { cfg.access_key_id = Some(v) } if let Some(v) = self.config.secret_access_key { cfg.secret_access_key = Some(v) } if let Some(v) = self.config.session_token { cfg.session_token = Some(v) } let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::S3) })? }; let mut loader: Option> = None; // If customized_credential_load is set, we will use it. if let Some(v) = self.customized_credential_load { loader = Some(v); } // If role_arn is set, we must use AssumeRoleLoad. if let Some(role_arn) = self.config.role_arn { // use current env as source credential loader. let default_loader = AwsDefaultLoader::new(GLOBAL_REQWEST_CLIENT.clone().clone(), cfg.clone()); // Build the config for assume role. let mut assume_role_cfg = AwsConfig { region: Some(region.clone()), role_arn: Some(role_arn), external_id: self.config.external_id.clone(), sts_regional_endpoints: "regional".to_string(), ..Default::default() }; // override default role_session_name if set if let Some(name) = self.config.role_session_name { assume_role_cfg.role_session_name = name; } let assume_role_loader = AwsAssumeRoleLoader::new( GLOBAL_REQWEST_CLIENT.clone().clone(), assume_role_cfg, Box::new(default_loader), ) .map_err(|err| { Error::new( ErrorKind::ConfigInvalid, "The assume_role_loader is misconfigured", ) .with_context("service", Scheme::S3) .set_source(err) })?; loader = Some(Box::new(assume_role_loader)); } // If loader is not set, we will use default loader. let loader = match loader { Some(v) => v, None => { let mut default_loader = AwsDefaultLoader::new(GLOBAL_REQWEST_CLIENT.clone().clone(), cfg); if self.config.disable_ec2_metadata { default_loader = default_loader.with_disable_ec2_metadata(); } Box::new(default_loader) } }; let signer = AwsV4Signer::new("s3", ®ion); let delete_max_size = self .config .delete_max_size .unwrap_or(DEFAULT_BATCH_MAX_OPERATIONS); Ok(S3Backend { core: Arc::new(S3Core { bucket: bucket.to_string(), endpoint, root, server_side_encryption, server_side_encryption_aws_kms_key_id, server_side_encryption_customer_algorithm, server_side_encryption_customer_key, server_side_encryption_customer_key_md5, default_storage_class, allow_anonymous: self.config.allow_anonymous, disable_stat_with_override: self.config.disable_stat_with_override, enable_versioning: self.config.enable_versioning, signer, loader, credential_loaded: AtomicBool::new(false), client, checksum_algorithm, delete_max_size, disable_write_with_if_match: self.config.disable_write_with_if_match, }), }) } } /// Backend for s3 services. #[derive(Debug, Clone)] pub struct S3Backend { core: Arc, } impl Access for S3Backend { type Reader = HttpBody; type Writer = S3Writers; type Lister = S3Listers; type Deleter = oio::BatchDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::S3) .set_root(&self.core.root) .set_name(&self.core.bucket) .set_native_capability(Capability { stat: true, stat_has_content_encoding: true, stat_with_if_match: true, stat_with_if_none_match: true, stat_with_if_modified_since: true, stat_with_if_unmodified_since: true, stat_with_override_cache_control: !self.core.disable_stat_with_override, stat_with_override_content_disposition: !self.core.disable_stat_with_override, stat_with_override_content_type: !self.core.disable_stat_with_override, stat_with_version: self.core.enable_versioning, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, stat_has_user_metadata: true, stat_has_version: true, read: true, read_with_if_match: true, read_with_if_none_match: true, read_with_if_modified_since: true, read_with_if_unmodified_since: true, read_with_override_cache_control: true, read_with_override_content_disposition: true, read_with_override_content_type: true, read_with_version: self.core.enable_versioning, write: true, write_can_empty: true, write_can_multi: true, write_with_cache_control: true, write_with_content_type: true, write_with_content_encoding: true, write_with_if_match: !self.core.disable_write_with_if_match, write_with_if_not_exists: true, write_with_user_metadata: true, write_has_content_length: true, write_has_etag: true, write_has_version: self.core.enable_versioning, // The min multipart size of S3 is 5 MiB. // // ref: write_multi_min_size: Some(5 * 1024 * 1024), // The max multipart size of S3 is 5 GiB. // // ref: write_multi_max_size: if cfg!(target_pointer_width = "64") { Some(5 * 1024 * 1024 * 1024) } else { Some(usize::MAX) }, delete: true, delete_max_size: Some(self.core.delete_max_size), delete_with_version: self.core.enable_versioning, copy: true, list: true, list_with_limit: true, list_with_start_after: true, list_with_recursive: true, list_with_versions: self.core.enable_versioning, list_with_deleted: self.core.enable_versioning, list_has_etag: true, list_has_content_md5: true, list_has_content_length: true, list_has_last_modified: true, presign: true, presign_stat: true, presign_read: true, presign_write: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, args: OpStat) -> Result { let resp = self.core.s3_head_object(path, args).await?; let status = resp.status(); match status { StatusCode::OK => { let headers = resp.headers(); let mut meta = parse_into_metadata(path, headers)?; let user_meta = parse_prefixed_headers(headers, X_AMZ_META_PREFIX); if !user_meta.is_empty() { meta.with_user_metadata(user_meta); } if let Some(v) = parse_header_to_str(headers, X_AMZ_VERSION_ID)? { meta.set_version(v); } Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.s3_get_object(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let concurrent = args.concurrent(); let executor = args.executor().cloned(); let writer = S3Writer::new(self.core.clone(), path, args); let w = oio::MultipartWriter::new(writer, executor, concurrent); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::BatchDeleter::new(S3Deleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = if args.versions() || args.deleted() { TwoWays::Two(PageLister::new(S3ObjectVersionsLister::new( self.core.clone(), path, args, ))) } else { TwoWays::One(PageLister::new(S3Lister::new( self.core.clone(), path, args, ))) }; Ok((RpList::default(), l)) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.s3_copy_object(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn presign(&self, path: &str, args: OpPresign) -> Result { let (expire, op) = args.into_parts(); // We will not send this request out, just for signing. let mut req = match op { PresignOperation::Stat(v) => self.core.s3_head_object_request(path, v)?, PresignOperation::Read(v) => { self.core .s3_get_object_request(path, BytesRange::default(), &v)? } PresignOperation::Write(_) => { self.core .s3_put_object_request(path, None, &OpWrite::default(), Buffer::new())? } }; self.core.sign_query(&mut req, expire).await?; // We don't need this request anymore, consume it directly. let (parts, _) = req.into_parts(); Ok(RpPresign::new(PresignedRequest::new( parts.method, parts.uri, parts.headers, ))) } } #[cfg(test)] mod tests { use super::*; #[test] fn test_is_valid_bucket() { let bucket_cases = vec![ ("", false, false), ("test", false, true), ("test.xyz", false, true), ("", true, false), ("test", true, true), ("test.xyz", true, false), ]; for (bucket, enable_virtual_host_style, expected) in bucket_cases { let mut b = S3Builder::default(); b = b.bucket(bucket); if enable_virtual_host_style { b = b.enable_virtual_host_style(); } assert_eq!(b.is_bucket_valid(), expected) } } #[test] fn test_build_endpoint() { let _ = tracing_subscriber::fmt().with_test_writer().try_init(); let endpoint_cases = vec![ Some("s3.amazonaws.com"), Some("https://s3.amazonaws.com"), Some("https://s3.us-east-2.amazonaws.com"), None, ]; for endpoint in &endpoint_cases { let mut b = S3Builder::default().bucket("test"); if let Some(endpoint) = endpoint { b = b.endpoint(endpoint); } let endpoint = b.build_endpoint("us-east-2"); assert_eq!(endpoint, "https://s3.us-east-2.amazonaws.com/test"); } for endpoint in &endpoint_cases { let mut b = S3Builder::default() .bucket("test") .enable_virtual_host_style(); if let Some(endpoint) = endpoint { b = b.endpoint(endpoint); } let endpoint = b.build_endpoint("us-east-2"); assert_eq!(endpoint, "https://test.s3.us-east-2.amazonaws.com"); } } #[tokio::test] async fn test_detect_region() { let cases = vec![ ( "aws s3 without region in endpoint", "https://s3.amazonaws.com", "example", Some("us-east-1"), ), ( "aws s3 with region in endpoint", "https://s3.us-east-1.amazonaws.com", "example", Some("us-east-1"), ), ( "oss with public endpoint", "https://oss-ap-southeast-1.aliyuncs.com", "example", Some("oss-ap-southeast-1"), ), ( "oss with internal endpoint", "https://oss-cn-hangzhou-internal.aliyuncs.com", "example", Some("oss-cn-hangzhou-internal"), ), ( "r2", "https://abc.xxxxx.r2.cloudflarestorage.com", "example", Some("auto"), ), ( "invalid service", "https://opendal.apache.org", "example", None, ), ]; for (name, endpoint, bucket, expected) in cases { let region = S3Builder::detect_region(endpoint, bucket).await; assert_eq!(region.as_deref(), expected, "{}", name); } } } opendal-0.52.0/src/services/s3/compatible_services.md000064400000000000000000000117531046102023000206440ustar 00000000000000 ## Compatible Services ### AWS S3 [AWS S3](https://aws.amazon.com/s3/) is the default implementations of s3 services. Only `bucket` is required. ```rust,ignore builder.bucket(""); ``` ### Alibaba Object Storage Service (OSS) [OSS](https://www.alibabacloud.com/product/object-storage-service) is a s3 compatible service provided by [Alibaba Cloud](https://www.alibabacloud.com). To connect to OSS, we need to set: - `endpoint`: The endpoint of oss, for example: `https://oss-cn-hangzhou.aliyuncs.com` - `bucket`: The bucket name of oss. > OSS provide internal endpoint for used at alibabacloud internally, please visit [OSS Regions and endpoints](https://www.alibabacloud.com/help/en/object-storage-service/latest/regions-and-endpoints) for more details. > OSS only supports the virtual host style, users could meet errors like: > > ```xml > > > SecondLevelDomainForbidden > The bucket you are attempting to access must be addressed using OSS third level domain. > 62A1C265292C0632377F021F > oss-cn-hangzhou.aliyuncs.com > > ``` > > In that case, please enable virtual host style for requesting. ```rust,ignore builder.endpoint("https://oss-cn-hangzhou.aliyuncs.com"); builder.region(""); builder.bucket(""); builder.enable_virtual_host_style(); ``` ### Minio [minio](https://min.io/) is an open-source s3 compatible services. To connect to minio, we need to set: - `endpoint`: The endpoint of minio, for example: `http://127.0.0.1:9000` - `region`: The region of minio. If you don't care about it, just set it to "auto", it will be ignored. - `bucket`: The bucket name of minio. ```rust,ignore builder.endpoint("http://127.0.0.1:9000"); builder.region(""); builder.bucket(""); ``` ### QingStor Object Storage [QingStor Object Storage](https://www.qingcloud.com/products/qingstor) is a S3-compatible service provided by [QingCloud](https://www.qingcloud.com/). To connect to QingStor Object Storage, we need to set: - `endpoint`: The endpoint of QingStor s3 compatible endpoint, for example: `https://s3.pek3b.qingstor.com` - `bucket`: The bucket name. ### Scaleway Object Storage [Scaleway Object Storage](https://www.scaleway.com/en/object-storage/) is a S3-compatible and multi-AZ redundant object storage service. To connect to Scaleway Object Storage, we need to set: - `endpoint`: The endpoint of scaleway, for example: `https://s3.nl-ams.scw.cloud` - `region`: The region of scaleway. - `bucket`: The bucket name of scaleway. ### Tencent Cloud Object Storage (COS) [COS](https://intl.cloud.tencent.com/products/cos) is a s3 compatible service provided by [Tencent Cloud](https://intl.cloud.tencent.com/). To connect to COS, we need to set: - `endpoint`: The endpoint of cos, for example: `https://cos.ap-beijing.myqcloud.com` - `bucket`: The bucket name of cos. ### Wasabi Object Storage [Wasabi](https://wasabi.com/) is a s3 compatible service. > Cloud storage pricing that is 80% less than Amazon S3. To connect to wasabi, we need to set: - `endpoint`: The endpoint of wasabi, for example: `https://s3.us-east-2.wasabisys.com` - `bucket`: The bucket name of wasabi. > Refer to [What are the service URLs for Wasabi's different storage regions?](https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031) for more details. ### Cloudflare R2 [Cloudflare R2](https://developers.cloudflare.com/r2/) provides s3 compatible API. > Cloudflare R2 Storage allows developers to store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services. To connect to r2, we need to set: - `endpoint`: The endpoint of r2, for example: `https://.r2.cloudflarestorage.com` - `bucket`: The bucket name of r2. - `region`: When you create a new bucket, the data location is set to Automatic by default. So please use `auto` for region. - `batch_max_operations`: R2's delete objects will return `Internal Error` if the batch is larger than `700`. Please set this value `<= 700` to make sure batch delete work as expected. - `enable_exact_buf_write`: R2 requires the non-tailing parts size to be exactly the same. Please enable this option to avoid the error `All non-trailing parts must have the same length`. ### Google Cloud Storage XML API [Google Cloud Storage XML API](https://cloud.google.com/storage/docs/xml-api/overview) provides s3 compatible API. - `endpoint`: The endpoint of Google Cloud Storage XML API, for example: `https://storage.googleapis.com` - `bucket`: The bucket name. - To access GCS via S3 API, please enable `features = ["native-tls"]` in your `Cargo.toml` to avoid connection being reset when using `rustls`. Tracking in ### Ceph Rados Gateway Ceph supports a RESTful API that is compatible with the basic data access model of the Amazon S3 API. For more information, refer: opendal-0.52.0/src/services/s3/config.rs000064400000000000000000000203671046102023000161140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Aws S3 and compatible services (including minio, digitalocean space, Tencent Cloud Object Storage(COS) and so on) support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct S3Config { /// root of this backend. /// /// All operations will happen under this root. /// /// default to `/` if not set. pub root: Option, /// bucket name of this backend. /// /// required. pub bucket: String, /// is bucket versioning enabled for this bucket pub enable_versioning: bool, /// endpoint of this backend. /// /// Endpoint must be full uri, e.g. /// /// - AWS S3: `https://s3.amazonaws.com` or `https://s3.{region}.amazonaws.com` /// - Cloudflare R2: `https://.r2.cloudflarestorage.com` /// - Aliyun OSS: `https://{region}.aliyuncs.com` /// - Tencent COS: `https://cos.{region}.myqcloud.com` /// - Minio: `http://127.0.0.1:9000` /// /// If user inputs endpoint without scheme like "s3.amazonaws.com", we /// will prepend "https://" before it. /// /// - If endpoint is set, we will take user's input first. /// - If not, we will try to load it from environment. /// - If still not set, default to `https://s3.amazonaws.com`. pub endpoint: Option, /// Region represent the signing region of this endpoint. This is required /// if you are using the default AWS S3 endpoint. /// /// If using a custom endpoint, /// - If region is set, we will take user's input first. /// - If not, we will try to load it from environment. pub region: Option, /// access_key_id of this backend. /// /// - If access_key_id is set, we will take user's input first. /// - If not, we will try to load it from environment. pub access_key_id: Option, /// secret_access_key of this backend. /// /// - If secret_access_key is set, we will take user's input first. /// - If not, we will try to load it from environment. pub secret_access_key: Option, /// session_token (aka, security token) of this backend. /// /// This token will expire after sometime, it's recommended to set session_token /// by hand. pub session_token: Option, /// role_arn for this backend. /// /// If `role_arn` is set, we will use already known config as source /// credential to assume role with `role_arn`. pub role_arn: Option, /// external_id for this backend. pub external_id: Option, /// role_session_name for this backend. pub role_session_name: Option, /// Disable config load so that opendal will not load config from /// environment. /// /// For examples: /// /// - envs like `AWS_ACCESS_KEY_ID` /// - files like `~/.aws/config` pub disable_config_load: bool, /// Disable load credential from ec2 metadata. /// /// This option is used to disable the default behavior of opendal /// to load credential from ec2 metadata, a.k.a, IMDSv2 pub disable_ec2_metadata: bool, /// Allow anonymous will allow opendal to send request without signing /// when credential is not loaded. pub allow_anonymous: bool, /// server_side_encryption for this backend. /// /// Available values: `AES256`, `aws:kms`. pub server_side_encryption: Option, /// server_side_encryption_aws_kms_key_id for this backend /// /// - If `server_side_encryption` set to `aws:kms`, and `server_side_encryption_aws_kms_key_id` /// is not set, S3 will use aws managed kms key to encrypt data. /// - If `server_side_encryption` set to `aws:kms`, and `server_side_encryption_aws_kms_key_id` /// is a valid kms key id, S3 will use the provided kms key to encrypt data. /// - If the `server_side_encryption_aws_kms_key_id` is invalid or not found, an error will be /// returned. /// - If `server_side_encryption` is not `aws:kms`, setting `server_side_encryption_aws_kms_key_id` /// is a noop. pub server_side_encryption_aws_kms_key_id: Option, /// server_side_encryption_customer_algorithm for this backend. /// /// Available values: `AES256`. pub server_side_encryption_customer_algorithm: Option, /// server_side_encryption_customer_key for this backend. /// /// Value: BASE64-encoded key that matches algorithm specified in /// `server_side_encryption_customer_algorithm`. pub server_side_encryption_customer_key: Option, /// Set server_side_encryption_customer_key_md5 for this backend. /// /// Value: MD5 digest of key specified in `server_side_encryption_customer_key`. pub server_side_encryption_customer_key_md5: Option, /// default storage_class for this backend. /// /// Available values: /// - `DEEP_ARCHIVE` /// - `GLACIER` /// - `GLACIER_IR` /// - `INTELLIGENT_TIERING` /// - `ONEZONE_IA` /// - `OUTPOSTS` /// - `REDUCED_REDUNDANCY` /// - `STANDARD` /// - `STANDARD_IA` /// /// S3 compatible services don't support all of them pub default_storage_class: Option, /// Enable virtual host style so that opendal will send API requests /// in virtual host style instead of path style. /// /// - By default, opendal will send API to `https://s3.us-east-1.amazonaws.com/bucket_name` /// - Enabled, opendal will send API to `https://bucket_name.s3.us-east-1.amazonaws.com` pub enable_virtual_host_style: bool, /// Set maximum batch operations of this backend. /// /// Some compatible services have a limit on the number of operations in a batch request. /// For example, R2 could return `Internal Error` while batch delete 1000 files. /// /// Please tune this value based on services' document. #[deprecated( since = "0.52.0", note = "Please use `delete_max_size` instead of `batch_max_operations`" )] pub batch_max_operations: Option, /// Set the maximum delete size of this backend. /// /// Some compatible services have a limit on the number of operations in a batch request. /// For example, R2 could return `Internal Error` while batch delete 1000 files. /// /// Please tune this value based on services' document. pub delete_max_size: Option, /// Disable stat with override so that opendal will not send stat request with override queries. /// /// For example, R2 doesn't support stat with `response_content_type` query. pub disable_stat_with_override: bool, /// Checksum Algorithm to use when sending checksums in HTTP headers. /// This is necessary when writing to AWS S3 Buckets with Object Lock enabled for example. /// /// Available options: /// - "crc32c" pub checksum_algorithm: Option, /// Disable write with if match so that opendal will not send write request with if match headers. /// /// For example, Ceph RADOS S3 doesn't support write with if match. pub disable_write_with_if_match: bool, } impl Debug for S3Config { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("S3Config"); d.field("root", &self.root) .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .field("region", &self.region); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/s3/core.rs000064400000000000000000001377371046102023000156110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt; use std::fmt::Debug; use std::fmt::Display; use std::fmt::Formatter; use std::fmt::Write; use std::sync::atomic; use std::sync::atomic::AtomicBool; use std::time::Duration; use base64::prelude::BASE64_STANDARD; use base64::Engine; use bytes::Bytes; use constants::X_AMZ_META_PREFIX; use http::header::CACHE_CONTROL; use http::header::CONTENT_DISPOSITION; use http::header::CONTENT_ENCODING; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::header::HOST; use http::header::IF_MATCH; use http::header::IF_NONE_MATCH; use http::header::{HeaderName, IF_MODIFIED_SINCE, IF_UNMODIFIED_SINCE}; use http::HeaderValue; use http::Request; use http::Response; use reqsign::AwsCredential; use reqsign::AwsCredentialLoad; use reqsign::AwsV4Signer; use serde::Deserialize; use serde::Serialize; use crate::raw::*; use crate::*; pub mod constants { pub const X_AMZ_COPY_SOURCE: &str = "x-amz-copy-source"; pub const X_AMZ_SERVER_SIDE_ENCRYPTION: &str = "x-amz-server-side-encryption"; pub const X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM: &str = "x-amz-server-side-encryption-customer-algorithm"; pub const X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY: &str = "x-amz-server-side-encryption-customer-key"; pub const X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5: &str = "x-amz-server-side-encryption-customer-key-md5"; pub const X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID: &str = "x-amz-server-side-encryption-aws-kms-key-id"; pub const X_AMZ_STORAGE_CLASS: &str = "x-amz-storage-class"; pub const X_AMZ_COPY_SOURCE_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM: &str = "x-amz-copy-source-server-side-encryption-customer-algorithm"; pub const X_AMZ_COPY_SOURCE_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY: &str = "x-amz-copy-source-server-side-encryption-customer-key"; pub const X_AMZ_COPY_SOURCE_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5: &str = "x-amz-copy-source-server-side-encryption-customer-key-md5"; pub const X_AMZ_META_PREFIX: &str = "x-amz-meta-"; pub const X_AMZ_VERSION_ID: &str = "x-amz-version-id"; pub const X_AMZ_OBJECT_SIZE: &str = "x-amz-object-size"; pub const RESPONSE_CONTENT_DISPOSITION: &str = "response-content-disposition"; pub const RESPONSE_CONTENT_TYPE: &str = "response-content-type"; pub const RESPONSE_CACHE_CONTROL: &str = "response-cache-control"; pub const S3_QUERY_VERSION_ID: &str = "versionId"; } pub struct S3Core { pub bucket: String, pub endpoint: String, pub root: String, pub server_side_encryption: Option, pub server_side_encryption_aws_kms_key_id: Option, pub server_side_encryption_customer_algorithm: Option, pub server_side_encryption_customer_key: Option, pub server_side_encryption_customer_key_md5: Option, pub default_storage_class: Option, pub allow_anonymous: bool, pub disable_stat_with_override: bool, pub enable_versioning: bool, pub signer: AwsV4Signer, pub loader: Box, pub credential_loaded: AtomicBool, pub client: HttpClient, pub delete_max_size: usize, pub checksum_algorithm: Option, pub disable_write_with_if_match: bool, } impl Debug for S3Core { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { f.debug_struct("S3Core") .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .field("root", &self.root) .finish_non_exhaustive() } } impl S3Core { /// If credential is not found, we will not sign the request. async fn load_credential(&self) -> Result> { let cred = self .loader .load_credential(GLOBAL_REQWEST_CLIENT.clone()) .await .map_err(new_request_credential_error)?; if let Some(cred) = cred { // Update credential_loaded to true if we have load credential successfully. self.credential_loaded .store(true, atomic::Ordering::Relaxed); return Ok(Some(cred)); } // If we have load credential before but failed to load this time, we should // return error instead. if self.credential_loaded.load(atomic::Ordering::Relaxed) { return Err(Error::new( ErrorKind::PermissionDenied, "credential was previously loaded successfully but has failed this time", ) .set_temporary()); } // Credential is empty and users allow anonymous access, we will not sign the request. if self.allow_anonymous { return Ok(None); } Err(Error::new( ErrorKind::PermissionDenied, "no valid credential found and anonymous access is not allowed", )) } pub async fn sign(&self, req: &mut Request) -> Result<()> { let cred = if let Some(cred) = self.load_credential().await? { cred } else { return Ok(()); }; self.signer .sign(req, &cred) .map_err(new_request_sign_error)?; // Always remove host header, let users' client to set it based on HTTP // version. // // As discussed in , // google server could send RST_STREAM of PROTOCOL_ERROR if our request // contains host header. req.headers_mut().remove(HOST); Ok(()) } pub async fn sign_query(&self, req: &mut Request, duration: Duration) -> Result<()> { let cred = if let Some(cred) = self.load_credential().await? { cred } else { return Ok(()); }; self.signer .sign_query(req, duration, &cred) .map_err(new_request_sign_error)?; // Always remove host header, let users' client to set it based on HTTP // version. // // As discussed in , // google server could send RST_STREAM of PROTOCOL_ERROR if our request // contains host header. req.headers_mut().remove(HOST); Ok(()) } #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } /// # Note /// /// header like X_AMZ_SERVER_SIDE_ENCRYPTION doesn't need to set while /// get or stat. pub fn insert_sse_headers( &self, mut req: http::request::Builder, is_write: bool, ) -> http::request::Builder { if is_write { if let Some(v) = &self.server_side_encryption { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_AMZ_SERVER_SIDE_ENCRYPTION), v, ) } if let Some(v) = &self.server_side_encryption_aws_kms_key_id { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_AMZ_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID), v, ) } } if let Some(v) = &self.server_side_encryption_customer_algorithm { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM), v, ) } if let Some(v) = &self.server_side_encryption_customer_key { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY), v, ) } if let Some(v) = &self.server_side_encryption_customer_key_md5 { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static(constants::X_AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5), v, ) } req } pub fn calculate_checksum(&self, body: &Buffer) -> Option { match self.checksum_algorithm { None => None, Some(ChecksumAlgorithm::Crc32c) => { let mut crc = 0u32; body.clone() .for_each(|b| crc = crc32c::crc32c_append(crc, &b)); Some(BASE64_STANDARD.encode(crc.to_be_bytes())) } } } pub fn insert_checksum_header( &self, mut req: http::request::Builder, checksum: &str, ) -> http::request::Builder { if let Some(checksum_algorithm) = self.checksum_algorithm.as_ref() { req = req.header(checksum_algorithm.to_header_name(), checksum); } req } pub fn insert_checksum_type_header( &self, mut req: http::request::Builder, ) -> http::request::Builder { if let Some(checksum_algorithm) = self.checksum_algorithm.as_ref() { req = req.header("x-amz-checksum-algorithm", checksum_algorithm.to_string()); } req } } impl S3Core { pub fn s3_head_object_request(&self, path: &str, args: OpStat) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); // Add query arguments to the URL based on response overrides let mut query_args = Vec::new(); if let Some(override_content_disposition) = args.override_content_disposition() { query_args.push(format!( "{}={}", constants::RESPONSE_CONTENT_DISPOSITION, percent_encode_path(override_content_disposition) )) } if let Some(override_content_type) = args.override_content_type() { query_args.push(format!( "{}={}", constants::RESPONSE_CONTENT_TYPE, percent_encode_path(override_content_type) )) } if let Some(override_cache_control) = args.override_cache_control() { query_args.push(format!( "{}={}", constants::RESPONSE_CACHE_CONTROL, percent_encode_path(override_cache_control) )) } if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::S3_QUERY_VERSION_ID, percent_decode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let mut req = Request::head(&url); req = self.insert_sse_headers(req, false); if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_modified_since) = args.if_modified_since() { req = req.header( IF_MODIFIED_SINCE, format_datetime_into_http_date(if_modified_since), ); } if let Some(if_unmodified_since) = args.if_unmodified_since() { req = req.header( IF_UNMODIFIED_SINCE, format_datetime_into_http_date(if_unmodified_since), ); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub fn s3_get_object_request( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path); // Construct headers to add to the request let mut url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); // Add query arguments to the URL based on response overrides let mut query_args = Vec::new(); if let Some(override_content_disposition) = args.override_content_disposition() { query_args.push(format!( "{}={}", constants::RESPONSE_CONTENT_DISPOSITION, percent_encode_path(override_content_disposition) )) } if let Some(override_content_type) = args.override_content_type() { query_args.push(format!( "{}={}", constants::RESPONSE_CONTENT_TYPE, percent_encode_path(override_content_type) )) } if let Some(override_cache_control) = args.override_cache_control() { query_args.push(format!( "{}={}", constants::RESPONSE_CACHE_CONTROL, percent_encode_path(override_cache_control) )) } if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::S3_QUERY_VERSION_ID, percent_decode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let mut req = Request::get(&url); if !range.is_full() { req = req.header(http::header::RANGE, range.to_header()); } if let Some(if_none_match) = args.if_none_match() { req = req.header(IF_NONE_MATCH, if_none_match); } if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if let Some(if_modified_since) = args.if_modified_since() { req = req.header( IF_MODIFIED_SINCE, format_datetime_into_http_date(if_modified_since), ); } if let Some(if_unmodified_since) = args.if_unmodified_since() { req = req.header( IF_UNMODIFIED_SINCE, format_datetime_into_http_date(if_unmodified_since), ); } // Set SSE headers. // TODO: how will this work with presign? req = self.insert_sse_headers(req, false); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; Ok(req) } pub async fn s3_get_object( &self, path: &str, range: BytesRange, args: &OpRead, ) -> Result> { let mut req = self.s3_get_object_request(path, range, args)?; self.sign(&mut req).await?; self.client.fetch(req).await } pub fn s3_put_object_request( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let mut req = Request::put(&url); if let Some(size) = size { req = req.header(CONTENT_LENGTH, size.to_string()) } if let Some(mime) = args.content_type() { req = req.header(CONTENT_TYPE, mime) } if let Some(pos) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, pos) } if let Some(encoding) = args.content_encoding() { req = req.header(CONTENT_ENCODING, encoding); } if let Some(cache_control) = args.cache_control() { req = req.header(CACHE_CONTROL, cache_control) } if let Some(if_match) = args.if_match() { req = req.header(IF_MATCH, if_match); } if args.if_not_exists() { req = req.header(IF_NONE_MATCH, "*"); } // Set storage class header if let Some(v) = &self.default_storage_class { req = req.header(HeaderName::from_static(constants::X_AMZ_STORAGE_CLASS), v); } // Set user metadata headers. if let Some(user_metadata) = args.user_metadata() { for (key, value) in user_metadata { req = req.header(format!("{X_AMZ_META_PREFIX}{key}"), value) } } // Set SSE headers. req = self.insert_sse_headers(req, true); // Calculate Checksum. if let Some(checksum) = self.calculate_checksum(&body) { // Set Checksum header. req = self.insert_checksum_header(req, &checksum); } // Set body let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn s3_head_object(&self, path: &str, args: OpStat) -> Result> { let mut req = self.s3_head_object_request(path, args)?; self.sign(&mut req).await?; self.send(req).await } pub async fn s3_delete_object(&self, path: &str, args: &OpDelete) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!("{}/{}", self.endpoint, percent_encode_path(&p)); let mut query_args = Vec::new(); if let Some(version) = args.version() { query_args.push(format!( "{}={}", constants::S3_QUERY_VERSION_ID, percent_encode_path(version) )) } if !query_args.is_empty() { url.push_str(&format!("?{}", query_args.join("&"))); } let mut req = Request::delete(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn s3_copy_object(&self, from: &str, to: &str) -> Result> { let from = build_abs_path(&self.root, from); let to = build_abs_path(&self.root, to); let source = format!("{}/{}", self.bucket, percent_encode_path(&from)); let target = format!("{}/{}", self.endpoint, percent_encode_path(&to)); let mut req = Request::put(&target); // Set SSE headers. req = self.insert_sse_headers(req, true); if let Some(v) = &self.server_side_encryption_customer_algorithm { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static( constants::X_AMZ_COPY_SOURCE_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, ), v, ) } if let Some(v) = &self.server_side_encryption_customer_key { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static( constants::X_AMZ_COPY_SOURCE_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY, ), v, ) } if let Some(v) = &self.server_side_encryption_customer_key_md5 { let mut v = v.clone(); v.set_sensitive(true); req = req.header( HeaderName::from_static( constants::X_AMZ_COPY_SOURCE_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, ), v, ) } let mut req = req .header(constants::X_AMZ_COPY_SOURCE, &source) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn s3_list_objects( &self, path: &str, continuation_token: &str, delimiter: &str, limit: Option, start_after: Option, ) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!("{}?list-type=2", self.endpoint); if !p.is_empty() { write!(url, "&prefix={}", percent_encode_path(&p)) .expect("write into string must succeed"); } if !delimiter.is_empty() { write!(url, "&delimiter={delimiter}").expect("write into string must succeed"); } if let Some(limit) = limit { write!(url, "&max-keys={limit}").expect("write into string must succeed"); } if let Some(start_after) = start_after { write!(url, "&start-after={}", percent_encode_path(&start_after)) .expect("write into string must succeed"); } if !continuation_token.is_empty() { // AWS S3 could return continuation-token that contains `=` // which could lead `reqsign` parse query wrongly. // URL encode continuation-token before starting signing so that // our signer will not be confused. write!( url, "&continuation-token={}", percent_encode_path(continuation_token) ) .expect("write into string must succeed"); } let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn s3_initiate_multipart_upload( &self, path: &str, args: &OpWrite, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!("{}/{}?uploads", self.endpoint, percent_encode_path(&p)); let mut req = Request::post(&url); if let Some(mime) = args.content_type() { req = req.header(CONTENT_TYPE, mime) } if let Some(content_disposition) = args.content_disposition() { req = req.header(CONTENT_DISPOSITION, content_disposition) } if let Some(cache_control) = args.cache_control() { req = req.header(CACHE_CONTROL, cache_control) } // Set storage class header if let Some(v) = &self.default_storage_class { req = req.header(HeaderName::from_static(constants::X_AMZ_STORAGE_CLASS), v); } // Set user metadata headers. if let Some(user_metadata) = args.user_metadata() { for (key, value) in user_metadata { req = req.header(format!("{X_AMZ_META_PREFIX}{key}"), value) } } // Set SSE headers. let req = self.insert_sse_headers(req, true); // Set SSE headers. let req = self.insert_checksum_type_header(req); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub fn s3_upload_part_request( &self, path: &str, upload_id: &str, part_number: usize, size: u64, body: Buffer, checksum: Option, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?partNumber={}&uploadId={}", self.endpoint, percent_encode_path(&p), part_number, percent_encode_path(upload_id) ); let mut req = Request::put(&url); req = req.header(CONTENT_LENGTH, size); // Set SSE headers. req = self.insert_sse_headers(req, true); if let Some(checksum) = checksum { // Set Checksum header. req = self.insert_checksum_header(req, &checksum); } // Set body let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn s3_complete_multipart_upload( &self, path: &str, upload_id: &str, parts: Vec, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?uploadId={}", self.endpoint, percent_encode_path(&p), percent_encode_path(upload_id) ); let req = Request::post(&url); // Set SSE headers. let req = self.insert_sse_headers(req, true); let content = quick_xml::se::to_string(&CompleteMultipartUploadRequest { part: parts }) .map_err(new_xml_deserialize_error)?; // Make sure content length has been set to avoid post with chunked encoding. let req = req.header(CONTENT_LENGTH, content.len()); // Set content-type to `application/xml` to avoid mixed with form post. let req = req.header(CONTENT_TYPE, "application/xml"); let mut req = req .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } /// Abort an on-going multipart upload. pub async fn s3_abort_multipart_upload( &self, path: &str, upload_id: &str, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}?uploadId={}", self.endpoint, percent_encode_path(&p), percent_encode_path(upload_id) ); let mut req = Request::delete(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn s3_delete_objects( &self, paths: Vec<(String, OpDelete)>, ) -> Result> { let url = format!("{}/?delete", self.endpoint); let req = Request::post(&url); let content = quick_xml::se::to_string(&DeleteObjectsRequest { object: paths .into_iter() .map(|(path, op)| DeleteObjectsRequestObject { key: build_abs_path(&self.root, &path), version_id: op.version().map(|v| v.to_owned()), }) .collect(), }) .map_err(new_xml_deserialize_error)?; // Make sure content length has been set to avoid post with chunked encoding. let req = req.header(CONTENT_LENGTH, content.len()); // Set content-type to `application/xml` to avoid mixed with form post. let req = req.header(CONTENT_TYPE, "application/xml"); // Set content-md5 as required by API. let req = req.header("CONTENT-MD5", format_content_md5(content.as_bytes())); let mut req = req .body(Buffer::from(Bytes::from(content))) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } pub async fn s3_list_object_versions( &self, prefix: &str, delimiter: &str, limit: Option, key_marker: &str, version_id_marker: &str, ) -> Result> { let p = build_abs_path(&self.root, prefix); let mut url = format!("{}?versions", self.endpoint); if !p.is_empty() { write!(url, "&prefix={}", percent_encode_path(p.as_str())) .expect("write into string must succeed"); } if !delimiter.is_empty() { write!(url, "&delimiter={}", delimiter).expect("write into string must succeed"); } if let Some(limit) = limit { write!(url, "&max-keys={}", limit).expect("write into string must succeed"); } if !key_marker.is_empty() { write!(url, "&key-marker={}", percent_encode_path(key_marker)) .expect("write into string must succeed"); } if !version_id_marker.is_empty() { write!( url, "&version-id-marker={}", percent_encode_path(version_id_marker) ) .expect("write into string must succeed"); } let mut req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req).await?; self.send(req).await } } /// Result of CreateMultipartUpload #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct InitiateMultipartUploadResult { pub upload_id: String, } /// Request of CompleteMultipartUploadRequest #[derive(Default, Debug, Serialize)] #[serde(default, rename = "CompleteMultipartUpload", rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequest { pub part: Vec, } #[derive(Clone, Default, Debug, Serialize)] #[serde(default, rename_all = "PascalCase")] pub struct CompleteMultipartUploadRequestPart { #[serde(rename = "PartNumber")] pub part_number: usize, /// # TODO /// /// quick-xml will do escape on `"` which leads to our serialized output is /// not the same as aws s3's example. /// /// Ideally, we could use `serialize_with` to address this (buf failed) /// /// ```ignore /// #[derive(Default, Debug, Serialize)] /// #[serde(default, rename_all = "PascalCase")] /// struct CompleteMultipartUploadRequestPart { /// #[serde(rename = "PartNumber")] /// part_number: usize, /// #[serde(rename = "ETag", serialize_with = "partial_escape")] /// etag: String, /// } /// /// fn partial_escape(s: &str, ser: S) -> Result /// where /// S: serde::Serializer, /// { /// ser.serialize_str(&String::from_utf8_lossy( /// &quick_xml::escape::partial_escape(s.as_bytes()), /// )) /// } /// ``` /// /// ref: #[serde(rename = "ETag")] pub etag: String, #[serde(rename = "ChecksumCRC32C", skip_serializing_if = "Option::is_none")] pub checksum_crc32c: Option, } /// Output of `CompleteMultipartUpload` operation #[derive(Debug, Default, Deserialize)] #[serde[default, rename_all = "PascalCase"]] pub struct CompleteMultipartUploadResult { pub bucket: String, pub key: String, pub location: String, #[serde(rename = "ETag")] pub etag: String, pub code: String, pub message: String, pub request_id: String, } /// Request of DeleteObjects. #[derive(Default, Debug, Serialize)] #[serde(default, rename = "Delete", rename_all = "PascalCase")] pub struct DeleteObjectsRequest { pub object: Vec, } #[derive(Default, Debug, Serialize)] #[serde(rename_all = "PascalCase")] pub struct DeleteObjectsRequestObject { pub key: String, #[serde(skip_serializing_if = "Option::is_none")] pub version_id: Option, } /// Result of DeleteObjects. #[derive(Default, Debug, Deserialize)] #[serde(default, rename = "DeleteResult", rename_all = "PascalCase")] pub struct DeleteObjectsResult { pub deleted: Vec, pub error: Vec, } #[derive(Default, Debug, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct DeleteObjectsResultDeleted { pub key: String, } #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct DeleteObjectsResultError { pub code: String, pub key: String, pub message: String, } /// Output of ListBucket/ListObjects. /// /// ## Note /// /// Use `Option` in `is_truncated` and `next_continuation_token` to make /// the behavior more clear so that we can be compatible to more s3 services. /// /// And enable `serde(default)` so that we can keep going even when some field /// is not exist. #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectsOutput { pub is_truncated: Option, pub next_continuation_token: Option, pub common_prefixes: Vec, pub contents: Vec, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct ListObjectsOutputContent { pub key: String, pub size: u64, pub last_modified: String, #[serde(rename = "ETag")] pub etag: Option, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct OutputCommonPrefix { pub prefix: String, } /// Output of ListObjectVersions #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] pub struct ListObjectVersionsOutput { pub is_truncated: Option, pub next_key_marker: Option, pub next_version_id_marker: Option, pub common_prefixes: Vec, pub version: Vec, pub delete_marker: Vec, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct ListObjectVersionsOutputVersion { pub key: String, pub version_id: String, pub is_latest: bool, pub size: u64, pub last_modified: String, #[serde(rename = "ETag")] pub etag: Option, } #[derive(Default, Debug, Eq, PartialEq, Deserialize)] #[serde(rename_all = "PascalCase")] pub struct ListObjectVersionsOutputDeleteMarker { pub key: String, pub version_id: String, pub is_latest: bool, pub last_modified: String, } pub enum ChecksumAlgorithm { Crc32c, } impl ChecksumAlgorithm { pub fn to_header_name(&self) -> HeaderName { match self { Self::Crc32c => HeaderName::from_static("x-amz-checksum-crc32c"), } } } impl Display for ChecksumAlgorithm { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { write!( f, "{}", match self { Self::Crc32c => "CRC32C", } ) } } #[cfg(test)] mod tests { use bytes::Buf; use bytes::Bytes; use super::*; /// This example is from https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html#API_CreateMultipartUpload_Examples #[test] fn test_deserialize_initiate_multipart_upload_result() { let bs = Bytes::from( r#" example-bucket example-object VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZA "#, ); let out: InitiateMultipartUploadResult = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert_eq!( out.upload_id, "VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZA" ) } /// This example is from https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_Examples #[test] fn test_serialize_complete_multipart_upload_request() { let req = CompleteMultipartUploadRequest { part: vec![ CompleteMultipartUploadRequestPart { part_number: 1, etag: "\"a54357aff0632cce46d942af68356b38\"".to_string(), ..Default::default() }, CompleteMultipartUploadRequestPart { part_number: 2, etag: "\"0c78aef83f66abc1fa1e8477f296d394\"".to_string(), ..Default::default() }, CompleteMultipartUploadRequestPart { part_number: 3, etag: "\"acbd18db4cc2f85cedef654fccc4a4d8\"".to_string(), ..Default::default() }, ], }; let actual = quick_xml::se::to_string(&req).expect("must succeed"); pretty_assertions::assert_eq!( actual, r#" 1 "a54357aff0632cce46d942af68356b38" 2 "0c78aef83f66abc1fa1e8477f296d394" 3 "acbd18db4cc2f85cedef654fccc4a4d8" "# // Cleanup space and new line .replace([' ', '\n'], "") ) } /// this example is from: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html #[test] fn test_deserialize_complete_multipart_upload_result() { let bs = Bytes::from( r#" http://Example-Bucket.s3.region.amazonaws.com/Example-Object Example-Bucket Example-Object "3858f62230ac3c915f300c664312c11f-9" "#, ); let out: CompleteMultipartUploadResult = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert_eq!(out.bucket, "Example-Bucket"); assert_eq!(out.key, "Example-Object"); assert_eq!( out.location, "http://Example-Bucket.s3.region.amazonaws.com/Example-Object" ); assert_eq!(out.etag, "\"3858f62230ac3c915f300c664312c11f-9\""); } #[test] fn test_deserialize_complete_multipart_upload_result_when_return_error() { let bs = Bytes::from( r#" InternalError We encountered an internal error. Please try again. 656c76696e6727732072657175657374 Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg== "#, ); let out: CompleteMultipartUploadResult = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert_eq!(out.code, "InternalError"); assert_eq!( out.message, "We encountered an internal error. Please try again." ); assert_eq!(out.request_id, "656c76696e6727732072657175657374"); } /// This example is from https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html#API_DeleteObjects_Examples #[test] fn test_serialize_delete_objects_request() { let req = DeleteObjectsRequest { object: vec![ DeleteObjectsRequestObject { key: "sample1.txt".to_string(), version_id: None, }, DeleteObjectsRequestObject { key: "sample2.txt".to_string(), version_id: Some("11111".to_owned()), }, ], }; let actual = quick_xml::se::to_string(&req).expect("must succeed"); pretty_assertions::assert_eq!( actual, r#" sample1.txt sample2.txt 11111 "# // Cleanup space and new line .replace([' ', '\n'], "") ) } /// This example is from https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html#API_DeleteObjects_Examples #[test] fn test_deserialize_delete_objects_result() { let bs = Bytes::from( r#" sample1.txt sample2.txt AccessDenied Access Denied "#, ); let out: DeleteObjectsResult = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert_eq!(out.deleted.len(), 1); assert_eq!(out.deleted[0].key, "sample1.txt"); assert_eq!(out.error.len(), 1); assert_eq!(out.error[0].key, "sample2.txt"); assert_eq!(out.error[0].code, "AccessDenied"); assert_eq!(out.error[0].message, "Access Denied"); } #[test] fn test_parse_list_output() { let bs = bytes::Bytes::from( r#" example-bucket photos/2006/ 3 1000 / false photos/2006 2016-04-30T23:51:29.000Z "d41d8cd98f00b204e9800998ecf8427e" 56 STANDARD photos/2007 2016-04-30T23:51:29.000Z "d41d8cd98f00b204e9800998ecf8427e" 100 STANDARD photos/2008 2016-05-30T23:51:29.000Z 42 photos/2006/February/ photos/2006/January/ "#, ); let out: ListObjectsOutput = quick_xml::de::from_reader(bs.reader()).expect("must success"); assert!(!out.is_truncated.unwrap()); assert!(out.next_continuation_token.is_none()); assert_eq!( out.common_prefixes .iter() .map(|v| v.prefix.clone()) .collect::>(), vec!["photos/2006/February/", "photos/2006/January/"] ); assert_eq!( out.contents, vec![ ListObjectsOutputContent { key: "photos/2006".to_string(), size: 56, etag: Some("\"d41d8cd98f00b204e9800998ecf8427e\"".to_string()), last_modified: "2016-04-30T23:51:29.000Z".to_string(), }, ListObjectsOutputContent { key: "photos/2007".to_string(), size: 100, last_modified: "2016-04-30T23:51:29.000Z".to_string(), etag: Some("\"d41d8cd98f00b204e9800998ecf8427e\"".to_string()), }, ListObjectsOutputContent { key: "photos/2008".to_string(), size: 42, last_modified: "2016-05-30T23:51:29.000Z".to_string(), etag: None, }, ] ) } #[test] fn test_parse_list_object_versions() { let bs = bytes::Bytes::from( r#" mtp-versioning-fresh key3 null key3 d-d309mfjFrUmoQ0DBsVqmcMV15OI. 3 true key3 8XECiENpj8pydEDJdd-_VRrvaGKAHOaGMNW7tg6UViI. true 2009-12-09T00:18:23.000Z "396fefef536d5ce46c7537ecf978a360" 217 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a STANDARD key3 d-d309mfjFri40QYukDozqBt3UmoQ0DBsVqmcMV15OI. false 2009-12-09T00:18:08.000Z "396fefef536d5ce46c7537ecf978a360" 217 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a STANDARD photos/ videos/ my-third-image.jpg 03jpff543dhffds434rfdsFDN943fdsFkdmqnh892 true 2009-10-15T17:50:30.000Z 75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a mtd@amazon.com "#, ); let output: ListObjectVersionsOutput = quick_xml::de::from_reader(bs.reader()).expect("must succeed"); assert!(output.is_truncated.unwrap()); assert_eq!(output.next_key_marker, Some("key3".to_owned())); assert_eq!( output.next_version_id_marker, Some("d-d309mfjFrUmoQ0DBsVqmcMV15OI.".to_owned()) ); assert_eq!( output.common_prefixes, vec![ OutputCommonPrefix { prefix: "photos/".to_owned() }, OutputCommonPrefix { prefix: "videos/".to_owned() } ] ); assert_eq!( output.version, vec![ ListObjectVersionsOutputVersion { key: "key3".to_owned(), version_id: "8XECiENpj8pydEDJdd-_VRrvaGKAHOaGMNW7tg6UViI.".to_owned(), is_latest: true, size: 217, last_modified: "2009-12-09T00:18:23.000Z".to_owned(), etag: Some("\"396fefef536d5ce46c7537ecf978a360\"".to_owned()), }, ListObjectVersionsOutputVersion { key: "key3".to_owned(), version_id: "d-d309mfjFri40QYukDozqBt3UmoQ0DBsVqmcMV15OI.".to_owned(), is_latest: false, size: 217, last_modified: "2009-12-09T00:18:08.000Z".to_owned(), etag: Some("\"396fefef536d5ce46c7537ecf978a360\"".to_owned()), } ] ); assert_eq!( output.delete_marker, vec![ListObjectVersionsOutputDeleteMarker { key: "my-third-image.jpg".to_owned(), version_id: "03jpff543dhffds434rfdsFDN943fdsFkdmqnh892".to_owned(), is_latest: true, last_modified: "2009-10-15T17:50:30.000Z".to_owned(), },] ); } } opendal-0.52.0/src/services/s3/delete.rs000064400000000000000000000071371046102023000161110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::{parse_error, parse_s3_error_code}; use crate::raw::oio::BatchDeleteResult; use crate::raw::*; use crate::*; use bytes::Buf; use http::StatusCode; use std::sync::Arc; pub struct S3Deleter { core: Arc, } impl S3Deleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::BatchDelete for S3Deleter { async fn delete_once(&self, path: String, args: OpDelete) -> Result<()> { // This would delete the bucket, do not perform if self.core.root == "/" && path == "/" { return Ok(()); } let resp = self.core.s3_delete_object(&path, &args).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT => Ok(()), // Allow 404 when deleting a non-existing object // This is not a standard behavior, only some s3 alike service like GCS XML API do this. // ref: StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } async fn delete_batch(&self, batch: Vec<(String, OpDelete)>) -> Result { let resp = self.core.s3_delete_objects(batch).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let mut result: DeleteObjectsResult = quick_xml::de::from_reader(bs.reader()).map_err(new_xml_deserialize_error)?; // If no object is deleted, return directly. if result.deleted.is_empty() { let err = result.error.remove(0); return Err(parse_delete_objects_result_error(err)); } let mut batched_result = BatchDeleteResult { succeeded: Vec::with_capacity(result.deleted.len()), failed: Vec::with_capacity(result.error.len()), }; for i in result.deleted { let path = build_rel_path(&self.core.root, &i.key); // TODO: fix https://github.com/apache/opendal/issues/5329 batched_result.succeeded.push((path, OpDelete::new())); } for i in result.error { let path = build_rel_path(&self.core.root, &i.key); batched_result.failed.push(( path, OpDelete::new(), parse_delete_objects_result_error(i), )); } Ok(batched_result) } } fn parse_delete_objects_result_error(err: DeleteObjectsResultError) -> Error { let (kind, retryable) = parse_s3_error_code(err.code.as_str()).unwrap_or((ErrorKind::Unexpected, false)); let mut err: Error = Error::new(kind, format!("{err:?}")); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/s3/docs.md000064400000000000000000000156101046102023000155460ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [x] presign - [ ] blocking ## Configuration - `root`: Set the work dir for backend. - `bucket`: Set the container name for backend. - `endpoint`: Set the endpoint for backend. - `region`: Set the region for backend. - `access_key_id`: Set the access_key_id for backend. - `secret_access_key`: Set the secret_access_key for backend. - `session_token`: Set the session_token for backend. - `default_storage_class`: Set the default storage_class for backend. - `server_side_encryption`: Set the server_side_encryption for backend. - `server_side_encryption_aws_kms_key_id`: Set the server_side_encryption_aws_kms_key_id for backend. - `server_side_encryption_customer_algorithm`: Set the server_side_encryption_customer_algorithm for backend. - `server_side_encryption_customer_key`: Set the server_side_encryption_customer_key for backend. - `server_side_encryption_customer_key_md5`: Set the server_side_encryption_customer_key_md5 for backend. - `disable_config_load`: Disable aws config load from env. - `enable_virtual_host_style`: Enable virtual host style. - `disable_write_with_if_match`: Disable write with if match. Refer to [`S3Builder`]'s public API docs for more information. ## Temporary security credentials OpenDAL now provides support for S3 temporary security credentials in IAM. The way to take advantage of this feature is to build your S3 backend with `Builder::session_token`. But OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time. ## Server Side Encryption OpenDAL provides full support of S3 Server Side Encryption(SSE) features. The easiest way to configure them is to use helper functions like - SSE-KMS: `server_side_encryption_with_aws_managed_kms_key` - SSE-KMS: `server_side_encryption_with_customer_managed_kms_key` - SSE-S3: `server_side_encryption_with_s3_key` - SSE-C: `server_side_encryption_with_customer_key` If those functions don't fulfill need, low-level options are also provided: - Use service managed kms key - `server_side_encryption="aws:kms"` - Use customer provided kms key - `server_side_encryption="aws:kms"` - `server_side_encryption_aws_kms_key_id="your-kms-key"` - Use S3 managed key - `server_side_encryption="AES256"` - Use customer key - `server_side_encryption_customer_algorithm="AES256"` - `server_side_encryption_customer_key="base64-of-your-aes256-key"` - `server_side_encryption_customer_key_md5="base64-of-your-aes256-key-md5"` After SSE have been configured, all requests send by this backed will attach those headers. Reference: [Protecting data using server-side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html) ## Example ## Via Builder ### Basic Setup ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::S3; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create s3 backend builder. let mut builder = S3::default() // Set the root for s3, all operations will happen under this root. // // NOTE: the root must be absolute path. .root("/path/to/dir") // Set the bucket name. This is required. .bucket("test") // Set the region. This is required for some services, if you don't care about it, for example Minio service, just set it to "auto", it will be ignored. .region("us-east-1") // Set the endpoint. // // For examples: // - "https://s3.amazonaws.com" // - "http://127.0.0.1:9000" // - "https://oss-ap-northeast-1.aliyuncs.com" // - "https://cos.ap-seoul.myqcloud.com" // // Default to "https://s3.amazonaws.com" .endpoint("https://s3.amazonaws.com") // Set the access_key_id and secret_access_key. // // OpenDAL will try load credential from the env. // If credential not set and no valid credential in env, OpenDAL will // send request without signing like anonymous user. .access_key_id("access_key_id") .secret_access_key("secret_access_key"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` ### S3 with SSE-C ```rust,no_run use anyhow::Result; use log::info; use opendal::services::S3; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = S3::default() .root("/path/to/dir") .bucket("test") .region("us-east-1") .endpoint("https://s3.amazonaws.com") .access_key_id("access_key_id") .secret_access_key("secret_access_key") // Enable SSE-C .server_side_encryption_with_customer_key("AES256", "customer_key".as_bytes()); let op = Operator::new(builder)?.finish(); info!("operator: {:?}", op); // Writing your testing code here. Ok(()) } ``` ### S3 with SSE-KMS and aws managed kms key ```rust,no_run use anyhow::Result; use log::info; use opendal::services::S3; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = S3::default() // Setup builders .root("/path/to/dir") .bucket("test") .region("us-east-1") .endpoint("https://s3.amazonaws.com") .access_key_id("access_key_id") .secret_access_key("secret_access_key") // Enable SSE-KMS with aws managed kms key .server_side_encryption_with_aws_managed_kms_key(); let op = Operator::new(builder)?.finish(); info!("operator: {:?}", op); // Writing your testing code here. Ok(()) } ``` ### S3 with SSE-KMS and customer managed kms key ```rust,no_run use anyhow::Result; use log::info; use opendal::services::S3; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = S3::default() // Setup builders .root("/path/to/dir") .bucket("test") .region("us-east-1") .endpoint("https://s3.amazonaws.com") .access_key_id("access_key_id") .secret_access_key("secret_access_key") // Enable SSE-KMS with customer managed kms key .server_side_encryption_with_customer_managed_kms_key("aws_kms_key_id"); let op = Operator::new(builder)?.finish(); info!("operator: {:?}", op); // Writing your testing code here. Ok(()) } ``` ### S3 with SSE-S3 ```rust,no_run use anyhow::Result; use log::info; use opendal::services::S3; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = S3::default() // Setup builders .root("/path/to/dir") .bucket("test") .region("us-east-1") .endpoint("https://s3.amazonaws.com") .access_key_id("access_key_id") .secret_access_key("secret_access_key") // Enable SSE-S3 .server_side_encryption_with_s3_key(); let op = Operator::new(builder)?.finish(); info!("operator: {:?}", op); // Writing your testing code here. Ok(()) } ``` opendal-0.52.0/src/services/s3/error.rs000064400000000000000000000136241046102023000157760ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::response::Parts; use http::Response; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// S3Error is the error returned by s3 service. #[derive(Default, Debug, Deserialize, PartialEq, Eq)] #[serde(default, rename_all = "PascalCase")] pub(crate) struct S3Error { pub code: String, pub message: String, pub resource: String, pub request_id: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (mut kind, mut retryable) = match parts.status.as_u16() { 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), 304 | 412 => (ErrorKind::ConditionNotMatch, false), // Service like R2 could return 499 error with a message like: // Client Disconnect, we should retry it. 499 => (ErrorKind::Unexpected, true), 500 | 502 | 503 | 504 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let body_content = bs.chunk(); let (message, s3_err) = de::from_reader::<_, S3Error>(body_content.reader()) .map(|s3_err| (format!("{s3_err:?}"), Some(s3_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); if let Some(s3_err) = s3_err { (kind, retryable) = parse_s3_error_code(s3_err.code.as_str()).unwrap_or((kind, retryable)); } let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } /// Util function to build [`Error`] from a [`S3Error`] object. pub(crate) fn from_s3_error(s3_error: S3Error, parts: Parts) -> Error { let (kind, retryable) = parse_s3_error_code(s3_error.code.as_str()).unwrap_or((ErrorKind::Unexpected, false)); let mut err = Error::new(kind, format!("{s3_error:?}")); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } /// Returns the `Error kind` of this code and whether the error is retryable. /// All possible error code: pub fn parse_s3_error_code(code: &str) -> Option<(ErrorKind, bool)> { match code { // > The specified bucket does not exist. // // Although the status code is 404, NoSuchBucket is // a config invalid error, and it's not retryable from OpenDAL. "NoSuchBucket" => Some((ErrorKind::ConfigInvalid, false)), // > Your socket connection to the server was not read from // > or written to within the timeout period." // // It's Ok for us to retry it again. "RequestTimeout" => Some((ErrorKind::Unexpected, true)), // > An internal error occurred. Try again. "InternalError" => Some((ErrorKind::Unexpected, true)), // > A conflicting conditional operation is currently in progress // > against this resource. Try again. "OperationAborted" => Some((ErrorKind::Unexpected, true)), // > Please reduce your request rate. // // It's Ok to retry since later on the request rate may get reduced. "SlowDown" => Some((ErrorKind::RateLimited, true)), // > Service is unable to handle request. // // ServiceUnavailable is considered a retryable error because it typically // indicates a temporary issue with the service or server, such as high load, // maintenance, or an internal problem. "ServiceUnavailable" => Some((ErrorKind::Unexpected, true)), _ => None, } } #[cfg(test)] mod tests { use super::*; /// Error response example is from https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html #[test] fn test_parse_error() { let bs = bytes::Bytes::from( r#" NoSuchKey The resource you requested does not exist /mybucket/myfoto.jpg 4442587FB7D0A2F9 "#, ); let out: S3Error = de::from_reader(bs.reader()).expect("must success"); println!("{out:?}"); assert_eq!(out.code, "NoSuchKey"); assert_eq!(out.message, "The resource you requested does not exist"); assert_eq!(out.resource, "/mybucket/myfoto.jpg"); assert_eq!(out.request_id, "4442587FB7D0A2F9"); } #[test] fn test_parse_error_from_unrelated_input() { let bs = bytes::Bytes::from( r#" http://Example-Bucket.s3.ap-southeast-1.amazonaws.com/Example-Object Example-Bucket Example-Object "3858f62230ac3c915f300c664312c11f-9" "#, ); let out: S3Error = de::from_reader(bs.reader()).expect("must success"); assert_eq!(out, S3Error::default()); } } opendal-0.52.0/src/services/s3/lister.rs000064400000000000000000000223731046102023000161500ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use super::core::S3Core; use super::core::{ListObjectVersionsOutput, ListObjectsOutput}; use super::error::parse_error; use crate::raw::oio::PageContext; use crate::raw::*; use crate::EntryMode; use crate::Error; use crate::Metadata; use crate::Result; use bytes::Buf; use quick_xml::de; pub type S3Listers = TwoWays, oio::PageLister>; pub struct S3Lister { core: Arc, path: String, args: OpList, delimiter: &'static str, abs_start_after: Option, } impl S3Lister { pub fn new(core: Arc, path: &str, args: OpList) -> Self { let delimiter = if args.recursive() { "" } else { "/" }; let abs_start_after = args .start_after() .map(|start_after| build_abs_path(&core.root, start_after)); Self { core, path: path.to_string(), args, delimiter, abs_start_after, } } } impl oio::PageList for S3Lister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .s3_list_objects( &self.path, &ctx.token, self.delimiter, self.args.limit(), // start after should only be set for the first page. if ctx.token.is_empty() { self.abs_start_after.clone() } else { None }, ) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let output: ListObjectsOutput = de::from_reader(bs.reader()) .map_err(new_xml_deserialize_error) // Allow S3 list to retry on XML deserialization errors. // // This is because the S3 list API may return incomplete XML data under high load. // We are confident that our XML decoding logic is correct. When this error occurs, // we allow retries to obtain the correct data. .map_err(Error::set_temporary)?; // Try our best to check whether this list is done. // // - Check `is_truncated` // - Check `next_continuation_token` // - Check the length of `common_prefixes` and `contents` (very rare case) ctx.done = if let Some(is_truncated) = output.is_truncated { !is_truncated } else if let Some(next_continuation_token) = output.next_continuation_token.as_ref() { next_continuation_token.is_empty() } else { output.common_prefixes.is_empty() && output.contents.is_empty() }; ctx.token = output.next_continuation_token.clone().unwrap_or_default(); for prefix in output.common_prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix.prefix), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de); } for object in output.contents { let mut path = build_rel_path(&self.core.root, &object.key); if path.is_empty() { path = "/".to_string(); } let mut meta = Metadata::new(EntryMode::from_path(&path)); meta.set_is_current(true); if let Some(etag) = &object.etag { meta.set_etag(etag); meta.set_content_md5(etag.trim_matches('"')); } meta.set_content_length(object.size); // object.last_modified provides more precise time that contains // nanosecond, let's trim them. meta.set_last_modified(parse_datetime_from_rfc3339(object.last_modified.as_str())?); let de = oio::Entry::with(path, meta); ctx.entries.push_back(de); } Ok(()) } } /// refer: https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectVersions.html pub struct S3ObjectVersionsLister { core: Arc, prefix: String, args: OpList, delimiter: &'static str, abs_start_after: Option, } impl S3ObjectVersionsLister { pub fn new(core: Arc, path: &str, args: OpList) -> Self { let delimiter = if args.recursive() { "" } else { "/" }; let abs_start_after = args .start_after() .map(|start_after| build_abs_path(&core.root, start_after)); Self { core, prefix: path.to_string(), args, delimiter, abs_start_after, } } } impl oio::PageList for S3ObjectVersionsLister { async fn next_page(&self, ctx: &mut PageContext) -> Result<()> { let markers = ctx.token.rsplit_once(" "); let (key_marker, version_id_marker) = if let Some(data) = markers { data } else if let Some(start_after) = &self.abs_start_after { (start_after.as_str(), "") } else { ("", "") }; let resp = self .core .s3_list_object_versions( &self.prefix, self.delimiter, self.args.limit(), key_marker, version_id_marker, ) .await?; if resp.status() != http::StatusCode::OK { return Err(parse_error(resp)); } let body = resp.into_body(); let output: ListObjectVersionsOutput = de::from_reader(body.reader()) .map_err(new_xml_deserialize_error) // Allow S3 list to retry on XML deserialization errors. // // This is because the S3 list API may return incomplete XML data under high load. // We are confident that our XML decoding logic is correct. When this error occurs, // we allow retries to obtain the correct data. .map_err(Error::set_temporary)?; ctx.done = if let Some(is_truncated) = output.is_truncated { !is_truncated } else { false }; ctx.token = format!( "{} {}", output.next_key_marker.unwrap_or_default(), output.next_version_id_marker.unwrap_or_default() ); for prefix in output.common_prefixes { let de = oio::Entry::new( &build_rel_path(&self.core.root, &prefix.prefix), Metadata::new(EntryMode::DIR), ); ctx.entries.push_back(de); } for version_object in output.version { // `list` must be additive, so we need to include the latest version object // even if `versions` is not enabled. // // Here we skip all non-latest version objects if `versions` is not enabled. if !(self.args.versions() || version_object.is_latest) { continue; } let mut path = build_rel_path(&self.core.root, &version_object.key); if path.is_empty() { path = "/".to_owned(); } let mut meta = Metadata::new(EntryMode::from_path(&path)); meta.set_version(&version_object.version_id); meta.set_is_current(version_object.is_latest); meta.set_content_length(version_object.size); meta.set_last_modified(parse_datetime_from_rfc3339( version_object.last_modified.as_str(), )?); if let Some(etag) = version_object.etag { meta.set_etag(&etag); meta.set_content_md5(etag.trim_matches('"')); } let entry = oio::Entry::new(&path, meta); ctx.entries.push_back(entry); } if self.args.deleted() { for delete_marker in output.delete_marker { let mut path = build_rel_path(&self.core.root, &delete_marker.key); if path.is_empty() { path = "/".to_owned(); } let mut meta = Metadata::new(EntryMode::FILE); meta.set_version(&delete_marker.version_id); meta.set_is_deleted(true); meta.set_is_current(delete_marker.is_latest); meta.set_last_modified(parse_datetime_from_rfc3339( delete_marker.last_modified.as_str(), )?); let entry = oio::Entry::new(&path, meta); ctx.entries.push_back(entry); } } Ok(()) } } opendal-0.52.0/src/services/s3/mod.rs000064400000000000000000000022221046102023000154140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-s3")] mod core; #[cfg(feature = "services-s3")] mod delete; #[cfg(feature = "services-s3")] mod error; #[cfg(feature = "services-s3")] mod lister; #[cfg(feature = "services-s3")] mod writer; #[cfg(feature = "services-s3")] mod backend; #[cfg(feature = "services-s3")] pub use backend::S3Builder as S3; mod config; pub use config::S3Config; opendal-0.52.0/src/services/s3/writer.rs000064400000000000000000000155501046102023000161610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use super::core::*; use super::error::from_s3_error; use super::error::parse_error; use super::error::S3Error; use crate::raw::*; use crate::*; use bytes::Buf; use constants::{X_AMZ_OBJECT_SIZE, X_AMZ_VERSION_ID}; use http::StatusCode; pub type S3Writers = oio::MultipartWriter; pub struct S3Writer { core: Arc, op: OpWrite, path: String, } impl S3Writer { pub fn new(core: Arc, path: &str, op: OpWrite) -> Self { S3Writer { core, path: path.to_string(), op, } } fn parse_header_into_meta(path: &str, headers: &http::HeaderMap) -> Result { let mut meta = Metadata::new(EntryMode::from_path(path)); if let Some(etag) = parse_etag(headers)? { meta.set_etag(etag); } if let Some(version) = parse_header_to_str(headers, X_AMZ_VERSION_ID)? { meta.set_version(version); } if let Some(size) = parse_header_to_str(headers, X_AMZ_OBJECT_SIZE)? { if let Ok(value) = size.parse() { meta.set_content_length(value); } } Ok(meta) } } impl oio::MultipartWrite for S3Writer { async fn write_once(&self, size: u64, body: Buffer) -> Result { let mut req = self .core .s3_put_object_request(&self.path, Some(size), &self.op, body)?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); let meta = S3Writer::parse_header_into_meta(&self.path, resp.headers())?; match status { StatusCode::CREATED | StatusCode::OK => Ok(meta), _ => Err(parse_error(resp)), } } async fn initiate_part(&self) -> Result { let resp = self .core .s3_initiate_multipart_upload(&self.path, &self.op) .await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let result: InitiateMultipartUploadResult = quick_xml::de::from_reader(bs.reader()).map_err(new_xml_deserialize_error)?; Ok(result.upload_id) } _ => Err(parse_error(resp)), } } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result { // AWS S3 requires part number must between [1..=10000] let part_number = part_number + 1; let checksum = self.core.calculate_checksum(&body); let mut req = self.core.s3_upload_part_request( &self.path, upload_id, part_number, size, body, checksum.clone(), )?; self.core.sign(&mut req).await?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let etag = parse_etag(resp.headers())? .ok_or_else(|| { Error::new( ErrorKind::Unexpected, "ETag not present in returning response", ) })? .to_string(); Ok(oio::MultipartPart { part_number, etag, checksum, }) } _ => Err(parse_error(resp)), } } async fn complete_part( &self, upload_id: &str, parts: &[oio::MultipartPart], ) -> Result { let parts = parts .iter() .map(|p| match &self.core.checksum_algorithm { None => CompleteMultipartUploadRequestPart { part_number: p.part_number, etag: p.etag.clone(), ..Default::default() }, Some(checksum_algorithm) => match checksum_algorithm { ChecksumAlgorithm::Crc32c => CompleteMultipartUploadRequestPart { part_number: p.part_number, etag: p.etag.clone(), checksum_crc32c: p.checksum.clone(), }, }, }) .collect(); let resp = self .core .s3_complete_multipart_upload(&self.path, upload_id, parts) .await?; let status = resp.status(); let mut meta = S3Writer::parse_header_into_meta(&self.path, resp.headers())?; match status { StatusCode::OK => { // still check if there is any error because S3 might return error for status code 200 // https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_Example_4 let (parts, body) = resp.into_parts(); let ret: CompleteMultipartUploadResult = quick_xml::de::from_reader(body.reader()).map_err(new_xml_deserialize_error)?; if !ret.code.is_empty() { return Err(from_s3_error( S3Error { code: ret.code, message: ret.message, resource: "".to_string(), request_id: ret.request_id, }, parts, )); } meta.set_etag(&ret.etag); Ok(meta) } _ => Err(parse_error(resp)), } } async fn abort_part(&self, upload_id: &str) -> Result<()> { let resp = self .core .s3_abort_multipart_upload(&self.path, upload_id) .await?; match resp.status() { // s3 returns code 204 if abort succeeds. StatusCode::NO_CONTENT => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/seafile/backend.rs000064400000000000000000000214301046102023000173110ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use log::debug; use tokio::sync::RwLock; use super::core::parse_dir_detail; use super::core::parse_file_detail; use super::core::SeafileCore; use super::core::SeafileSigner; use super::delete::SeafileDeleter; use super::error::parse_error; use super::lister::SeafileLister; use super::writer::SeafileWriter; use super::writer::SeafileWriters; use crate::raw::*; use crate::services::SeafileConfig; use crate::*; impl Configurator for SeafileConfig { type Builder = SeafileBuilder; fn into_builder(self) -> Self::Builder { SeafileBuilder { config: self, http_client: None, } } } /// [seafile](https://www.seafile.com) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct SeafileBuilder { config: SeafileConfig, http_client: Option, } impl Debug for SeafileBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("SeafileBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl SeafileBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// endpoint of this backend. /// /// It is required. e.g. `http://127.0.0.1:80` pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.to_string()) }; self } /// username of this backend. /// /// It is required. e.g. `me@example.com` pub fn username(mut self, username: &str) -> Self { self.config.username = if username.is_empty() { None } else { Some(username.to_string()) }; self } /// password of this backend. /// /// It is required. e.g. `asecret` pub fn password(mut self, password: &str) -> Self { self.config.password = if password.is_empty() { None } else { Some(password.to_string()) }; self } /// Set repo name of this backend. /// /// It is required. e.g. `myrepo` pub fn repo_name(mut self, repo_name: &str) -> Self { self.config.repo_name = repo_name.to_string(); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for SeafileBuilder { const SCHEME: Scheme = Scheme::Seafile; type Config = SeafileConfig; /// Builds the backend and returns the result of SeafileBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle bucket. if self.config.repo_name.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "repo_name is empty") .with_operation("Builder::build") .with_context("service", Scheme::Seafile)); } debug!("backend use repo_name {}", &self.config.repo_name); let endpoint = match &self.config.endpoint { Some(endpoint) => Ok(endpoint.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_operation("Builder::build") .with_context("service", Scheme::Seafile)), }?; let username = match &self.config.username { Some(username) => Ok(username.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "username is empty") .with_operation("Builder::build") .with_context("service", Scheme::Seafile)), }?; let password = match &self.config.password { Some(password) => Ok(password.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty") .with_operation("Builder::build") .with_context("service", Scheme::Seafile)), }?; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Seafile) })? }; Ok(SeafileBackend { core: Arc::new(SeafileCore { root, endpoint, username, password, repo_name: self.config.repo_name.clone(), signer: Arc::new(RwLock::new(SeafileSigner::default())), client, }), }) } } /// Backend for seafile services. #[derive(Debug, Clone)] pub struct SeafileBackend { core: Arc, } impl Access for SeafileBackend { type Reader = HttpBody; type Writer = SeafileWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Seafile) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, write: true, write_can_empty: true, delete: true, list: true, list_has_content_length: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, _args: OpStat) -> Result { if path == "/" { return Ok(RpStat::new(Metadata::new(EntryMode::DIR))); } let metadata = if path.ends_with('/') { let dir_detail = self.core.dir_detail(path).await?; parse_dir_detail(dir_detail) } else { let file_detail = self.core.file_detail(path).await?; parse_file_detail(file_detail) }; metadata.map(RpStat::new) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.download_file(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let w = SeafileWriter::new(self.core.clone(), args, path.to_string()); let w = oio::OneShotWriter::new(w); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(SeafileDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, _args: OpList) -> Result<(RpList, Self::Lister)> { let l = SeafileLister::new(self.core.clone(), path); Ok((RpList::default(), oio::PageLister::new(l))) } } opendal-0.52.0/src/services/seafile/config.rs000064400000000000000000000035031046102023000171700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for seafile services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct SeafileConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, /// endpoint address of this backend. pub endpoint: Option, /// username of this backend. pub username: Option, /// password of this backend. pub password: Option, /// repo_name of this backend. /// /// required. pub repo_name: String, } impl Debug for SeafileConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("SeafileConfig"); d.field("root", &self.root) .field("endpoint", &self.endpoint) .field("username", &self.username) .field("repo_name", &self.repo_name); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/seafile/core.rs000064400000000000000000000267351046102023000166670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use bytes::Bytes; use http::header; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use tokio::sync::RwLock; use super::error::parse_error; use crate::raw::*; use crate::*; /// Core of [seafile](https://www.seafile.com) services support. #[derive(Clone)] pub struct SeafileCore { /// The root of this core. pub root: String, /// The endpoint of this backend. pub endpoint: String, /// The username of this backend. pub username: String, /// The password id of this backend. pub password: String, /// The repo name of this backend. pub repo_name: String, /// signer of this backend. pub signer: Arc>, pub client: HttpClient, } impl Debug for SeafileCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("username", &self.username) .field("repo_name", &self.repo_name) .finish_non_exhaustive() } } impl SeafileCore { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } /// get auth info pub async fn get_auth_info(&self) -> Result { { let signer = self.signer.read().await; if !signer.auth_info.token.is_empty() { let auth_info = signer.auth_info.clone(); return Ok(auth_info.clone()); } } { let mut signer = self.signer.write().await; let body = format!( "username={}&password={}", percent_encode_path(&self.username), percent_encode_path(&self.password) ); let req = Request::post(format!("{}/api2/auth-token/", self.endpoint)) .header(header::CONTENT_TYPE, "application/x-www-form-urlencoded") .body(Buffer::from(Bytes::from(body))) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let auth_response: AuthTokenResponse = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; signer.auth_info = AuthInfo { token: auth_response.token, repo_id: "".to_string(), }; } _ => { return Err(parse_error(resp)); } } let url = format!("{}/api2/repos", self.endpoint); let req = Request::get(url) .header( header::AUTHORIZATION, format!("Token {}", signer.auth_info.token), ) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let list_library_response: Vec = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; for library in list_library_response { if library.name == self.repo_name { signer.auth_info.repo_id = library.id; break; } } // repo not found if signer.auth_info.repo_id.is_empty() { return Err(Error::new( ErrorKind::NotFound, format!("repo {} not found", self.repo_name), )); } } _ => { return Err(parse_error(resp)); } } Ok(signer.auth_info.clone()) } } } impl SeafileCore { /// get upload url pub async fn get_upload_url(&self) -> Result { let auth_info = self.get_auth_info().await?; let req = Request::get(format!( "{}/api2/repos/{}/upload-link/", self.endpoint, auth_info.repo_id )); let req = req .header(header::AUTHORIZATION, format!("Token {}", auth_info.token)) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let upload_url = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; Ok(upload_url) } _ => Err(parse_error(resp)), } } /// get download pub async fn get_download_url(&self, path: &str) -> Result { let path = build_abs_path(&self.root, path); let path = percent_encode_path(&path); let auth_info = self.get_auth_info().await?; let req = Request::get(format!( "{}/api2/repos/{}/file/?p={}", self.endpoint, auth_info.repo_id, path )); let req = req .header(header::AUTHORIZATION, format!("Token {}", auth_info.token)) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let download_url = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; Ok(download_url) } _ => Err(parse_error(resp)), } } /// download file pub async fn download_file(&self, path: &str, range: BytesRange) -> Result> { let download_url = self.get_download_url(path).await?; let req = Request::get(download_url); let req = req .header(header::RANGE, range.to_header()) .body(Buffer::new()) .map_err(new_request_build_error)?; self.client.fetch(req).await } /// file detail pub async fn file_detail(&self, path: &str) -> Result { let path = build_abs_path(&self.root, path); let path = percent_encode_path(&path); let auth_info = self.get_auth_info().await?; let req = Request::get(format!( "{}/api2/repos/{}/file/detail/?p={}", self.endpoint, auth_info.repo_id, path )); let req = req .header(header::AUTHORIZATION, format!("Token {}", auth_info.token)) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let file_detail: FileDetail = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; Ok(file_detail) } _ => Err(parse_error(resp)), } } /// dir detail pub async fn dir_detail(&self, path: &str) -> Result { let path = build_abs_path(&self.root, path); let path = percent_encode_path(&path); let auth_info = self.get_auth_info().await?; let req = Request::get(format!( "{}/api/v2.1/repos/{}/dir/detail/?path={}", self.endpoint, auth_info.repo_id, path )); let req = req .header(header::AUTHORIZATION, format!("Token {}", auth_info.token)) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let dir_detail: DirDetail = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; Ok(dir_detail) } _ => Err(parse_error(resp)), } } /// delete file or dir pub async fn delete(&self, path: &str) -> Result<()> { let path = build_abs_path(&self.root, path); let path = percent_encode_path(&path); let auth_info = self.get_auth_info().await?; let url = if path.ends_with('/') { format!( "{}/api2/repos/{}/dir/?p={}", self.endpoint, auth_info.repo_id, path ) } else { format!( "{}/api2/repos/{}/file/?p={}", self.endpoint, auth_info.repo_id, path ) }; let req = Request::delete(url); let req = req .header(header::AUTHORIZATION, format!("Token {}", auth_info.token)) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } } #[derive(Deserialize)] pub struct AuthTokenResponse { pub token: String, } #[derive(Deserialize)] pub struct FileDetail { pub last_modified: String, pub size: u64, } #[derive(Debug, Deserialize)] pub struct DirDetail { mtime: String, } pub fn parse_dir_detail(dir_detail: DirDetail) -> Result { let mut md = Metadata::new(EntryMode::DIR); md.set_last_modified(parse_datetime_from_rfc3339(&dir_detail.mtime)?); Ok(md) } pub fn parse_file_detail(file_detail: FileDetail) -> Result { let mut md = Metadata::new(EntryMode::FILE); md.set_content_length(file_detail.size); md.set_last_modified(parse_datetime_from_rfc3339(&file_detail.last_modified)?); Ok(md) } #[derive(Clone, Default)] pub struct SeafileSigner { pub auth_info: AuthInfo, } #[derive(Clone, Default)] pub struct AuthInfo { /// The repo id of this auth info. pub repo_id: String, /// The token of this auth info, pub token: String, } #[derive(Deserialize)] pub struct ListLibraryResponse { pub name: String, pub id: String, } opendal-0.52.0/src/services/seafile/delete.rs000064400000000000000000000023051046102023000171640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use crate::raw::*; use crate::*; use std::sync::Arc; pub struct SeafileDeleter { core: Arc, } impl SeafileDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for SeafileDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { self.core.delete(&path).await?; Ok(()) } } opendal-0.52.0/src/services/seafile/docs.md000064400000000000000000000021401046102023000166230ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `endpoint`: Seafile endpoint address - `username` Seafile username - `password` Seafile password - `repo_name` Seafile repo name You can refer to [`SeafileBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Seafile; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Seafile::default() // set the storage bucket for OpenDAL .root("/") // set the endpoint for OpenDAL .endpoint("http://127.0.0.1:80") // set the username for OpenDAL .username("xxxxxxxxxx") // set the password name for OpenDAL .password("opendal") // set the repo_name for OpenDAL .repo_name("xxxxxxxxxxxxx"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/seafile/error.rs000064400000000000000000000052161046102023000170570ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use serde::Deserialize; use crate::raw::*; use crate::*; /// the error response of seafile #[derive(Default, Debug, Deserialize)] #[allow(dead_code)] struct SeafileError { error_msg: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, _retryable) = match parts.status.as_u16() { 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), 520 => (ErrorKind::Unexpected, false), _ => (ErrorKind::Unexpected, false), }; let (message, _seafile_err) = serde_json::from_reader::<_, SeafileError>(bs.clone().reader()) .map(|seafile_err| (format!("{seafile_err:?}"), Some(seafile_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); err } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[tokio::test] async fn test_parse_error() { let err_res = vec![ ( r#"{"error_msg": "Permission denied"}"#, ErrorKind::PermissionDenied, StatusCode::FORBIDDEN, ), ( r#"{"error_msg": "Folder /e982e75a-fead-487c-9f41-63094d9bf0de/a9d867b9-778d-4612-b674-47e674c14c28/ not found."}"#, ErrorKind::NotFound, StatusCode::NOT_FOUND, ), ]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/seafile/lister.rs000064400000000000000000000073741046102023000172370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::header; use http::Request; use http::StatusCode; use serde::Deserialize; use super::core::SeafileCore; use super::error::parse_error; use crate::raw::oio::Entry; use crate::raw::*; use crate::*; pub struct SeafileLister { core: Arc, path: String, } impl SeafileLister { pub(super) fn new(core: Arc, path: &str) -> Self { SeafileLister { core, path: path.to_string(), } } } impl oio::PageList for SeafileLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let path = build_rooted_abs_path(&self.core.root, &self.path); let auth_info = self.core.get_auth_info().await?; let url = format!( "{}/api2/repos/{}/dir/?p={}", self.core.endpoint, auth_info.repo_id, percent_encode_path(&path) ); let req = Request::get(url); let req = req .header(header::AUTHORIZATION, format!("Token {}", auth_info.token)) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let resp_body = resp.into_body(); let infos: Vec = serde_json::from_reader(resp_body.reader()) .map_err(new_json_deserialize_error)?; // add path itself ctx.entries.push_back(Entry::new( self.path.as_str(), Metadata::new(EntryMode::DIR), )); for info in infos { if !info.name.is_empty() { let rel_path = build_rel_path(&self.core.root, &format!("{}{}", path, info.name)); let entry = if info.type_field == "file" { let meta = Metadata::new(EntryMode::FILE) .with_last_modified(parse_datetime_from_from_timestamp(info.mtime)?) .with_content_length(info.size.unwrap_or(0)); Entry::new(&rel_path, meta) } else { let path = format!("{}/", rel_path); Entry::new(&path, Metadata::new(EntryMode::DIR)) }; ctx.entries.push_back(entry); } } ctx.done = true; Ok(()) } // return nothing when not exist StatusCode::NOT_FOUND => { ctx.done = true; Ok(()) } _ => Err(parse_error(resp)), } } } #[derive(Debug, Deserialize)] struct Info { #[serde(rename = "type")] pub type_field: String, pub mtime: i64, pub size: Option, pub name: String, } opendal-0.52.0/src/services/seafile/mod.rs000064400000000000000000000023041046102023000165000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-seafile")] mod core; #[cfg(feature = "services-seafile")] mod delete; #[cfg(feature = "services-seafile")] mod error; #[cfg(feature = "services-seafile")] mod lister; #[cfg(feature = "services-seafile")] mod writer; #[cfg(feature = "services-seafile")] mod backend; #[cfg(feature = "services-seafile")] pub use backend::SeafileBuilder as Seafile; mod config; pub use config::SeafileConfig; opendal-0.52.0/src/services/seafile/writer.rs000064400000000000000000000052321046102023000172400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::header; use http::Request; use http::StatusCode; use super::core::SeafileCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type SeafileWriters = oio::OneShotWriter; pub struct SeafileWriter { core: Arc, _op: OpWrite, path: String, } impl SeafileWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { SeafileWriter { core, _op: op, path, } } } impl oio::OneShotWrite for SeafileWriter { async fn write_once(&self, bs: Buffer) -> Result { let upload_url = self.core.get_upload_url().await?; let req = Request::post(upload_url); let (filename, relative_path) = if self.path.ends_with('/') { ("", build_abs_path(&self.core.root, &self.path)) } else { let (filename, relative_path) = (get_basename(&self.path), get_parent(&self.path)); (filename, build_abs_path(&self.core.root, relative_path)) }; let file_part = FormDataPart::new("file") .header( header::CONTENT_DISPOSITION, format!("form-data; name=\"file\"; filename=\"{filename}\"") .parse() .unwrap(), ) .content(bs); let multipart = Multipart::new() .part(FormDataPart::new("parent_dir").content("/")) .part(FormDataPart::new("relative_path").content(relative_path.clone())) .part(FormDataPart::new("replace").content("1")) .part(file_part); let req = multipart.apply(req)?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/sftp/backend.rs000064400000000000000000000353171046102023000166660ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::io::SeekFrom; use std::path::Path; use std::path::PathBuf; use std::sync::Arc; use bb8::PooledConnection; use bb8::RunError; use log::debug; use openssh::KnownHosts; use openssh::SessionBuilder; use openssh_sftp_client::Sftp; use openssh_sftp_client::SftpOptions; use tokio::io::AsyncSeekExt; use tokio::sync::OnceCell; use super::delete::SftpDeleter; use super::error::is_not_found; use super::error::is_sftp_protocol_error; use super::error::parse_sftp_error; use super::error::parse_ssh_error; use super::lister::SftpLister; use super::reader::SftpReader; use super::writer::SftpWriter; use crate::raw::*; use crate::services::SftpConfig; use crate::*; impl Configurator for SftpConfig { type Builder = SftpBuilder; fn into_builder(self) -> Self::Builder { SftpBuilder { config: self } } } /// SFTP services support. (only works on unix) /// /// If you are interested in working on windows, please refer to [this](https://github.com/apache/opendal/issues/2963) issue. /// Welcome to leave your comments or make contributions. /// /// Warning: Maximum number of file holdings is depending on the remote system configuration. /// /// For example, the default value is 255 in macOS, and 1024 in linux. If you want to open /// lots of files, you should pay attention to close the file after using it. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct SftpBuilder { config: SftpConfig, } impl Debug for SftpBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("SftpBuilder") .field("config", &self.config) .finish() } } impl SftpBuilder { /// set endpoint for sftp backend. /// The format is same as `openssh`, using either `[user@]hostname` or `ssh://[user@]hostname[:port]`. A username or port that is specified in the endpoint overrides the one set in the builder (but does not change the builder). pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.to_string()) }; self } /// set root path for sftp backend. /// It uses the default directory set by the remote `sftp-server` as default. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// set user for sftp backend. pub fn user(mut self, user: &str) -> Self { self.config.user = if user.is_empty() { None } else { Some(user.to_string()) }; self } /// set key path for sftp backend. pub fn key(mut self, key: &str) -> Self { self.config.key = if key.is_empty() { None } else { Some(key.to_string()) }; self } /// set known_hosts strategy for sftp backend. /// available values: /// - Strict (default) /// - Accept /// - Add pub fn known_hosts_strategy(mut self, strategy: &str) -> Self { self.config.known_hosts_strategy = if strategy.is_empty() { None } else { Some(strategy.to_string()) }; self } /// set enable_copy for sftp backend. /// It requires the server supports copy-file extension. pub fn enable_copy(mut self, enable_copy: bool) -> Self { self.config.enable_copy = enable_copy; self } } impl Builder for SftpBuilder { const SCHEME: Scheme = Scheme::Sftp; type Config = SftpConfig; fn build(self) -> Result { debug!("sftp backend build started: {:?}", &self); let endpoint = match self.config.endpoint.clone() { Some(v) => v, None => return Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty")), }; let user = self.config.user.clone(); let root = self .config .root .clone() .map(|r| normalize_root(r.as_str())) .unwrap_or_default(); let known_hosts_strategy = match &self.config.known_hosts_strategy { Some(v) => { let v = v.to_lowercase(); if v == "strict" { KnownHosts::Strict } else if v == "accept" { KnownHosts::Accept } else if v == "add" { KnownHosts::Add } else { return Err(Error::new( ErrorKind::ConfigInvalid, format!("unknown known_hosts strategy: {}", v).as_str(), )); } } None => KnownHosts::Strict, }; debug!("sftp backend finished: {:?}", &self); Ok(SftpBackend { endpoint, root, user, key: self.config.key.clone(), known_hosts_strategy, copyable: self.config.enable_copy, client: OnceCell::new(), }) } } /// Backend is used to serve `Accessor` support for sftp. #[derive(Clone)] pub struct SftpBackend { copyable: bool, endpoint: String, pub root: String, user: Option, key: Option, known_hosts_strategy: KnownHosts, pub client: OnceCell>, } pub struct Manager { endpoint: String, root: String, user: Option, key: Option, known_hosts_strategy: KnownHosts, } #[async_trait::async_trait] impl bb8::ManageConnection for Manager { type Connection = Sftp; type Error = Error; async fn connect(&self) -> Result { let mut session = SessionBuilder::default(); if let Some(user) = &self.user { session.user(user.clone()); } if let Some(key) = &self.key { session.keyfile(key); } session.known_hosts_check(self.known_hosts_strategy.clone()); let session = session .connect(&self.endpoint) .await .map_err(parse_ssh_error)?; let sftp = Sftp::from_session(session, SftpOptions::default()) .await .map_err(parse_sftp_error)?; if !self.root.is_empty() { let mut fs = sftp.fs(); let paths = Path::new(&self.root).components(); let mut current = PathBuf::new(); for p in paths { current.push(p); let res = fs.create_dir(p).await; if let Err(e) = res { // ignore error if dir already exists if !is_sftp_protocol_error(&e) { return Err(parse_sftp_error(e)); } } fs.set_cwd(¤t); } } debug!("sftp connection created at {}", self.root); Ok(sftp) } // Check if connect valid by checking the root path. async fn is_valid(&self, conn: &mut Self::Connection) -> Result<(), Self::Error> { let _ = conn.fs().metadata("./").await.map_err(parse_sftp_error)?; Ok(()) } /// Always allow reuse conn. fn has_broken(&self, _: &mut Self::Connection) -> bool { false } } impl Debug for SftpBackend { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend").finish() } } impl SftpBackend { pub async fn connect(&self) -> Result> { let client = self .client .get_or_try_init(|| async { bb8::Pool::builder() .max_size(64) .build(Manager { endpoint: self.endpoint.clone(), root: self.root.clone(), user: self.user.clone(), key: self.key.clone(), known_hosts_strategy: self.known_hosts_strategy.clone(), }) .await }) .await?; client.get_owned().await.map_err(|err| match err { RunError::User(err) => err, RunError::TimedOut => { Error::new(ErrorKind::Unexpected, "connection request: timeout").set_temporary() } }) } } impl Access for SftpBackend { type Reader = SftpReader; type Writer = SftpWriter; type Lister = Option; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_root(self.root.as_str()) .set_scheme(Scheme::Sftp) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, write: true, write_can_multi: true, create_dir: true, delete: true, list: true, list_with_limit: true, list_has_content_length: true, list_has_last_modified: true, copy: self.copyable, rename: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let client = self.connect().await?; let mut fs = client.fs(); fs.set_cwd(&self.root); let paths = Path::new(&path).components(); let mut current = PathBuf::from(&self.root); for p in paths { current = current.join(p); let res = fs.create_dir(p).await; if let Err(e) = res { // ignore error if dir already exists if !is_sftp_protocol_error(&e) { return Err(parse_sftp_error(e)); } } fs.set_cwd(¤t); } Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _: OpStat) -> Result { let client = self.connect().await?; let mut fs = client.fs(); fs.set_cwd(&self.root); let meta: Metadata = fs.metadata(path).await.map_err(parse_sftp_error)?.into(); Ok(RpStat::new(meta)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let client = self.connect().await?; let mut fs = client.fs(); fs.set_cwd(&self.root); let path = fs.canonicalize(path).await.map_err(parse_sftp_error)?; let mut f = client .open(path.as_path()) .await .map_err(parse_sftp_error)?; if args.range().offset() != 0 { f.seek(SeekFrom::Start(args.range().offset())) .await .map_err(new_std_io_error)?; } Ok(( RpRead::default(), SftpReader::new(client, f, args.range().size()), )) } async fn write(&self, path: &str, op: OpWrite) -> Result<(RpWrite, Self::Writer)> { if let Some((dir, _)) = path.rsplit_once('/') { self.create_dir(dir, OpCreateDir::default()).await?; } let client = self.connect().await?; let mut fs = client.fs(); fs.set_cwd(&self.root); let path = fs.canonicalize(path).await.map_err(parse_sftp_error)?; let mut option = client.options(); option.create(true); if op.append() { option.append(true); } else { option.write(true).truncate(true); } let file = option.open(path).await.map_err(parse_sftp_error)?; Ok((RpWrite::new(), SftpWriter::new(file))) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(SftpDeleter::new(Arc::new(self.clone()))), )) } async fn list(&self, path: &str, _: OpList) -> Result<(RpList, Self::Lister)> { let client = self.connect().await?; let mut fs = client.fs(); fs.set_cwd(&self.root); let file_path = format!("./{}", path); let dir = match fs.open_dir(&file_path).await { Ok(dir) => dir, Err(e) => { if is_not_found(&e) { return Ok((RpList::default(), None)); } else { return Err(parse_sftp_error(e)); } } } .read_dir(); Ok(( RpList::default(), Some(SftpLister::new(dir, path.to_owned())), )) } async fn copy(&self, from: &str, to: &str, _: OpCopy) -> Result { let client = self.connect().await?; let mut fs = client.fs(); fs.set_cwd(&self.root); if let Some((dir, _)) = to.rsplit_once('/') { self.create_dir(dir, OpCreateDir::default()).await?; } let src = fs.canonicalize(from).await.map_err(parse_sftp_error)?; let dst = fs.canonicalize(to).await.map_err(parse_sftp_error)?; let mut src_file = client.open(&src).await.map_err(parse_sftp_error)?; let mut dst_file = client.create(dst).await.map_err(parse_sftp_error)?; src_file .copy_all_to(&mut dst_file) .await .map_err(parse_sftp_error)?; Ok(RpCopy::default()) } async fn rename(&self, from: &str, to: &str, _: OpRename) -> Result { let client = self.connect().await?; let mut fs = client.fs(); fs.set_cwd(&self.root); if let Some((dir, _)) = to.rsplit_once('/') { self.create_dir(dir, OpCreateDir::default()).await?; } fs.rename(from, to).await.map_err(parse_sftp_error)?; Ok(RpRename::default()) } } opendal-0.52.0/src/services/sftp/config.rs000064400000000000000000000032641046102023000165400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Sftp Service support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct SftpConfig { /// endpoint of this backend pub endpoint: Option, /// root of this backend pub root: Option, /// user of this backend pub user: Option, /// key of this backend pub key: Option, /// known_hosts_strategy of this backend pub known_hosts_strategy: Option, /// enable_copy of this backend pub enable_copy: bool, } impl Debug for SftpConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("SftpConfig") .field("endpoint", &self.endpoint) .field("root", &self.root) .finish_non_exhaustive() } } opendal-0.52.0/src/services/sftp/delete.rs000064400000000000000000000031621046102023000165320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::backend::SftpBackend; use super::error::{is_not_found, parse_sftp_error}; use crate::raw::*; use crate::*; use std::sync::Arc; pub struct SftpDeleter { core: Arc, } impl SftpDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for SftpDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let client = self.core.connect().await?; let mut fs = client.fs(); fs.set_cwd(&self.core.root); let res = if path.ends_with('/') { fs.remove_dir(path).await } else { fs.remove_file(path).await }; match res { Ok(()) => Ok(()), Err(e) if is_not_found(&e) => Ok(()), Err(e) => Err(parse_sftp_error(e)), } } } opendal-0.52.0/src/services/sftp/docs.md000064400000000000000000000024771046102023000162040ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] append - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [ ] ~~presign~~ - [ ] blocking ## Configuration - `endpoint`: Set the endpoint for connection. The format is same as `openssh`, using either `[user@]hostname` or `ssh://[user@]hostname[:port]`. A username or port that is specified in the endpoint overrides the one set in the builder (but does not change the builder). - `root`: Set the work directory for backend. It uses the default directory set by the remote `sftp-server` as default - `user`: Set the login user - `key`: Set the public key for login - `known_hosts_strategy`: Set the strategy for known hosts, default to `Strict` - `enable_copy`: Set whether the remote server has copy-file extension For security reasons, it doesn't support password login, you can use public key or ssh-copy-id instead. You can refer to [`SftpBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Sftp; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Sftp::default() .endpoint("127.0.0.1") .user("test") .key("test_key"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/sftp/error.rs000064400000000000000000000040421046102023000164170ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use openssh::Error as SshError; use openssh_sftp_client::error::SftpErrorKind; use openssh_sftp_client::Error as SftpClientError; use crate::Error; use crate::ErrorKind; pub fn parse_sftp_error(e: SftpClientError) -> Error { let kind = match &e { SftpClientError::UnsupportedSftpProtocol { version: _ } => ErrorKind::Unsupported, SftpClientError::SftpError(kind, _msg) => match kind { SftpErrorKind::NoSuchFile => ErrorKind::NotFound, SftpErrorKind::PermDenied => ErrorKind::PermissionDenied, SftpErrorKind::OpUnsupported => ErrorKind::Unsupported, _ => ErrorKind::Unexpected, }, _ => ErrorKind::Unexpected, }; let mut err = Error::new(kind, "sftp error").set_source(e); // Mark error as temporary if it's unexpected. if kind == ErrorKind::Unexpected { err = err.set_temporary(); } err } pub fn parse_ssh_error(e: SshError) -> Error { Error::new(ErrorKind::Unexpected, "ssh error").set_source(e) } pub(super) fn is_not_found(e: &SftpClientError) -> bool { matches!(e, SftpClientError::SftpError(SftpErrorKind::NoSuchFile, _)) } pub(super) fn is_sftp_protocol_error(e: &SftpClientError) -> bool { matches!(e, SftpClientError::SftpError(_, _)) } opendal-0.52.0/src/services/sftp/lister.rs000064400000000000000000000051131046102023000165700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::pin::Pin; use futures::StreamExt; use openssh_sftp_client::fs::DirEntry; use openssh_sftp_client::fs::ReadDir; use super::error::parse_sftp_error; use crate::raw::oio; use crate::raw::oio::Entry; use crate::Result; pub struct SftpLister { dir: Pin>, prefix: String, } impl SftpLister { pub fn new(dir: ReadDir, path: String) -> Self { let prefix = if path == "/" { "".to_owned() } else { path }; SftpLister { dir: Box::pin(dir), prefix, } } } impl oio::List for SftpLister { async fn next(&mut self) -> Result> { loop { let item = self .dir .next() .await .transpose() .map_err(parse_sftp_error)?; match item { Some(e) => { if e.filename().to_str() == Some("..") { continue; } else if e.filename().to_str() == Some(".") { let mut path = self.prefix.as_str(); if self.prefix.is_empty() { path = "/"; } return Ok(Some(Entry::new(path, e.metadata().into()))); } else { return Ok(Some(map_entry(self.prefix.as_str(), e))); } } None => return Ok(None), } } } } fn map_entry(prefix: &str, value: DirEntry) -> Entry { let path = format!( "{}{}{}", prefix, value.filename().to_str().unwrap(), if value.file_type().unwrap().is_dir() { "/" } else { "" } ); Entry::new(path.as_str(), value.metadata().into()) } opendal-0.52.0/src/services/sftp/mod.rs000064400000000000000000000023251046102023000160470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-sftp")] mod delete; #[cfg(feature = "services-sftp")] mod error; #[cfg(feature = "services-sftp")] mod lister; #[cfg(feature = "services-sftp")] mod reader; #[cfg(feature = "services-sftp")] mod utils; #[cfg(feature = "services-sftp")] mod writer; #[cfg(feature = "services-sftp")] mod backend; #[cfg(feature = "services-sftp")] pub use backend::SftpBuilder as Sftp; mod config; pub use config::SftpConfig; opendal-0.52.0/src/services/sftp/reader.rs000064400000000000000000000043551046102023000165370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bb8::PooledConnection; use bytes::BytesMut; use openssh_sftp_client::file::File; use super::backend::Manager; use super::error::parse_sftp_error; use crate::raw::*; use crate::*; pub struct SftpReader { /// Keep the connection alive while data stream is alive. _conn: PooledConnection<'static, Manager>, file: File, chunk: usize, size: Option, read: usize, buf: BytesMut, } impl SftpReader { pub fn new(conn: PooledConnection<'static, Manager>, file: File, size: Option) -> Self { Self { _conn: conn, file, size: size.map(|v| v as usize), chunk: 2 * 1024 * 1024, read: 0, buf: BytesMut::new(), } } } impl oio::Read for SftpReader { async fn read(&mut self) -> Result { if self.read >= self.size.unwrap_or(usize::MAX) { return Ok(Buffer::new()); } let size = if let Some(size) = self.size { (size - self.read).min(self.chunk) } else { self.chunk }; self.buf.reserve(size); let Some(bytes) = self .file .read(size as u32, self.buf.split_off(0)) .await .map_err(parse_sftp_error)? else { return Ok(Buffer::new()); }; self.read += bytes.len(); self.buf = bytes; let bs = self.buf.split(); Ok(Buffer::from(bs.freeze())) } } opendal-0.52.0/src/services/sftp/utils.rs000064400000000000000000000033051046102023000164270ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use openssh_sftp_client::metadata::MetaData as SftpMeta; use crate::EntryMode; use crate::Metadata; /// REMOVE ME: we should not implement `From for Metadata`. impl From for Metadata { fn from(meta: SftpMeta) -> Self { let mode = meta .file_type() .map(|filetype| { if filetype.is_file() { EntryMode::FILE } else if filetype.is_dir() { EntryMode::DIR } else { EntryMode::Unknown } }) .unwrap_or(EntryMode::Unknown); let mut metadata = Metadata::new(mode); if let Some(size) = meta.len() { metadata.set_content_length(size); } if let Some(modified) = meta.modified() { metadata.set_last_modified(modified.as_system_time().into()); } metadata } } opendal-0.52.0/src/services/sftp/writer.rs000064400000000000000000000035771046102023000166160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::pin::Pin; use bytes::Buf; use openssh_sftp_client::file::File; use openssh_sftp_client::file::TokioCompatFile; use tokio::io::AsyncWriteExt; use crate::raw::*; use crate::*; pub struct SftpWriter { /// TODO: maybe we can use `File` directly? file: Pin>, } impl SftpWriter { pub fn new(file: File) -> Self { SftpWriter { file: Box::pin(TokioCompatFile::new(file)), } } } impl oio::Write for SftpWriter { async fn write(&mut self, mut bs: Buffer) -> Result<()> { while bs.has_remaining() { let n = self .file .write(bs.chunk()) .await .map_err(new_std_io_error)?; bs.advance(n); } Ok(()) } async fn close(&mut self) -> Result { self.file.shutdown().await.map_err(new_std_io_error)?; Ok(Metadata::default()) } async fn abort(&mut self) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "SftpWriter doesn't support abort", )) } } opendal-0.52.0/src/services/sled/backend.rs000064400000000000000000000153011046102023000166300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::str; use tokio::task; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::SledConfig; use crate::Builder; use crate::Error; use crate::ErrorKind; use crate::Scheme; use crate::*; // https://github.com/spacejam/sled/blob/69294e59c718289ab3cb6bd03ac3b9e1e072a1e7/src/db.rs#L5 const DEFAULT_TREE_ID: &str = r#"__sled__default"#; impl Configurator for SledConfig { type Builder = SledBuilder; fn into_builder(self) -> Self::Builder { SledBuilder { config: self } } } /// Sled services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct SledBuilder { config: SledConfig, } impl Debug for SledBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("SledBuilder") .field("config", &self.config) .finish() } } impl SledBuilder { /// Set the path to the sled data directory. Will create if not exists. pub fn datadir(mut self, path: &str) -> Self { self.config.datadir = Some(path.into()); self } /// Set the root for sled. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the tree for sled. pub fn tree(mut self, tree: &str) -> Self { self.config.tree = Some(tree.into()); self } } impl Builder for SledBuilder { const SCHEME: Scheme = Scheme::Sled; type Config = SledConfig; fn build(self) -> Result { let datadir_path = self.config.datadir.ok_or_else(|| { Error::new(ErrorKind::ConfigInvalid, "datadir is required but not set") .with_context("service", Scheme::Sled) })?; let db = sled::open(&datadir_path).map_err(|e| { Error::new(ErrorKind::ConfigInvalid, "open db") .with_context("service", Scheme::Sled) .with_context("datadir", datadir_path.clone()) .set_source(e) })?; // use "default" tree if not set let tree_name = self .config .tree .unwrap_or_else(|| DEFAULT_TREE_ID.to_string()); let tree = db.open_tree(&tree_name).map_err(|e| { Error::new(ErrorKind::ConfigInvalid, "open tree") .with_context("service", Scheme::Sled) .with_context("datadir", datadir_path.clone()) .with_context("tree", tree_name.clone()) .set_source(e) })?; Ok(SledBackend::new(Adapter { datadir: datadir_path, tree, }) .with_root(self.config.root.as_deref().unwrap_or("/"))) } } /// Backend for sled services. pub type SledBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { datadir: String, tree: sled::Tree, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.field("path", &self.datadir); ds.finish() } } impl kv::Adapter for Adapter { type Scanner = kv::Scanner; fn info(&self) -> kv::Info { kv::Info::new( Scheme::Sled, &self.datadir, Capability { read: true, write: true, list: true, blocking: true, shared: false, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_get(cloned_path.as_str())) .await .map_err(new_task_join_error)? } fn blocking_get(&self, path: &str) -> Result> { Ok(self .tree .get(path) .map_err(parse_error)? .map(|v| Buffer::from(v.to_vec()))) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_set(cloned_path.as_str(), value)) .await .map_err(new_task_join_error)? } fn blocking_set(&self, path: &str, value: Buffer) -> Result<()> { self.tree .insert(path, value.to_vec()) .map_err(parse_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let cloned_self = self.clone(); let cloned_path = path.to_string(); task::spawn_blocking(move || cloned_self.blocking_delete(cloned_path.as_str())) .await .map_err(new_task_join_error)? } fn blocking_delete(&self, path: &str) -> Result<()> { self.tree.remove(path).map_err(parse_error)?; Ok(()) } async fn scan(&self, path: &str) -> Result { let cloned_self = self.clone(); let cloned_path = path.to_string(); let res = task::spawn_blocking(move || cloned_self.blocking_scan(cloned_path.as_str())) .await .map_err(new_task_join_error)??; Ok(Box::new(kv::ScanStdIter::new(res.into_iter().map(Ok)))) } fn blocking_scan(&self, path: &str) -> Result> { let it = self.tree.scan_prefix(path).keys(); let mut res = Vec::default(); for i in it { let bs = i.map_err(parse_error)?.to_vec(); let v = String::from_utf8(bs).map_err(|err| { Error::new(ErrorKind::Unexpected, "store key is not valid utf-8 string") .set_source(err) })?; res.push(v); } Ok(res) } } fn parse_error(err: sled::Error) -> Error { Error::new(ErrorKind::Unexpected, "error from sled").set_source(err) } opendal-0.52.0/src/services/sled/config.rs000064400000000000000000000030001046102023000164770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Sled services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct SledConfig { /// That path to the sled data directory. pub datadir: Option, /// The root for sled. pub root: Option, /// The tree for sled. pub tree: Option, } impl Debug for SledConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("SledConfig") .field("datadir", &self.datadir) .field("root", &self.root) .field("tree", &self.tree) .finish() } } opendal-0.52.0/src/services/sled/docs.md000064400000000000000000000011721046102023000161460ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [x] blocking ## Configuration - `datadir`: Set the path to the sled data directory You can refer to [`SledBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Sled; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Sled::default() .datadir("/tmp/opendal/sled"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/sled/mod.rs000064400000000000000000000017021046102023000160200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-sled")] mod backend; #[cfg(feature = "services-sled")] pub use backend::SledBuilder as Sled; mod config; pub use config::SledConfig; opendal-0.52.0/src/services/sqlite/backend.rs000064400000000000000000000211051046102023000172010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::pin::Pin; use std::str::FromStr; use std::task::Context; use std::task::Poll; use futures::stream::BoxStream; use futures::Stream; use futures::StreamExt; use ouroboros::self_referencing; use sqlx::sqlite::SqliteConnectOptions; use sqlx::SqlitePool; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::*; use crate::services::SqliteConfig; use crate::*; impl Configurator for SqliteConfig { type Builder = SqliteBuilder; fn into_builder(self) -> Self::Builder { SqliteBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct SqliteBuilder { config: SqliteConfig, } impl Debug for SqliteBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("SqliteBuilder"); ds.field("config", &self.config); ds.finish() } } impl SqliteBuilder { /// Set the connection_string of the sqlite service. /// /// This connection string is used to connect to the sqlite service. There are url based formats: /// /// ## Url /// /// This format resembles the url format of the sqlite client: /// /// - `sqlite::memory:` /// - `sqlite:data.db` /// - `sqlite://data.db` /// /// For more information, please visit . pub fn connection_string(mut self, v: &str) -> Self { if !v.is_empty() { self.config.connection_string = Some(v.to_string()); } self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the table name of the sqlite service to read/write. pub fn table(mut self, table: &str) -> Self { if !table.is_empty() { self.config.table = Some(table.to_string()); } self } /// Set the key field name of the sqlite service to read/write. /// /// Default to `key` if not specified. pub fn key_field(mut self, key_field: &str) -> Self { if !key_field.is_empty() { self.config.key_field = Some(key_field.to_string()); } self } /// Set the value field name of the sqlite service to read/write. /// /// Default to `value` if not specified. pub fn value_field(mut self, value_field: &str) -> Self { if !value_field.is_empty() { self.config.value_field = Some(value_field.to_string()); } self } } impl Builder for SqliteBuilder { const SCHEME: Scheme = Scheme::Sqlite; type Config = SqliteConfig; fn build(self) -> Result { let conn = match self.config.connection_string { Some(v) => v, None => { return Err(Error::new( ErrorKind::ConfigInvalid, "connection_string is required but not set", ) .with_context("service", Scheme::Sqlite)); } }; let config = SqliteConnectOptions::from_str(&conn).map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "connection_string is invalid") .with_context("service", Scheme::Sqlite) .set_source(err) })?; let table = match self.config.table { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "table is empty") .with_context("service", Scheme::Sqlite)); } }; let key_field = self.config.key_field.unwrap_or_else(|| "key".to_string()); let value_field = self .config .value_field .unwrap_or_else(|| "value".to_string()); let root = normalize_root(self.config.root.as_deref().unwrap_or("/")); Ok(SqliteBackend::new(Adapter { pool: OnceCell::new(), config, table, key_field, value_field, }) .with_normalized_root(root)) } } pub type SqliteBackend = kv::Backend; #[derive(Debug, Clone)] pub struct Adapter { pool: OnceCell, config: SqliteConnectOptions, table: String, key_field: String, value_field: String, } impl Adapter { async fn get_client(&self) -> Result<&SqlitePool> { self.pool .get_or_try_init(|| async { let pool = SqlitePool::connect_with(self.config.clone()) .await .map_err(parse_sqlite_error)?; Ok(pool) }) .await } } #[self_referencing] pub struct SqliteScanner { pool: SqlitePool, query: String, #[borrows(pool, query)] #[covariant] stream: BoxStream<'this, Result>, } impl Stream for SqliteScanner { type Item = Result; fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { self.with_stream_mut(|s| s.poll_next_unpin(cx)) } } unsafe impl Sync for SqliteScanner {} impl kv::Scan for SqliteScanner { async fn next(&mut self) -> Result> { ::next(self).await.transpose() } } impl kv::Adapter for Adapter { type Scanner = SqliteScanner; fn info(&self) -> kv::Info { kv::Info::new( Scheme::Sqlite, &self.table, Capability { read: true, write: true, delete: true, list: true, shared: false, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let pool = self.get_client().await?; let value: Option> = sqlx::query_scalar(&format!( "SELECT `{}` FROM `{}` WHERE `{}` = $1 LIMIT 1", self.value_field, self.table, self.key_field )) .bind(path) .fetch_optional(pool) .await .map_err(parse_sqlite_error)?; Ok(value.map(Buffer::from)) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { let pool = self.get_client().await?; sqlx::query(&format!( "INSERT OR REPLACE INTO `{}` (`{}`, `{}`) VALUES ($1, $2)", self.table, self.key_field, self.value_field, )) .bind(path) .bind(value.to_vec()) .execute(pool) .await .map_err(parse_sqlite_error)?; Ok(()) } async fn delete(&self, path: &str) -> Result<()> { let pool = self.get_client().await?; sqlx::query(&format!( "DELETE FROM `{}` WHERE `{}` = $1", self.table, self.key_field )) .bind(path) .execute(pool) .await .map_err(parse_sqlite_error)?; Ok(()) } async fn scan(&self, path: &str) -> Result { let pool = self.get_client().await?; let stream = SqliteScannerBuilder { pool: pool.clone(), query: format!( "SELECT `{}` FROM `{}` WHERE `{}` LIKE $1", self.key_field, self.table, self.key_field ), stream_builder: |pool, query| { sqlx::query_scalar(query) .bind(format!("{path}%")) .fetch(pool) .map(|v| v.map_err(parse_sqlite_error)) .boxed() }, } .build(); Ok(stream) } } fn parse_sqlite_error(err: sqlx::Error) -> Error { Error::new(ErrorKind::Unexpected, "unhandled error from sqlite").set_source(err) } opendal-0.52.0/src/services/sqlite/config.rs000064400000000000000000000047421046102023000170670ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Sqlite support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct SqliteConfig { /// Set the connection_string of the sqlite service. /// /// This connection string is used to connect to the sqlite service. /// /// The format of connect string resembles the url format of the sqlite client: /// /// - `sqlite::memory:` /// - `sqlite:data.db` /// - `sqlite://data.db` /// /// For more information, please visit . pub connection_string: Option, /// Set the table name of the sqlite service to read/write. pub table: Option, /// Set the key field name of the sqlite service to read/write. /// /// Default to `key` if not specified. pub key_field: Option, /// Set the value field name of the sqlite service to read/write. /// /// Default to `value` if not specified. pub value_field: Option, /// set the working directory, all operations will be performed under it. /// /// default: "/" pub root: Option, } impl Debug for SqliteConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("SqliteConfig"); d.field("connection_string", &self.connection_string) .field("table", &self.table) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .field("root", &self.root); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/sqlite/docs.md000064400000000000000000000017761046102023000165320ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [x] list - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `connection_string`: Set the connection string of sqlite database - `table`: Set the table of sqlite - `key_field`: Set the key field of sqlite - `value_field`: Set the value field of sqlite ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Sqlite; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Sqlite::default() .root("/") .connection_string("file//abc.db") .table("your_table") // key field type in the table should be compatible with Rust's &str like text .key_field("key") // value field type in the table should be compatible with Rust's Vec like bytea .value_field("value"); let op = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/sqlite/mod.rs000064400000000000000000000017141046102023000163750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-sqlite")] mod backend; #[cfg(feature = "services-sqlite")] pub use backend::SqliteBuilder as Sqlite; mod config; pub use config::SqliteConfig; opendal-0.52.0/src/services/supabase/backend.rs000064400000000000000000000145551046102023000175160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use log::debug; use super::core::*; use super::error::parse_error; use super::writer::*; use crate::raw::*; use crate::services::SupabaseConfig; use crate::*; impl Configurator for SupabaseConfig { type Builder = SupabaseBuilder; fn into_builder(self) -> Self::Builder { SupabaseBuilder { config: self, http_client: None, } } } /// [Supabase](https://supabase.com/) service support #[doc = include_str!("docs.md")] #[derive(Default)] pub struct SupabaseBuilder { config: SupabaseConfig, http_client: Option, } impl Debug for SupabaseBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("SupabaseBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl SupabaseBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set bucket name of this backend. pub fn bucket(mut self, bucket: &str) -> Self { self.config.bucket = bucket.to_string(); self } /// Set endpoint of this backend. /// /// Endpoint must be full uri pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.trim_end_matches('/').to_string()) }; self } /// Set the authorization key for this backend /// Do not set this key if you want to read public bucket pub fn key(mut self, key: &str) -> Self { self.config.key = Some(key.to_string()); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for SupabaseBuilder { const SCHEME: Scheme = Scheme::Supabase; type Config = SupabaseConfig; fn build(self) -> Result { let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", &root); let bucket = &self.config.bucket; let endpoint = self.config.endpoint.unwrap_or_default(); let http_client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Supabase) })? }; let key = self.config.key.as_ref().map(|k| k.to_owned()); let core = SupabaseCore::new(&root, bucket, &endpoint, key, http_client); let core = Arc::new(core); Ok(SupabaseBackend { core }) } } #[derive(Debug)] pub struct SupabaseBackend { core: Arc, } impl Access for SupabaseBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; // todo: implement Lister to support list type Lister = (); type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Supabase) .set_root(&self.core.root) .set_name(&self.core.bucket) .set_native_capability(Capability { stat: true, read: true, write: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, _args: OpStat) -> Result { // The get_object_info does not contain the file size. Therefore // we first try the get the metadata through head, if we fail, // we then use get_object_info to get the actual error info let mut resp = self.core.supabase_head_object(path).await?; match resp.status() { StatusCode::OK => parse_into_metadata(path, resp.headers()).map(RpStat::new), _ => { resp = self.core.supabase_get_object_info(path).await?; match resp.status() { StatusCode::NOT_FOUND if path.ends_with('/') => { Ok(RpStat::new(Metadata::new(EntryMode::DIR))) } _ => Err(parse_error(resp)), } } } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.supabase_get_object(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { Ok(( RpWrite::default(), oio::OneShotWriter::new(SupabaseWriter::new(self.core.clone(), path, args)), )) } } opendal-0.52.0/src/services/supabase/config.rs000064400000000000000000000034461046102023000173710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for supabase service support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct SupabaseConfig { /// The root for supabase service. pub root: Option, /// The bucket for supabase service. pub bucket: String, /// The endpoint for supabase service. pub endpoint: Option, /// The key for supabase service. pub key: Option, // TODO(1) optional public, currently true always // TODO(2) optional file_size_limit, currently 0 // TODO(3) optional allowed_mime_types, currently only string } impl Debug for SupabaseConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("SupabaseConfig") .field("root", &self.root) .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .finish_non_exhaustive() } } opendal-0.52.0/src/services/supabase/core.rs000064400000000000000000000161061046102023000170510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::HeaderValue; use http::Request; use http::Response; use crate::raw::*; use crate::*; pub struct SupabaseCore { pub root: String, pub bucket: String, pub endpoint: String, /// The key used for authorization /// If loaded, the read operation will always access the nonpublic resources. /// If you want to read the public resources, please do not set the key. pub key: Option, pub http_client: HttpClient, } impl Debug for SupabaseCore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("SupabaseCore") .field("root", &self.root) .field("bucket", &self.bucket) .field("endpoint", &self.endpoint) .finish_non_exhaustive() } } impl SupabaseCore { pub fn new( root: &str, bucket: &str, endpoint: &str, key: Option, client: HttpClient, ) -> Self { Self { root: root.to_string(), bucket: bucket.to_string(), endpoint: endpoint.to_string(), key, http_client: client, } } /// Add authorization header to the request if the key is set. Otherwise leave /// the request as-is. pub fn sign(&self, req: &mut Request) -> Result<()> { if let Some(k) = &self.key { let v = HeaderValue::from_str(&format!("Bearer {}", k)).unwrap(); req.headers_mut().insert(http::header::AUTHORIZATION, v); } Ok(()) } } // requests impl SupabaseCore { pub fn supabase_upload_object_request( &self, path: &str, size: Option, content_type: Option<&str>, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/object/{}/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); let mut req = Request::post(&url); if let Some(size) = size { req = req.header(CONTENT_LENGTH, size) } if let Some(mime) = content_type { req = req.header(CONTENT_TYPE, mime) } let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub fn supabase_get_object_public_request( &self, path: &str, _: BytesRange, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/object/public/{}/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); let req = Request::get(&url); req.body(Buffer::new()).map_err(new_request_build_error) } pub fn supabase_get_object_auth_request( &self, path: &str, _: BytesRange, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/object/authenticated/{}/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); let req = Request::get(&url); req.body(Buffer::new()).map_err(new_request_build_error) } pub fn supabase_head_object_public_request(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/object/public/{}/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); Request::head(&url) .body(Buffer::new()) .map_err(new_request_build_error) } pub fn supabase_head_object_auth_request(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/object/authenticated/{}/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); Request::head(&url) .body(Buffer::new()) .map_err(new_request_build_error) } pub fn supabase_get_object_info_public_request(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/object/info/public/{}/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error) } pub fn supabase_get_object_info_auth_request(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/storage/v1/object/info/authenticated/{}/{}", self.endpoint, self.bucket, percent_encode_path(&p) ); Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error) } } // core utils impl SupabaseCore { pub async fn send(&self, req: Request) -> Result> { self.http_client.send(req).await } pub async fn supabase_get_object( &self, path: &str, range: BytesRange, ) -> Result> { let mut req = if self.key.is_some() { self.supabase_get_object_auth_request(path, range)? } else { self.supabase_get_object_public_request(path, range)? }; self.sign(&mut req)?; self.http_client.fetch(req).await } pub async fn supabase_head_object(&self, path: &str) -> Result> { let mut req = if self.key.is_some() { self.supabase_head_object_auth_request(path)? } else { self.supabase_head_object_public_request(path)? }; self.sign(&mut req)?; self.send(req).await } pub async fn supabase_get_object_info(&self, path: &str) -> Result> { let mut req = if self.key.is_some() { self.supabase_get_object_info_auth_request(path)? } else { self.supabase_get_object_info_public_request(path)? }; self.sign(&mut req)?; self.send(req).await } } opendal-0.52.0/src/services/supabase/docs.md000064400000000000000000000022701046102023000170220ustar 00000000000000## Capabilities - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work dir for backend. - `bucket`: Set the container name for backend. - `endpoint`: Set the endpoint for backend. - `key`: Set the authorization key for the backend, do not set if you want to read public bucket ### Authorization keys There are two types of key in the Supabase, one is anon_key(Client key), another one is service_role_key(Secret key). The former one can only write public resources while the latter one can access all resources. Note that if you want to read public resources, do not set the key. ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Supabase; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Supabase::default() .root("/") .bucket("test_bucket") .endpoint("http://127.0.0.1:54321") // this sets up the anon_key, which means this operator can only write public resource .key("some_anon_key"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/supabase/error.rs000064400000000000000000000055051046102023000172530ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use serde::Deserialize; use serde_json::from_slice; use crate::raw::*; use crate::*; #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "camelCase")] /// The error returned by Supabase struct SupabaseError { status_code: String, error: String, message: String, } /// Parse the supabase error type to the OpenDAL error type pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); // Check HTTP status code first/ let (mut kind, mut retryable) = match parts.status.as_u16() { 500 | 502 | 503 | 504 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; // Than extrace the error message. let (message, _) = from_slice::(&bs) .map(|sb_err| { (kind, retryable) = parse_supabase_error(&sb_err); (format!("{sb_err:?}"), Some(sb_err)) }) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } // Return the error kind and whether it is retryable fn parse_supabase_error(err: &SupabaseError) -> (ErrorKind, bool) { let code = err.status_code.parse::().unwrap(); let status_code = StatusCode::from_u16(code).unwrap(); match status_code { StatusCode::CONFLICT => (ErrorKind::AlreadyExists, false), StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED | StatusCode::NOT_MODIFIED => { (ErrorKind::ConditionNotMatch, false) } StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), } } opendal-0.52.0/src/services/supabase/mod.rs000064400000000000000000000021521046102023000166740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-supabase")] mod core; #[cfg(feature = "services-supabase")] mod error; #[cfg(feature = "services-supabase")] mod writer; #[cfg(feature = "services-supabase")] mod backend; #[cfg(feature = "services-supabase")] pub use backend::SupabaseBuilder as Supabase; mod config; pub use config::SupabaseConfig; opendal-0.52.0/src/services/supabase/writer.rs000064400000000000000000000033341046102023000174340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct SupabaseWriter { core: Arc, op: OpWrite, path: String, } impl SupabaseWriter { pub fn new(core: Arc, path: &str, op: OpWrite) -> Self { SupabaseWriter { core, op, path: path.to_string(), } } } impl oio::OneShotWrite for SupabaseWriter { async fn write_once(&self, bs: Buffer) -> Result { let mut req = self.core.supabase_upload_object_request( &self.path, Some(bs.len()), self.op.content_type(), bs, )?; self.core.sign(&mut req)?; let resp = self.core.send(req).await?; match resp.status() { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/surrealdb/backend.rs000064400000000000000000000261151046102023000176710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use surrealdb::engine::any::Any; use surrealdb::opt::auth::Database; use surrealdb::Surreal; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::normalize_root; use crate::raw::Access; use crate::services::SurrealdbConfig; use crate::*; impl Configurator for SurrealdbConfig { type Builder = SurrealdbBuilder; fn into_builder(self) -> Self::Builder { SurrealdbBuilder { config: self } } } #[doc = include_str!("docs.md")] #[derive(Default)] pub struct SurrealdbBuilder { config: SurrealdbConfig, } impl Debug for SurrealdbBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("SurrealdbBuilder") .field("config", &self.config) .finish() } } impl SurrealdbBuilder { /// Set the connection_string of the surrealdb service. /// /// This connection string is used to connect to the surrealdb service. There are url based formats: /// /// ## Url /// /// - `ws://ip:port` /// - `wss://ip:port` /// - `http://ip:port` /// - `https://ip:port` pub fn connection_string(mut self, connection_string: &str) -> Self { if !connection_string.is_empty() { self.config.connection_string = Some(connection_string.to_string()); } self } /// set the working directory, all operations will be performed under it. /// /// default: "/" pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the table name of the surrealdb service for read/write. pub fn table(mut self, table: &str) -> Self { if !table.is_empty() { self.config.table = Some(table.to_string()); } self } /// Set the username of the surrealdb service for signin. pub fn username(mut self, username: &str) -> Self { if !username.is_empty() { self.config.username = Some(username.to_string()); } self } /// Set the password of the surrealdb service for signin. pub fn password(mut self, password: &str) -> Self { if !password.is_empty() { self.config.password = Some(password.to_string()); } self } /// Set the namespace of the surrealdb service for read/write. pub fn namespace(mut self, namespace: &str) -> Self { if !namespace.is_empty() { self.config.namespace = Some(namespace.to_string()); } self } /// Set the database of the surrealdb service for read/write. pub fn database(mut self, database: &str) -> Self { if !database.is_empty() { self.config.database = Some(database.to_string()); } self } /// Set the key field name of the surrealdb service for read/write. /// /// Default to `key` if not specified. pub fn key_field(mut self, key_field: &str) -> Self { if !key_field.is_empty() { self.config.key_field = Some(key_field.to_string()); } self } /// Set the value field name of the surrealdb service for read/write. /// /// Default to `value` if not specified. pub fn value_field(mut self, value_field: &str) -> Self { if !value_field.is_empty() { self.config.value_field = Some(value_field.to_string()); } self } } impl Builder for SurrealdbBuilder { const SCHEME: Scheme = Scheme::Surrealdb; type Config = SurrealdbConfig; fn build(self) -> Result { let connection_string = match self.config.connection_string.clone() { Some(v) => v, None => { return Err( Error::new(ErrorKind::ConfigInvalid, "connection_string is empty") .with_context("service", Scheme::Surrealdb), ) } }; let namespace = match self.config.namespace.clone() { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "namespace is empty") .with_context("service", Scheme::Surrealdb)) } }; let database = match self.config.database.clone() { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "database is empty") .with_context("service", Scheme::Surrealdb)) } }; let table = match self.config.table.clone() { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "table is empty") .with_context("service", Scheme::Surrealdb)) } }; let username = self.config.username.clone().unwrap_or_default(); let password = self.config.password.clone().unwrap_or_default(); let key_field = self .config .key_field .clone() .unwrap_or_else(|| "key".to_string()); let value_field = self .config .value_field .clone() .unwrap_or_else(|| "value".to_string()); let root = normalize_root( self.config .root .clone() .unwrap_or_else(|| "/".to_string()) .as_str(), ); Ok(SurrealdbBackend::new(Adapter { db: OnceCell::new(), connection_string, username, password, namespace, database, table, key_field, value_field, }) .with_normalized_root(root)) } } /// Backend for Surrealdb service pub type SurrealdbBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { db: OnceCell>>, connection_string: String, username: String, password: String, namespace: String, database: String, table: String, key_field: String, value_field: String, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Adapter") .field("connection_string", &self.connection_string) .field("username", &self.username) .field("password", &"") .field("namespace", &self.namespace) .field("database", &self.database) .field("table", &self.table) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .finish() } } impl Adapter { async fn get_connection(&self) -> crate::Result<&Surreal> { self.db .get_or_try_init(|| async { let namespace = self.namespace.as_str(); let database = self.database.as_str(); let db: Surreal = Surreal::init(); db.connect(self.connection_string.clone()) .await .map_err(parse_surrealdb_error)?; if !self.username.is_empty() && !self.password.is_empty() { db.signin(Database { namespace, database, username: self.username.as_str(), password: self.password.as_str(), }) .await .map_err(parse_surrealdb_error)?; } db.use_ns(namespace) .use_db(database) .await .map_err(parse_surrealdb_error)?; Ok(Arc::new(db)) }) .await .map(|v| v.as_ref()) } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Surrealdb, &self.table, Capability { read: true, write: true, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> crate::Result> { let query: String = if self.key_field == "id" { "SELECT type::field($value_field) FROM type::thing($table, $path)".to_string() } else { format!("SELECT type::field($value_field) FROM type::table($table) WHERE {} = $path LIMIT 1", self.key_field) }; let mut result = self .get_connection() .await? .query(query) .bind(("namespace", "opendal")) .bind(("path", path.to_string())) .bind(("table", self.table.to_string())) .bind(("value_field", self.value_field.to_string())) .await .map_err(parse_surrealdb_error)?; let value: Option> = result .take((0, self.value_field.as_str())) .map_err(parse_surrealdb_error)?; Ok(value.map(Buffer::from)) } async fn set(&self, path: &str, value: Buffer) -> crate::Result<()> { let query = format!( "INSERT INTO {} ({}, {}) \ VALUES ($path, $value) \ ON DUPLICATE KEY UPDATE {} = $value", self.table, self.key_field, self.value_field, self.value_field ); self.get_connection() .await? .query(query) .bind(("path", path.to_string())) .bind(("value", value.to_vec())) .await .map_err(parse_surrealdb_error)?; Ok(()) } async fn delete(&self, path: &str) -> crate::Result<()> { let query: String = if self.key_field == "id" { "DELETE FROM type::thing($table, $path)".to_string() } else { format!( "DELETE FROM type::table($table) WHERE {} = $path", self.key_field ) }; self.get_connection() .await? .query(query.as_str()) .bind(("path", path.to_string())) .bind(("table", self.table.to_string())) .await .map_err(parse_surrealdb_error)?; Ok(()) } } fn parse_surrealdb_error(err: surrealdb::Error) -> Error { Error::new(ErrorKind::Unexpected, "unhandled error from surrealdb").set_source(err) } opendal-0.52.0/src/services/surrealdb/config.rs000064400000000000000000000044331046102023000175460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Surrealdb services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct SurrealdbConfig { /// The connection string for surrealdb. pub connection_string: Option, /// The username for surrealdb. pub username: Option, /// The password for surrealdb. pub password: Option, /// The namespace for surrealdb. pub namespace: Option, /// The database for surrealdb. pub database: Option, /// The table for surrealdb. pub table: Option, /// The key field for surrealdb. pub key_field: Option, /// The value field for surrealdb. pub value_field: Option, /// The root for surrealdb. pub root: Option, } impl Debug for SurrealdbConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("SurrealdbConfig"); d.field("connection_string", &self.connection_string) .field("username", &self.username) .field("password", &"") .field("namespace", &self.namespace) .field("database", &self.database) .field("table", &self.table) .field("key_field", &self.key_field) .field("value_field", &self.value_field) .field("root", &self.root) .finish() } } opendal-0.52.0/src/services/surrealdb/docs.md000064400000000000000000000022311046102023000171770ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [ ] create_dir - [x] delete - [ ] copy - [ ] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `root`: Set the working directory of `OpenDAL` - `connection_string`: Set the connection string of surrealdb server - `username`: set the username of surrealdb - `password`: set the password of surrealdb - `namespace`: set the namespace of surrealdb - `database`: set the database of surrealdb - `table`: Set the table of surrealdb - `key_field`: Set the key field of surrealdb - `value_field`: Set the value field of surrealdb - ## Example ### Via Builder ```rust use anyhow::Result; use opendal::services::Surrealdb; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Surrealdb::default() .root("/") .connection_string("ws://127.0.0.1:8000") .username("username") .password("password") .namespace("namespace") .database("database") .table("table") .key_field("key") .value_field("value"); let op = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/surrealdb/mod.rs000064400000000000000000000017331046102023000170600ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-surrealdb")] mod backend; #[cfg(feature = "services-surrealdb")] pub use backend::SurrealdbBuilder as Surrealdb; mod config; pub use config::SurrealdbConfig; opendal-0.52.0/src/services/swift/backend.rs000064400000000000000000000216011046102023000170350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use log::debug; use super::core::*; use super::delete::SwfitDeleter; use super::error::parse_error; use super::lister::SwiftLister; use super::writer::SwiftWriter; use crate::raw::*; use crate::services::SwiftConfig; use crate::*; impl Configurator for SwiftConfig { type Builder = SwiftBuilder; fn into_builder(self) -> Self::Builder { SwiftBuilder { config: self } } } /// [OpenStack Swift](https://docs.openstack.org/api-ref/object-store/#)'s REST API support. /// For more information about swift-compatible services, refer to [Compatible Services](#compatible-services). #[doc = include_str!("docs.md")] #[doc = include_str!("compatible_services.md")] #[derive(Default, Clone)] pub struct SwiftBuilder { config: SwiftConfig, } impl Debug for SwiftBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("SwiftBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl SwiftBuilder { /// Set the remote address of this backend /// /// Endpoints should be full uri, e.g. /// /// - `http://127.0.0.1:8080/v1/AUTH_test` /// - `http://192.168.66.88:8080/swift/v1` /// - `https://openstack-controller.example.com:8080/v1/ccount` /// /// If user inputs endpoint without scheme, we will /// prepend `https://` to it. pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.trim_end_matches('/').to_string()) }; self } /// Set container of this backend. /// /// All operations will happen under this container. It is required. e.g. `snapshots` pub fn container(mut self, container: &str) -> Self { self.config.container = if container.is_empty() { None } else { Some(container.trim_end_matches('/').to_string()) }; self } /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the token of this backend. /// /// Default to empty string. pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()); } self } } impl Builder for SwiftBuilder { const SCHEME: Scheme = Scheme::Swift; type Config = SwiftConfig; /// Build a SwiftBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {}", root); let endpoint = match self.config.endpoint { Some(endpoint) => { if endpoint.starts_with("http") { endpoint } else { format!("https://{endpoint}") } } None => { return Err(Error::new( ErrorKind::ConfigInvalid, "missing endpoint for Swift", )); } }; debug!("backend use endpoint: {}", &endpoint); let container = match self.config.container { Some(container) => container, None => { return Err(Error::new( ErrorKind::ConfigInvalid, "missing container for Swift", )); } }; let token = self.config.token.unwrap_or_default(); let client = HttpClient::new()?; Ok(SwiftBackend { core: Arc::new(SwiftCore { root, endpoint, container, token, client, }), }) } } /// Backend for Swift service #[derive(Debug, Clone)] pub struct SwiftBackend { core: Arc, } impl Access for SwiftBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Swift) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, stat_has_user_metadata: true, read: true, write: true, write_can_empty: true, write_with_user_metadata: true, delete: true, list: true, list_with_recursive: true, list_has_content_length: true, list_has_content_md5: true, list_has_content_type: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, _args: OpStat) -> Result { let resp = self.core.swift_get_metadata(path).await?; match resp.status() { StatusCode::OK | StatusCode::NO_CONTENT => { let headers = resp.headers(); let mut meta = parse_into_metadata(path, headers)?; let user_meta = parse_prefixed_headers(headers, "x-object-meta-"); if !user_meta.is_empty() { meta.with_user_metadata(user_meta); } Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.swift_read(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = SwiftWriter::new(self.core.clone(), args.clone(), path.to_string()); let w = oio::OneShotWriter::new(writer); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(SwfitDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = SwiftLister::new( self.core.clone(), path.to_string(), args.recursive(), args.limit(), ); Ok((RpList::default(), oio::PageLister::new(l))) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { // cannot copy objects larger than 5 GB. // Reference: https://docs.openstack.org/api-ref/object-store/#copy-object let resp = self.core.swift_copy(from, to).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/swift/compatible_services.md000064400000000000000000000046361046102023000214550ustar 00000000000000## Compatible Services ### OpenStack Swift [OpenStack Swift](https://docs.openstack.org/swift/latest/) is the default implementations of swift services. To connect to OpenStack Swift, we need to set: - `endpoint`: The endpoint of OpenStack Swift, for example: `http://127.0.0.1:8080/v1/AUTH_test`. - `container`: The name of OpenStack Swift container. - `token`: OpenStack Swift container personal access token. ```rust,ignore builder.endpoint("http://127.0.0.1:8080/v1/AUTH_test"); builder.container("container"); builder.token("token"); ``` `endpoint` is the full URL that serves as the access point to all containers under an OpenStack Swift account. It represents the entry point for accessing the resources of the account. Alongside `endpoint`, `token` is used as a credential to verify the user's identity and authorize access to the relevant resources. Both `endpoint` and `token` can be obtained through OpenStack Swift authentication service. `endpoint` consists of server address and port, API version, authenticated account ID. For instance, it might appear as follows: - `http://127.0.0.1:8080/v1/AUTH_test`. - `http://192.168.66.88:8080/swift/v1`. - `https://openstack-controller.example.com:8080/v1/account`. Please note that the exact format of `endpoint` may vary depending on the deployment configuration and version of swift services. Users can refer to the specific services documentation for the correct `endpoint` format and authentication method. For more information, refer: - [OpenStack Swift API](https://docs.openstack.org/api-ref/object-store/). - [OpenStack Swift Authentication](https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html). ### Ceph Rados Gateway [Ceph Rados Gateway](https://docs.ceph.com/en/quincy/radosgw/) supports a RESTful API that is compatible with the basic data access model of OpenStack Swift API. To connect to Ceph Rados Gateway, we need to set: - `endpoint`: The endpoint of swift services, for example: `http://127.0.0.1:8080/swift/v1`. - `container`: The name of swift container. - `token`: swift container personal access token. ```rust,ignore builder.endpoint("http://127.0.0.1:8080/swift/v1"); builder.container("container"); builder.token("token"); ``` For more information, refer: - [Ceph Rados Gateway Swift API](https://docs.ceph.com/en/latest/radosgw/swift/#api). - [Ceph Rados Gateway Swift Authentication](https://docs.ceph.com/en/latest/radosgw/swift/auth/). opendal-0.52.0/src/services/swift/config.rs000064400000000000000000000032621046102023000167160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for OpenStack Swift support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct SwiftConfig { /// The endpoint for Swift. pub endpoint: Option, /// The container for Swift. pub container: Option, /// The root for Swift. pub root: Option, /// The token for Swift. pub token: Option, } impl Debug for SwiftConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("SwiftConfig"); ds.field("root", &self.root); ds.field("endpoint", &self.endpoint); ds.field("container", &self.container); if self.token.is_some() { ds.field("token", &""); } ds.finish() } } opendal-0.52.0/src/services/swift/core.rs000064400000000000000000000177471046102023000164160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use http::header; use http::Request; use http::Response; use serde::Deserialize; use crate::raw::*; use crate::*; pub struct SwiftCore { pub root: String, pub endpoint: String, pub container: String, pub token: String, pub client: HttpClient, } impl Debug for SwiftCore { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("SwiftCore") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("container", &self.container) .finish_non_exhaustive() } } impl SwiftCore { pub async fn swift_delete(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", &self.endpoint, &self.container, percent_encode_path(&p) ); let mut req = Request::delete(&url); req = req.header("X-Auth-Token", &self.token); let body = Buffer::new(); let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn swift_list( &self, path: &str, delimiter: &str, limit: Option, marker: &str, ) -> Result> { let p = build_abs_path(&self.root, path); // The delimiter is used to disable recursive listing. // Swift returns a 200 status code when there is no such pseudo directory in prefix. let mut url = format!( "{}/{}/?prefix={}&delimiter={}&format=json", &self.endpoint, &self.container, percent_encode_path(&p), delimiter ); if let Some(limit) = limit { url += &format!("&limit={}", limit); } if !marker.is_empty() { url += &format!("&marker={}", marker); } let mut req = Request::get(&url); req = req.header("X-Auth-Token", &self.token); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn swift_create_object( &self, path: &str, length: u64, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", &self.endpoint, &self.container, percent_encode_path(&p) ); let mut req = Request::put(&url); // Set user metadata headers. if let Some(user_metadata) = args.user_metadata() { for (k, v) in user_metadata { req = req.header(format!("X-Object-Meta-{}", k), v); } } req = req.header("X-Auth-Token", &self.token); req = req.header(header::CONTENT_LENGTH, length); let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn swift_read( &self, path: &str, range: BytesRange, _arg: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path) .trim_end_matches('/') .to_string(); let url = format!( "{}/{}/{}", &self.endpoint, &self.container, percent_encode_path(&p) ); let mut req = Request::get(&url); req = req.header("X-Auth-Token", &self.token); if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn swift_copy(&self, src_p: &str, dst_p: &str) -> Result> { // NOTE: current implementation is limited to same container and root let src_p = format!( "/{}/{}", self.container, build_abs_path(&self.root, src_p).trim_end_matches('/') ); let dst_p = build_abs_path(&self.root, dst_p) .trim_end_matches('/') .to_string(); let url = format!( "{}/{}/{}", &self.endpoint, &self.container, percent_encode_path(&dst_p) ); // Request method doesn't support for COPY, we use PUT instead. // Reference: https://docs.openstack.org/api-ref/object-store/#copy-object let mut req = Request::put(&url); req = req.header("X-Auth-Token", &self.token); req = req.header("X-Copy-From", percent_encode_path(&src_p)); // if use PUT method, we need to set the content-length to 0. req = req.header("Content-Length", "0"); let body = Buffer::new(); let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn swift_get_metadata(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "{}/{}/{}", &self.endpoint, &self.container, percent_encode_path(&p) ); let mut req = Request::head(&url); req = req.header("X-Auth-Token", &self.token); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } } #[derive(Debug, Eq, PartialEq, Deserialize)] #[serde(untagged)] pub enum ListOpResponse { Subdir { subdir: String, }, FileInfo { bytes: u64, hash: String, name: String, last_modified: String, content_type: Option, }, } #[cfg(test)] mod tests { use super::*; #[test] fn parse_list_response_test() -> Result<()> { let resp = bytes::Bytes::from( r#" [ { "subdir": "animals/" }, { "subdir": "fruit/" }, { "bytes": 147, "hash": "5e6b5b70b0426b1cc1968003e1afa5ad", "name": "test.txt", "content_type": "text/plain", "last_modified": "2023-11-01T03:00:23.147480" } ] "#, ); let mut out = serde_json::from_slice::>(&resp) .map_err(new_json_deserialize_error)?; assert_eq!(out.len(), 3); assert_eq!( out.pop().unwrap(), ListOpResponse::FileInfo { bytes: 147, hash: "5e6b5b70b0426b1cc1968003e1afa5ad".to_string(), name: "test.txt".to_string(), last_modified: "2023-11-01T03:00:23.147480".to_string(), content_type: Some("text/plain".to_string()), } ); assert_eq!( out.pop().unwrap(), ListOpResponse::Subdir { subdir: "fruit/".to_string() } ); assert_eq!( out.pop().unwrap(), ListOpResponse::Subdir { subdir: "animals/".to_string() } ); Ok(()) } } opendal-0.52.0/src/services/swift/delete.rs000064400000000000000000000027151046102023000167150ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct SwfitDeleter { core: Arc, } impl SwfitDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for SwfitDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.swift_delete(&path).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT | StatusCode::OK => Ok(()), StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/swift/docs.md000064400000000000000000000021251046102023000163520ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [ ] ~~rename~~ - [x] list - [ ] ~~presign~~ - [ ] blocking ## Configurations - `endpoint`: Set the endpoint for backend. - `container`: Swift container. - `token`: Swift personal access token. Refer to [`SwiftBuilder`]'s public API docs for more information. ## Examples ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Swift; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Create Swift backend builder let mut builder = Swift::default() // Set the root for swift, all operations will happen under this root .root("/path/to/dir") // set the endpoint of Swift backend .endpoint("https://openstack-controller.example.com:8080/v1/account") // set the container name of Swift workspace .container("container") // set the auth token for builder .token("token"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/swift/error.rs000064400000000000000000000050251046102023000166010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use bytes::Bytes; use http::Response; use http::StatusCode; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; #[allow(dead_code)] #[derive(Debug, Deserialize)] struct ErrorResponse { h1: String, p: String, } pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::PRECONDITION_FAILED => (ErrorKind::ConditionNotMatch, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = parse_error_response(&bs); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } fn parse_error_response(resp: &Bytes) -> String { match de::from_reader::<_, ErrorResponse>(resp.clone().reader()) { Ok(swift_err) => swift_err.p, Err(_) => String::from_utf8_lossy(resp).into_owned(), } } #[cfg(test)] mod tests { use super::*; #[test] fn parse_error_response_test() -> Result<()> { let resp = Bytes::from( r#"

Not Found

The resource could not be found.

"#, ); let msg = parse_error_response(&resp); assert_eq!(msg, "The resource could not be found.".to_string(),); Ok(()) } } opendal-0.52.0/src/services/swift/lister.rs000064400000000000000000000101721046102023000167510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct SwiftLister { core: Arc, path: String, delimiter: &'static str, limit: Option, } impl SwiftLister { pub fn new(core: Arc, path: String, recursive: bool, limit: Option) -> Self { let delimiter = if recursive { "" } else { "/" }; Self { core, path, delimiter, limit, } } } impl oio::PageList for SwiftLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let response = self .core .swift_list(&self.path, self.delimiter, self.limit, &ctx.token) .await?; let status_code = response.status(); if !status_code.is_success() { let error = parse_error(response); return Err(error); } let bytes = response.into_body(); let decoded_response: Vec = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; // Update token and done based on resp. if let Some(entry) = decoded_response.last() { let path = match entry { ListOpResponse::Subdir { subdir } => subdir, ListOpResponse::FileInfo { name, .. } => name, }; ctx.token.clone_from(path); } else { ctx.done = true; } for status in decoded_response { let entry: oio::Entry = match status { ListOpResponse::Subdir { subdir } => { let mut path = build_rel_path(self.core.root.as_str(), subdir.as_str()); if path.is_empty() { path = "/".to_string(); } let meta = Metadata::new(EntryMode::DIR); oio::Entry::with(path, meta) } ListOpResponse::FileInfo { bytes, hash, name, content_type, mut last_modified, } => { let mut path = build_rel_path(self.core.root.as_str(), name.as_str()); if path.is_empty() { path = "/".to_string(); } let mut meta = Metadata::new(EntryMode::from_path(path.as_str())); meta.set_content_length(bytes); meta.set_content_md5(hash.as_str()); // OpenStack Swift returns time without 'Z' at the end, // which causes an error in parse_datetime_from_rfc3339. // we'll change "2023-10-28T19:18:11.682610" to "2023-10-28T19:18:11.682610Z". if !last_modified.ends_with('Z') { last_modified.push('Z'); } meta.set_last_modified(parse_datetime_from_rfc3339(last_modified.as_str())?); if let Some(content_type) = content_type { meta.set_content_type(content_type.as_str()); } oio::Entry::with(path, meta) } }; ctx.entries.push_back(entry); } Ok(()) } } opendal-0.52.0/src/services/swift/mod.rs000064400000000000000000000022601046102023000162250ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-swift")] mod core; #[cfg(feature = "services-swift")] mod delete; #[cfg(feature = "services-swift")] mod error; #[cfg(feature = "services-swift")] mod lister; #[cfg(feature = "services-swift")] mod writer; #[cfg(feature = "services-swift")] mod backend; #[cfg(feature = "services-swift")] pub use backend::SwiftBuilder as Swift; mod config; pub use config::SwiftConfig; opendal-0.52.0/src/services/swift/writer.rs000064400000000000000000000031131046102023000167600ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::SwiftCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct SwiftWriter { core: Arc, op: OpWrite, path: String, } impl SwiftWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { SwiftWriter { core, op, path } } } impl oio::OneShotWrite for SwiftWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self .core .swift_create_object(&self.path, bs.len() as u64, &self.op, bs) .await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/tikv/backend.rs000064400000000000000000000156221046102023000166640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use tikv_client::Config; use tikv_client::RawClient; use tokio::sync::OnceCell; use crate::raw::adapters::kv; use crate::raw::Access; use crate::services::TikvConfig; use crate::Builder; use crate::Capability; use crate::Error; use crate::ErrorKind; use crate::Scheme; use crate::*; impl Configurator for TikvConfig { type Builder = TikvBuilder; fn into_builder(self) -> Self::Builder { TikvBuilder { config: self } } } /// TiKV backend builder #[doc = include_str!("docs.md")] #[derive(Clone, Default)] pub struct TikvBuilder { config: TikvConfig, } impl Debug for TikvBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("TikvBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl TikvBuilder { /// Set the network address of the TiKV service. pub fn endpoints(mut self, endpoints: Vec) -> Self { if !endpoints.is_empty() { self.config.endpoints = Some(endpoints) } self } /// Set the insecure connection to TiKV. pub fn insecure(mut self) -> Self { self.config.insecure = true; self } /// Set the certificate authority file path. pub fn ca_path(mut self, ca_path: &str) -> Self { if !ca_path.is_empty() { self.config.ca_path = Some(ca_path.to_string()) } self } /// Set the certificate file path. pub fn cert_path(mut self, cert_path: &str) -> Self { if !cert_path.is_empty() { self.config.cert_path = Some(cert_path.to_string()) } self } /// Set the key file path. pub fn key_path(mut self, key_path: &str) -> Self { if !key_path.is_empty() { self.config.key_path = Some(key_path.to_string()) } self } } impl Builder for TikvBuilder { const SCHEME: Scheme = Scheme::Tikv; type Config = TikvConfig; fn build(self) -> Result { let endpoints = self.config.endpoints.ok_or_else(|| { Error::new( ErrorKind::ConfigInvalid, "endpoints is required but not set", ) .with_context("service", Scheme::Tikv) })?; if self.config.insecure && (self.config.ca_path.is_some() || self.config.key_path.is_some() || self.config.cert_path.is_some()) { return Err( Error::new(ErrorKind::ConfigInvalid, "invalid tls configuration") .with_context("service", Scheme::Tikv) .with_context("endpoints", format!("{:?}", endpoints)), )?; } Ok(TikvBackend::new(Adapter { client: OnceCell::new(), endpoints, insecure: self.config.insecure, ca_path: self.config.ca_path.clone(), cert_path: self.config.cert_path.clone(), key_path: self.config.key_path.clone(), })) } } /// Backend for TiKV service pub type TikvBackend = kv::Backend; #[derive(Clone)] pub struct Adapter { client: OnceCell, endpoints: Vec, insecure: bool, ca_path: Option, cert_path: Option, key_path: Option, } impl Debug for Adapter { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Adapter"); ds.field("endpoints", &self.endpoints); ds.finish() } } impl Adapter { async fn get_connection(&self) -> Result { if let Some(client) = self.client.get() { return Ok(client.clone()); } let client = if self.insecure { RawClient::new(self.endpoints.clone()) .await .map_err(parse_tikv_config_error)? } else if self.ca_path.is_some() && self.key_path.is_some() && self.cert_path.is_some() { let (ca_path, key_path, cert_path) = ( self.ca_path.clone().unwrap(), self.key_path.clone().unwrap(), self.cert_path.clone().unwrap(), ); let config = Config::default().with_security(ca_path, cert_path, key_path); RawClient::new_with_config(self.endpoints.clone(), config) .await .map_err(parse_tikv_config_error)? } else { return Err( Error::new(ErrorKind::ConfigInvalid, "invalid configuration") .with_context("service", Scheme::Tikv) .with_context("endpoints", format!("{:?}", self.endpoints)), ); }; self.client.set(client.clone()).ok(); Ok(client) } } impl kv::Adapter for Adapter { type Scanner = (); fn info(&self) -> kv::Info { kv::Info::new( Scheme::Tikv, "TiKV", Capability { read: true, write: true, blocking: false, shared: true, ..Default::default() }, ) } async fn get(&self, path: &str) -> Result> { let result = self .get_connection() .await? .get(path.to_owned()) .await .map_err(parse_tikv_error)?; Ok(result.map(Buffer::from)) } async fn set(&self, path: &str, value: Buffer) -> Result<()> { self.get_connection() .await? .put(path.to_owned(), value.to_vec()) .await .map_err(parse_tikv_error) } async fn delete(&self, path: &str) -> Result<()> { self.get_connection() .await? .delete(path.to_owned()) .await .map_err(parse_tikv_error) } } fn parse_tikv_error(e: tikv_client::Error) -> Error { Error::new(ErrorKind::Unexpected, "error from tikv").set_source(e) } fn parse_tikv_config_error(e: tikv_client::Error) -> Error { Error::new(ErrorKind::ConfigInvalid, "invalid configuration") .with_context("service", Scheme::Tikv) .set_source(e) } opendal-0.52.0/src/services/tikv/config.rs000064400000000000000000000034141046102023000165360ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Tikv services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct TikvConfig { /// network address of the TiKV service. pub endpoints: Option>, /// whether using insecure connection to TiKV pub insecure: bool, /// certificate authority file path pub ca_path: Option, /// cert path pub cert_path: Option, /// key path pub key_path: Option, } impl Debug for TikvConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("TikvConfig"); d.field("endpoints", &self.endpoints) .field("insecure", &self.insecure) .field("ca_path", &self.ca_path) .field("cert_path", &self.cert_path) .field("key_path", &self.key_path) .finish() } } opendal-0.52.0/src/services/tikv/docs.md000064400000000000000000000015511046102023000161750ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] ~~blocking~~ ## Configuration - `endpoints`: Set the endpoints to the tikv cluster - `insecure`: Set the insecure flag to the tikv cluster - `ca_path`: Set the ca path to the tikv connection - `cert_path`: Set the cert path to the tikv connection - `key_path`: Set the key path to the tikv connection You can refer to [`TikvBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Tikv; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Tikv::default() .endpoints(vec!["127.0.0.1:2379".to_string()]); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/tikv/mod.rs000064400000000000000000000017021046102023000160460ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-tikv")] mod backend; #[cfg(feature = "services-tikv")] pub use backend::TikvBuilder as Tikv; mod config; pub use config::TikvConfig; opendal-0.52.0/src/services/upyun/backend.rs000064400000000000000000000227741046102023000170750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use http::Response; use http::StatusCode; use log::debug; use super::core::*; use super::delete::UpyunDeleter; use super::error::parse_error; use super::lister::UpyunLister; use super::writer::UpyunWriter; use super::writer::UpyunWriters; use crate::raw::*; use crate::services::UpyunConfig; use crate::*; impl Configurator for UpyunConfig { type Builder = UpyunBuilder; fn into_builder(self) -> Self::Builder { UpyunBuilder { config: self, http_client: None, } } } /// [upyun](https://www.upyun.com/products/file-storage) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct UpyunBuilder { config: UpyunConfig, http_client: Option, } impl Debug for UpyunBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("UpyunBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl UpyunBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// bucket of this backend. /// /// It is required. e.g. `test` pub fn bucket(mut self, bucket: &str) -> Self { self.config.bucket = bucket.to_string(); self } /// operator of this backend. /// /// It is required. e.g. `test` pub fn operator(mut self, operator: &str) -> Self { self.config.operator = if operator.is_empty() { None } else { Some(operator.to_string()) }; self } /// password of this backend. /// /// It is required. e.g. `asecret` pub fn password(mut self, password: &str) -> Self { self.config.password = if password.is_empty() { None } else { Some(password.to_string()) }; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for UpyunBuilder { const SCHEME: Scheme = Scheme::Upyun; type Config = UpyunConfig; /// Builds the backend and returns the result of UpyunBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle bucket. if self.config.bucket.is_empty() { return Err(Error::new(ErrorKind::ConfigInvalid, "bucket is empty") .with_operation("Builder::build") .with_context("service", Scheme::Upyun)); } debug!("backend use bucket {}", &self.config.bucket); let operator = match &self.config.operator { Some(operator) => Ok(operator.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "operator is empty") .with_operation("Builder::build") .with_context("service", Scheme::Upyun)), }?; let password = match &self.config.password { Some(password) => Ok(password.clone()), None => Err(Error::new(ErrorKind::ConfigInvalid, "password is empty") .with_operation("Builder::build") .with_context("service", Scheme::Upyun)), }?; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Upyun) })? }; let signer = UpyunSigner { operator: operator.clone(), password: password.clone(), }; Ok(UpyunBackend { core: Arc::new(UpyunCore { root, operator, bucket: self.config.bucket.clone(), signer, client, }), }) } } /// Backend for upyun services. #[derive(Debug, Clone)] pub struct UpyunBackend { core: Arc, } impl Access for UpyunBackend { type Reader = HttpBody; type Writer = UpyunWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Upyun) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_md5: true, stat_has_cache_control: true, stat_has_content_disposition: true, create_dir: true, read: true, write: true, write_can_empty: true, write_can_multi: true, write_with_cache_control: true, write_with_content_type: true, // https://help.upyun.com/knowledge-base/rest_api/#e5b9b6e8a18ce5bc8fe696ade782b9e7bbade4bca0 write_multi_min_size: Some(1024 * 1024), write_multi_max_size: Some(50 * 1024 * 1024), delete: true, rename: true, copy: true, list: true, list_with_limit: true, list_has_content_length: true, list_has_content_type: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let resp = self.core.create_dir(path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCreateDir::default()), _ => Err(parse_error(resp)), } } async fn stat(&self, path: &str, _args: OpStat) -> Result { let resp = self.core.info(path).await?; let status = resp.status(); match status { StatusCode::OK => parse_info(resp.headers()).map(RpStat::new), _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.download_file(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let concurrent = args.concurrent(); let executor = args.executor().cloned(); let writer = UpyunWriter::new(self.core.clone(), args, path.to_string()); let w = oio::MultipartWriter::new(writer, executor, concurrent); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(UpyunDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = UpyunLister::new(self.core.clone(), path, args.limit()); Ok((RpList::default(), oio::PageLister::new(l))) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.copy(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let resp = self.core.move_object(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpRename::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/upyun/config.rs000064400000000000000000000032171046102023000167420ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for upyun services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct UpyunConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, /// bucket address of this backend. pub bucket: String, /// username of this backend. pub operator: Option, /// password of this backend. pub password: Option, } impl Debug for UpyunConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Config"); ds.field("root", &self.root); ds.field("bucket", &self.bucket); ds.field("operator", &self.operator); ds.finish() } } opendal-0.52.0/src/services/upyun/core.rs000064400000000000000000000332561046102023000164330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use base64::Engine; use hmac::Hmac; use hmac::Mac; use http::header; use http::HeaderMap; use http::Request; use http::Response; use md5::Digest; use serde::Deserialize; use sha1::Sha1; use self::constants::*; use crate::raw::*; use crate::*; pub(super) mod constants { pub const X_UPYUN_FILE_TYPE: &str = "x-upyun-file-type"; pub const X_UPYUN_FILE_SIZE: &str = "x-upyun-file-size"; pub const X_UPYUN_CACHE_CONTROL: &str = "x-upyun-meta-cache-control"; pub const X_UPYUN_CONTENT_DISPOSITION: &str = "x-upyun-meta-content-disposition"; pub const X_UPYUN_MULTI_STAGE: &str = "X-Upyun-Multi-Stage"; pub const X_UPYUN_MULTI_TYPE: &str = "X-Upyun-Multi-Type"; pub const X_UPYUN_MULTI_DISORDER: &str = "X-Upyun-Multi-Disorder"; pub const X_UPYUN_MULTI_UUID: &str = "X-Upyun-Multi-Uuid"; pub const X_UPYUN_PART_ID: &str = "X-Upyun-Part-Id"; pub const X_UPYUN_FOLDER: &str = "x-upyun-folder"; pub const X_UPYUN_MOVE_SOURCE: &str = "X-Upyun-Move-Source"; pub const X_UPYUN_COPY_SOURCE: &str = "X-Upyun-Copy-Source"; pub const X_UPYUN_METADATA_DIRECTIVE: &str = "X-Upyun-Metadata-Directive"; pub const X_UPYUN_LIST_ITER: &str = "x-list-iter"; pub const X_UPYUN_LIST_LIMIT: &str = "X-List-Limit"; pub const X_UPYUN_LIST_MAX_LIMIT: usize = 4096; pub const X_UPYUN_LIST_DEFAULT_LIMIT: usize = 256; } #[derive(Clone)] pub struct UpyunCore { /// The root of this core. pub root: String, /// The endpoint of this backend. pub operator: String, /// The bucket of this backend. pub bucket: String, /// signer of this backend. pub signer: UpyunSigner, pub client: HttpClient, } impl Debug for UpyunCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .field("bucket", &self.bucket) .field("operator", &self.operator) .finish_non_exhaustive() } } impl UpyunCore { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } pub fn sign(&self, req: &mut Request) -> Result<()> { // get rfc1123 date let date = chrono::Utc::now() .format("%a, %d %b %Y %H:%M:%S GMT") .to_string(); let authorization = self.signer .authorization(&date, req.method().as_str(), req.uri().path()); req.headers_mut() .insert("Authorization", authorization.parse().unwrap()); req.headers_mut().insert("Date", date.parse().unwrap()); Ok(()) } } impl UpyunCore { pub async fn download_file(&self, path: &str, range: BytesRange) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&path) ); let req = Request::get(url); let mut req = req .header(header::RANGE, range.to_header()) .body(Buffer::new()) .map_err(new_request_build_error)?; self.sign(&mut req)?; self.client.fetch(req).await } pub async fn info(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&path) ); let req = Request::head(url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req)?; self.send(req).await } pub fn upload( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&p) ); let mut req = Request::put(&url); if let Some(size) = size { req = req.header(header::CONTENT_LENGTH, size.to_string()) } if let Some(mime) = args.content_type() { req = req.header(header::CONTENT_TYPE, mime) } if let Some(pos) = args.content_disposition() { req = req.header(X_UPYUN_CONTENT_DISPOSITION, pos) } if let Some(cache_control) = args.cache_control() { req = req.header(X_UPYUN_CACHE_CONTROL, cache_control) } // Set body let mut req = req.body(body).map_err(new_request_build_error)?; self.sign(&mut req)?; Ok(req) } pub async fn delete(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&path) ); let req = Request::delete(url); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req)?; self.send(req).await } pub async fn copy(&self, from: &str, to: &str) -> Result> { let from = format!("/{}/{}", self.bucket, build_abs_path(&self.root, from)); let to = build_abs_path(&self.root, to); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&to) ); let mut req = Request::put(url); req = req.header(header::CONTENT_LENGTH, "0"); req = req.header(X_UPYUN_COPY_SOURCE, from); req = req.header(X_UPYUN_METADATA_DIRECTIVE, "copy"); // Set body let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req)?; self.send(req).await } pub async fn move_object(&self, from: &str, to: &str) -> Result> { let from = format!("/{}/{}", self.bucket, build_abs_path(&self.root, from)); let to = build_abs_path(&self.root, to); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&to) ); let mut req = Request::put(url); req = req.header(header::CONTENT_LENGTH, "0"); req = req.header(X_UPYUN_MOVE_SOURCE, from); req = req.header(X_UPYUN_METADATA_DIRECTIVE, "copy"); // Set body let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req)?; self.send(req).await } pub async fn create_dir(&self, path: &str) -> Result> { let path = build_abs_path(&self.root, path); let path = path[..path.len() - 1].to_string(); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&path) ); let mut req = Request::post(url); req = req.header("folder", "true"); req = req.header(X_UPYUN_FOLDER, "true"); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req)?; self.send(req).await } pub async fn initiate_multipart_upload( &self, path: &str, args: &OpWrite, ) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&path) ); let mut req = Request::put(url); req = req.header(X_UPYUN_MULTI_STAGE, "initiate"); req = req.header(X_UPYUN_MULTI_DISORDER, "true"); if let Some(content_type) = args.content_type() { req = req.header(X_UPYUN_MULTI_TYPE, content_type); } if let Some(content_disposition) = args.content_disposition() { req = req.header(X_UPYUN_CONTENT_DISPOSITION, content_disposition) } if let Some(cache_control) = args.cache_control() { req = req.header(X_UPYUN_CACHE_CONTROL, cache_control) } let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req)?; self.send(req).await } pub fn upload_part( &self, path: &str, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&p), ); let mut req = Request::put(&url); req = req.header(header::CONTENT_LENGTH, size); req = req.header(X_UPYUN_MULTI_STAGE, "upload"); req = req.header(X_UPYUN_MULTI_UUID, upload_id); req = req.header(X_UPYUN_PART_ID, part_number); // Set body let mut req = req.body(body).map_err(new_request_build_error)?; self.sign(&mut req)?; Ok(req) } pub async fn complete_multipart_upload( &self, path: &str, upload_id: &str, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&p), ); let mut req = Request::put(url); req = req.header(X_UPYUN_MULTI_STAGE, "complete"); req = req.header(X_UPYUN_MULTI_UUID, upload_id); let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req)?; self.send(req).await } pub async fn list_objects( &self, path: &str, iter: &str, limit: Option, ) -> Result> { let path = build_abs_path(&self.root, path); let url = format!( "https://v0.api.upyun.com/{}/{}", self.bucket, percent_encode_path(&path), ); let mut req = Request::get(url.clone()); req = req.header(header::ACCEPT, "application/json"); if !iter.is_empty() { req = req.header(X_UPYUN_LIST_ITER, iter); } if let Some(mut limit) = limit { if limit > X_UPYUN_LIST_MAX_LIMIT { limit = X_UPYUN_LIST_DEFAULT_LIMIT; } req = req.header(X_UPYUN_LIST_LIMIT, limit); } // Set body let mut req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.sign(&mut req)?; self.send(req).await } } #[derive(Clone, Default)] pub struct UpyunSigner { pub operator: String, pub password: String, } type HmacSha1 = Hmac; impl UpyunSigner { pub fn authorization(&self, date: &str, method: &str, uri: &str) -> String { let sign = vec![method, uri, date]; let sign = sign .into_iter() .filter(|s| !s.is_empty()) .collect::>() .join("&"); let mut mac = HmacSha1::new_from_slice(format_md5(self.password.as_bytes()).as_bytes()) .expect("HMAC can take key of any size"); mac.update(sign.as_bytes()); let sign_str = mac.finalize().into_bytes(); let sign = base64::engine::general_purpose::STANDARD.encode(sign_str.as_slice()); format!("UPYUN {}:{}", self.operator, sign) } } pub(super) fn parse_info(headers: &HeaderMap) -> Result { let mode = if parse_header_to_str(headers, X_UPYUN_FILE_TYPE)? == Some("file") { EntryMode::FILE } else { EntryMode::DIR }; let mut m = Metadata::new(mode); if let Some(v) = parse_header_to_str(headers, X_UPYUN_FILE_SIZE)? { let size = v.parse::().map_err(|e| { Error::new(ErrorKind::Unexpected, "header value is not valid integer") .with_operation("parse_info") .set_source(e) })?; m.set_content_length(size); } if let Some(v) = parse_content_type(headers)? { m.set_content_type(v); } if let Some(v) = parse_content_md5(headers)? { m.set_content_md5(v); } if let Some(v) = parse_header_to_str(headers, X_UPYUN_CACHE_CONTROL)? { m.set_cache_control(v); } if let Some(v) = parse_header_to_str(headers, X_UPYUN_CONTENT_DISPOSITION)? { m.set_content_disposition(v); } Ok(m) } pub fn format_md5(bs: &[u8]) -> String { let mut hasher = md5::Md5::new(); hasher.update(bs); format!("{:x}", hasher.finalize()) } #[derive(Debug, Deserialize)] pub(super) struct File { #[serde(rename = "type")] pub type_field: String, pub name: String, pub length: u64, pub last_modified: i64, } #[derive(Debug, Deserialize)] pub(super) struct ListObjectsResponse { pub iter: String, pub files: Vec, } opendal-0.52.0/src/services/upyun/delete.rs000064400000000000000000000027531046102023000167430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct UpyunDeleter { core: Arc, } impl UpyunDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for UpyunDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.delete(&path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), // Allow 404 when deleting a non-existing object StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/upyun/docs.md000064400000000000000000000017151046102023000164020ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `bucket`: Upyun bucket name - `operator` Upyun operator - `password` Upyun password You can refer to [`UpyunBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Upyun; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Upyun::default() // set the storage bucket for OpenDAL .root("/") // set the bucket for OpenDAL .bucket("test") // set the operator for OpenDAL .operator("xxxxxxxxxx") // set the password name for OpenDAL .password("opendal"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/upyun/error.rs000064400000000000000000000060651046102023000166320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// UpyunError is the error returned by upyun service. #[derive(Default, Debug, Deserialize)] #[serde(default, rename_all = "PascalCase")] struct UpyunError { code: i64, msg: String, id: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status.as_u16() { 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), 304 | 412 => (ErrorKind::ConditionNotMatch, false), // Service like Upyun could return 499 error with a message like: // Client Disconnect, we should retry it. 499 => (ErrorKind::Unexpected, true), 500 | 502 | 503 | 504 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, _upyun_err) = de::from_reader::<_, UpyunError>(bs.clone().reader()) .map(|upyun_err| (format!("{upyun_err:?}"), Some(upyun_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[tokio::test] async fn test_parse_error() { let err_res = vec![ ( r#"{"code": 40100016, "msg": "invalid date value in header", "id": "f5b30c720ddcecc70abd2f5c1c64bde8"}"#, ErrorKind::Unexpected, StatusCode::UNAUTHORIZED, ), ( r#"{"code": 40300010, "msg": "file type error", "id": "f5b30c720ddcecc70abd2f5c1c64bde7"}"#, ErrorKind::PermissionDenied, StatusCode::FORBIDDEN, ), ]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/upyun/lister.rs000064400000000000000000000061421046102023000167770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use super::core::ListObjectsResponse; use super::core::UpyunCore; use super::error::parse_error; use crate::raw::oio::Entry; use crate::raw::*; use crate::EntryMode; use crate::Metadata; use crate::Result; pub struct UpyunLister { core: Arc, path: String, limit: Option, } impl UpyunLister { pub(super) fn new(core: Arc, path: &str, limit: Option) -> Self { UpyunLister { core, path: path.to_string(), limit, } } } impl oio::PageList for UpyunLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self .core .list_objects(&self.path, &ctx.token, self.limit) .await?; if resp.status() == http::StatusCode::NOT_FOUND { ctx.done = true; return Ok(()); } match resp.status() { http::StatusCode::OK => {} http::StatusCode::NOT_FOUND => { ctx.done = true; return Ok(()); } _ => { return Err(parse_error(resp)); } } let bs = resp.into_body(); let response: ListObjectsResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; // ref https://help.upyun.com/knowledge-base/rest_api/#e88eb7e58f96e79baee5bd95e69687e4bbb6e58897e8a1a8 // when iter is "g2gCZAAEbmV4dGQAA2VvZg", it means the list is done. ctx.done = response.iter == "g2gCZAAEbmV4dGQAA2VvZg"; ctx.token = response.iter; for file in response.files { let path = build_abs_path(&normalize_root(&self.path), &file.name); let entry = if file.type_field == "folder" { let path = format!("{}/", path); Entry::new(&path, Metadata::new(EntryMode::DIR)) } else { let m = Metadata::new(EntryMode::FILE) .with_content_length(file.length) .with_content_type(file.type_field) .with_last_modified(parse_datetime_from_from_timestamp(file.last_modified)?); Entry::new(&path, m) }; ctx.entries.push_back(entry); } Ok(()) } } opendal-0.52.0/src/services/upyun/mod.rs000064400000000000000000000022601046102023000162510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-upyun")] mod core; #[cfg(feature = "services-upyun")] mod delete; #[cfg(feature = "services-upyun")] mod error; #[cfg(feature = "services-upyun")] mod lister; #[cfg(feature = "services-upyun")] mod writer; #[cfg(feature = "services-upyun")] mod backend; #[cfg(feature = "services-upyun")] pub use backend::UpyunBuilder as Upyun; mod config; pub use config::UpyunConfig; opendal-0.52.0/src/services/upyun/writer.rs000064400000000000000000000071061046102023000170120ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::constants::X_UPYUN_MULTI_UUID; use super::core::UpyunCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type UpyunWriters = oio::MultipartWriter; pub struct UpyunWriter { core: Arc, op: OpWrite, path: String, } impl UpyunWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { UpyunWriter { core, op, path } } } impl oio::MultipartWrite for UpyunWriter { async fn write_once(&self, size: u64, body: Buffer) -> Result { let req = self.core.upload(&self.path, Some(size), &self.op, body)?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn initiate_part(&self) -> Result { let resp = self .core .initiate_multipart_upload(&self.path, &self.op) .await?; let status = resp.status(); match status { StatusCode::NO_CONTENT => { let id = parse_header_to_str(resp.headers(), X_UPYUN_MULTI_UUID)?.ok_or(Error::new( ErrorKind::Unexpected, format!("{} header is missing", X_UPYUN_MULTI_UUID), ))?; Ok(id.to_string()) } _ => Err(parse_error(resp)), } } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result { let req = self .core .upload_part(&self.path, upload_id, part_number, size, body)?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT | StatusCode::CREATED => Ok(oio::MultipartPart { part_number, etag: "".to_string(), checksum: None, }), _ => Err(parse_error(resp)), } } async fn complete_part( &self, upload_id: &str, _parts: &[oio::MultipartPart], ) -> Result { let resp = self .core .complete_multipart_upload(&self.path, upload_id) .await?; let status = resp.status(); match status { StatusCode::NO_CONTENT => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn abort_part(&self, _upload_id: &str) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "Upyun does not support abort multipart upload", )) } } opendal-0.52.0/src/services/vercel_artifacts/backend.rs000064400000000000000000000133441046102023000212260ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::sync::Arc; use http::header; use http::Request; use http::Response; use http::StatusCode; use super::error::parse_error; use super::writer::VercelArtifactsWriter; use crate::raw::*; use crate::*; #[doc = include_str!("docs.md")] #[derive(Clone)] pub struct VercelArtifactsBackend { pub(crate) access_token: String, pub(crate) client: HttpClient, } impl Debug for VercelArtifactsBackend { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { let mut de = f.debug_struct("VercelArtifactsBackend"); de.field("access_token", &self.access_token); de.finish() } } impl Access for VercelArtifactsBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; type Lister = (); type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut ma = AccessorInfo::default(); ma.set_scheme(Scheme::VercelArtifacts) .set_native_capability(Capability { stat: true, stat_has_cache_control: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_content_encoding: true, stat_has_content_range: true, stat_has_etag: true, stat_has_content_md5: true, stat_has_last_modified: true, stat_has_content_disposition: true, read: true, write: true, shared: true, ..Default::default() }); ma.into() } async fn stat(&self, path: &str, _args: OpStat) -> Result { let res = self.vercel_artifacts_stat(path).await?; let status = res.status(); match status { StatusCode::OK => { let meta = parse_into_metadata(path, res.headers())?; Ok(RpStat::new(meta)) } _ => Err(parse_error(res)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.vercel_artifacts_get(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { Ok(( RpWrite::default(), oio::OneShotWriter::new(VercelArtifactsWriter::new( self.clone(), args, path.to_string(), )), )) } } impl VercelArtifactsBackend { pub async fn vercel_artifacts_get( &self, hash: &str, range: BytesRange, _: &OpRead, ) -> Result> { let url: String = format!( "https://api.vercel.com/v8/artifacts/{}", percent_encode_path(hash) ); let mut req = Request::get(&url); if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn vercel_artifacts_put( &self, hash: &str, size: u64, body: Buffer, ) -> Result> { let url = format!( "https://api.vercel.com/v8/artifacts/{}", percent_encode_path(hash) ); let mut req = Request::put(&url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::CONTENT_TYPE, "application/octet-stream"); req = req.header(header::AUTHORIZATION, auth_header_content); req = req.header(header::CONTENT_LENGTH, size); let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn vercel_artifacts_stat(&self, hash: &str) -> Result> { let url = format!( "https://api.vercel.com/v8/artifacts/{}", percent_encode_path(hash) ); let mut req = Request::head(&url); let auth_header_content = format!("Bearer {}", self.access_token); req = req.header(header::AUTHORIZATION, auth_header_content); req = req.header(header::CONTENT_LENGTH, 0); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } } opendal-0.52.0/src/services/vercel_artifacts/builder.rs000064400000000000000000000061311046102023000212610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use super::backend::VercelArtifactsBackend; use crate::raw::Access; use crate::raw::HttpClient; use crate::services::VercelArtifactsConfig; use crate::Scheme; use crate::*; impl Configurator for VercelArtifactsConfig { type Builder = VercelArtifactsBuilder; fn into_builder(self) -> Self::Builder { VercelArtifactsBuilder { config: self, http_client: None, } } } /// [Vercel Cache](https://vercel.com/docs/concepts/monorepos/remote-caching) backend support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct VercelArtifactsBuilder { config: VercelArtifactsConfig, http_client: Option, } impl Debug for VercelArtifactsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("VercelArtifactsBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl VercelArtifactsBuilder { /// set the bearer access token for Vercel /// /// default: no access token, which leads to failure pub fn access_token(mut self, access_token: &str) -> Self { self.config.access_token = Some(access_token.to_string()); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, http_client: HttpClient) -> Self { self.http_client = Some(http_client); self } } impl Builder for VercelArtifactsBuilder { const SCHEME: Scheme = Scheme::VercelArtifacts; type Config = VercelArtifactsConfig; fn build(self) -> Result { let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::VercelArtifacts) })? }; match self.config.access_token.clone() { Some(access_token) => Ok(VercelArtifactsBackend { access_token, client, }), None => Err(Error::new(ErrorKind::ConfigInvalid, "access_token not set")), } } } opendal-0.52.0/src/services/vercel_artifacts/config.rs000064400000000000000000000025411046102023000211010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for Vercel Cache support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct VercelArtifactsConfig { /// The access token for Vercel. pub access_token: Option, } impl Debug for VercelArtifactsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("VercelArtifactsConfig") .field("access_token", &"") .finish() } } opendal-0.52.0/src/services/vercel_artifacts/docs.md000064400000000000000000000012621046102023000205370ustar 00000000000000## Capabilities This service can be used to: - [ ] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] ~~copy~~ - [ ] ~~rename~~ - [ ] ~~list~~ - [ ] ~~presign~~ - [ ] blocking ## Configuration - `access_token`: set the access_token for Rest API You can refer to [`VercelArtifactsBuilder`]'s docs for more information ## Example ### Via Builder ```no_run use anyhow::Result; use opendal::services::VercelArtifacts; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = VercelArtifacts::default() .access_token("xxx"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/vercel_artifacts/error.rs000064400000000000000000000032371046102023000207700ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use crate::raw::*; use crate::*; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = String::from_utf8_lossy(&bs); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/vercel_artifacts/mod.rs000064400000000000000000000022521046102023000204120ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-vercel-artifacts")] mod backend; #[cfg(feature = "services-vercel-artifacts")] mod error; #[cfg(feature = "services-vercel-artifacts")] mod writer; #[cfg(feature = "services-vercel-artifacts")] mod builder; #[cfg(feature = "services-vercel-artifacts")] pub use builder::VercelArtifactsBuilder as VercelArtifacts; mod config; pub use config::VercelArtifactsConfig; opendal-0.52.0/src/services/vercel_artifacts/writer.rs000064400000000000000000000032761046102023000211560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::StatusCode; use super::backend::VercelArtifactsBackend; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct VercelArtifactsWriter { backend: VercelArtifactsBackend, _op: OpWrite, path: String, } impl VercelArtifactsWriter { pub fn new(backend: VercelArtifactsBackend, op: OpWrite, path: String) -> Self { VercelArtifactsWriter { backend, _op: op, path, } } } impl oio::OneShotWrite for VercelArtifactsWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self .backend .vercel_artifacts_put(self.path.as_str(), bs.len() as u64, bs) .await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::ACCEPTED => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/vercel_blob/backend.rs000064400000000000000000000165531046102023000201710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::Response; use http::StatusCode; use log::debug; use super::core::parse_blob; use super::core::Blob; use super::core::VercelBlobCore; use super::error::parse_error; use super::lister::VercelBlobLister; use super::writer::VercelBlobWriter; use super::writer::VercelBlobWriters; use crate::raw::*; use crate::services::VercelBlobConfig; use crate::*; impl Configurator for VercelBlobConfig { type Builder = VercelBlobBuilder; fn into_builder(self) -> Self::Builder { VercelBlobBuilder { config: self, http_client: None, } } } /// [VercelBlob](https://vercel.com/docs/storage/vercel-blob) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct VercelBlobBuilder { config: VercelBlobConfig, http_client: Option, } impl Debug for VercelBlobBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("VercelBlobBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl VercelBlobBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Vercel Blob token. /// /// Get from Vercel environment variable `BLOB_READ_WRITE_TOKEN`. /// It is required. pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()); } self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for VercelBlobBuilder { const SCHEME: Scheme = Scheme::VercelBlob; type Config = VercelBlobConfig; /// Builds the backend and returns the result of VercelBlobBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle token. let Some(token) = self.config.token.clone() else { return Err(Error::new(ErrorKind::ConfigInvalid, "token is empty") .with_operation("Builder::build") .with_context("service", Scheme::VercelBlob)); }; let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::VercelBlob) })? }; Ok(VercelBlobBackend { core: Arc::new(VercelBlobCore { root, token, client, }), }) } } /// Backend for VercelBlob services. #[derive(Debug, Clone)] pub struct VercelBlobBackend { core: Arc, } impl Access for VercelBlobBackend { type Reader = HttpBody; type Writer = VercelBlobWriters; type Lister = oio::PageLister; type Deleter = (); type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::VercelBlob) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_type: true, stat_has_content_length: true, stat_has_last_modified: true, stat_has_content_disposition: true, read: true, write: true, write_can_empty: true, write_can_multi: true, write_multi_min_size: Some(5 * 1024 * 1024), copy: true, list: true, list_with_limit: true, list_has_content_type: true, list_has_content_length: true, list_has_last_modified: true, list_has_content_disposition: true, shared: true, ..Default::default() }); am.into() } async fn stat(&self, path: &str, _args: OpStat) -> Result { let resp = self.core.head(path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: Blob = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; parse_blob(&resp).map(RpStat::new) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.download(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let concurrent = args.concurrent(); let executor = args.executor().cloned(); let writer = VercelBlobWriter::new(self.core.clone(), args, path.to_string()); let w = oio::MultipartWriter::new(writer, executor, concurrent); Ok((RpWrite::default(), w)) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.copy(from, to).await?; let status = resp.status(); match status { StatusCode::OK => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = VercelBlobLister::new(self.core.clone(), path, args.limit()); Ok((RpList::default(), oio::PageLister::new(l))) } } opendal-0.52.0/src/services/vercel_blob/config.rs000064400000000000000000000026701046102023000200420ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for VercelBlob services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct VercelBlobConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, /// vercel blob token. pub token: Option, } impl Debug for VercelBlobConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Config"); ds.field("root", &self.root); ds.finish() } } opendal-0.52.0/src/services/vercel_blob/core.rs000064400000000000000000000261201046102023000175210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use bytes::Buf; use bytes::Bytes; use http::header; use http::request; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use serde::Serialize; use serde_json::json; use self::constants::*; use super::error::parse_error; use crate::raw::*; use crate::*; pub(super) mod constants { // https://github.com/vercel/storage/blob/main/packages/blob/src/put.ts#L16 // x-content-type specifies the MIME type of the file being uploaded. pub const X_VERCEL_BLOB_CONTENT_TYPE: &str = "x-content-type"; // x-add-random-suffix specifying whether to add a random suffix to the pathname // Default value is 1, which means to add a random suffix. // Set it to 0 to disable the random suffix. pub const X_VERCEL_BLOB_ADD_RANDOM_SUFFIX: &str = "x-add-random-suffix"; // https://github.com/vercel/storage/blob/main/packages/blob/src/put-multipart.ts#L84 // x-mpu-action specifies the action to perform on the MPU. // Possible values are: // - create: create a new MPU. // - upload: upload a part to an existing MPU. // - complete: complete an existing MPU. pub const X_VERCEL_BLOB_MPU_ACTION: &str = "x-mpu-action"; pub const X_VERCEL_BLOB_MPU_KEY: &str = "x-mpu-key"; pub const X_VERCEL_BLOB_MPU_PART_NUMBER: &str = "x-mpu-part-number"; pub const X_VERCEL_BLOB_MPU_UPLOAD_ID: &str = "x-mpu-upload-id"; } #[derive(Clone)] pub struct VercelBlobCore { /// The root of this core. pub root: String, /// Vercel Blob token. pub token: String, pub client: HttpClient, } impl Debug for VercelBlobCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .finish_non_exhaustive() } } impl VercelBlobCore { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } pub fn sign(&self, req: request::Builder) -> request::Builder { req.header(header::AUTHORIZATION, format!("Bearer {}", self.token)) } } impl VercelBlobCore { pub async fn download( &self, path: &str, range: BytesRange, _: &OpRead, ) -> Result> { let p = build_abs_path(&self.root, path); // Vercel blob use an unguessable random id url to download the file // So we use list to get the url of the file and then use it to download the file let resp = self.list(&p, Some(1)).await?; // Use the mtach url to download the file let url = resolve_blob(resp.blobs, p); if url.is_empty() { return Err(Error::new(ErrorKind::NotFound, "Blob not found")); } let mut req = Request::get(url); if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub fn get_put_request( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "https://blob.vercel-storage.com/{}", percent_encode_path(&p) ); let mut req = Request::put(&url); req = req.header(X_VERCEL_BLOB_ADD_RANDOM_SUFFIX, "0"); if let Some(size) = size { req = req.header(header::CONTENT_LENGTH, size.to_string()) } if let Some(mime) = args.content_type() { req = req.header(X_VERCEL_BLOB_CONTENT_TYPE, mime) } let req = self.sign(req); // Set body let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn head(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let resp = self.list(&p, Some(1)).await?; let url = resolve_blob(resp.blobs, p); if url.is_empty() { return Err(Error::new(ErrorKind::NotFound, "Blob not found")); } let req = Request::get(format!( "https://blob.vercel-storage.com?url={}", percent_encode_path(&url) )); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn copy(&self, from: &str, to: &str) -> Result> { let from = build_abs_path(&self.root, from); let resp = self.list(&from, Some(1)).await?; let from_url = resolve_blob(resp.blobs, from); if from_url.is_empty() { return Err(Error::new(ErrorKind::NotFound, "Blob not found")); } let to = build_abs_path(&self.root, to); let to_url = format!( "https://blob.vercel-storage.com/{}?fromUrl={}", percent_encode_path(&to), percent_encode_path(&from_url), ); let req = Request::put(&to_url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn list(&self, prefix: &str, limit: Option) -> Result { let prefix = if prefix == "/" { "" } else { prefix }; let mut url = format!( "https://blob.vercel-storage.com?prefix={}", percent_encode_path(prefix) ); if let Some(limit) = limit { url.push_str(&format!("&limit={}", limit)) } let req = Request::get(&url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } let body = resp.into_body(); let resp: ListResponse = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; Ok(resp) } pub async fn initiate_multipart_upload( &self, path: &str, args: &OpWrite, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "https://blob.vercel-storage.com/mpu/{}", percent_encode_path(&p) ); let req = Request::post(&url); let mut req = self.sign(req); req = req.header(X_VERCEL_BLOB_MPU_ACTION, "create"); req = req.header(X_VERCEL_BLOB_ADD_RANDOM_SUFFIX, "0"); if let Some(mime) = args.content_type() { req = req.header(X_VERCEL_BLOB_CONTENT_TYPE, mime); }; // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn upload_part( &self, path: &str, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "https://blob.vercel-storage.com/mpu/{}", percent_encode_path(&p) ); let mut req = Request::post(&url); req = req.header(header::CONTENT_LENGTH, size); req = req.header(X_VERCEL_BLOB_MPU_ACTION, "upload"); req = req.header(X_VERCEL_BLOB_MPU_KEY, p); req = req.header(X_VERCEL_BLOB_MPU_UPLOAD_ID, upload_id); req = req.header(X_VERCEL_BLOB_MPU_PART_NUMBER, part_number); let req = self.sign(req); // Set body let req = req.body(body).map_err(new_request_build_error)?; self.send(req).await } pub async fn complete_multipart_upload( &self, path: &str, upload_id: &str, parts: Vec, ) -> Result> { let p = build_abs_path(&self.root, path); let url = format!( "https://blob.vercel-storage.com/mpu/{}", percent_encode_path(&p) ); let mut req = Request::post(&url); req = req.header(X_VERCEL_BLOB_MPU_ACTION, "complete"); req = req.header(X_VERCEL_BLOB_MPU_KEY, p); req = req.header(X_VERCEL_BLOB_MPU_UPLOAD_ID, upload_id); let req = self.sign(req); let parts_json = json!(parts); let req = req .header(header::CONTENT_TYPE, "application/json") .body(Buffer::from(Bytes::from(parts_json.to_string()))) .map_err(new_request_build_error)?; self.send(req).await } } pub fn parse_blob(blob: &Blob) -> Result { let mode = if blob.pathname.ends_with('/') { EntryMode::DIR } else { EntryMode::FILE }; let mut md = Metadata::new(mode); if let Some(content_type) = blob.content_type.clone() { md.set_content_type(&content_type); } md.set_content_length(blob.size); md.set_last_modified(parse_datetime_from_rfc3339(&blob.uploaded_at)?); md.set_content_disposition(&blob.content_disposition); Ok(md) } fn resolve_blob(blobs: Vec, path: String) -> String { for blob in blobs { if blob.pathname == path { return blob.url; } } "".to_string() } #[derive(Default, Debug, Clone, PartialEq, Deserialize)] #[serde(rename_all = "camelCase")] pub struct ListResponse { pub cursor: Option, pub has_more: bool, pub blobs: Vec, } #[derive(Default, Debug, Clone, PartialEq, Deserialize)] #[serde(rename_all = "camelCase")] pub struct Blob { pub url: String, pub pathname: String, pub size: u64, pub uploaded_at: String, pub content_disposition: String, pub content_type: Option, } #[derive(Default, Debug, Clone, PartialEq, Serialize)] #[serde(rename_all = "camelCase")] pub struct Part { pub part_number: usize, pub etag: String, } #[derive(Default, Debug, Clone, PartialEq, Deserialize)] #[serde(rename_all = "camelCase")] pub struct InitiateMultipartUploadResponse { pub upload_id: String, pub key: String, } #[derive(Default, Debug, Clone, PartialEq, Deserialize)] #[serde(rename_all = "camelCase")] pub struct UploadPartResponse { pub etag: String, } opendal-0.52.0/src/services/vercel_blob/docs.md000064400000000000000000000014741046102023000175020ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [ ] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `token`: VercelBlob token, environment var `BLOB_READ_WRITE_TOKEN` You can refer to [`VercelBlobBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::VercelBlob; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = VercelBlob::default() // set the storage bucket for OpenDAL .root("/") // set the token for OpenDAL .token("you_token"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/vercel_blob/error.rs000064400000000000000000000063301046102023000177230ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// VercelBlobError is the error returned by VercelBlob service. #[derive(Default, Debug, Deserialize)] #[serde(default)] struct VercelBlobError { error: VercelBlobErrorDetail, } #[derive(Default, Debug, Deserialize)] #[serde(default)] struct VercelBlobErrorDetail { code: String, message: Option, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status.as_u16() { 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), 500 | 502 | 503 | 504 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, _vercel_blob_err) = de::from_reader::<_, VercelBlobError>(bs.clone().reader()) .map(|vercel_blob_err| (format!("{vercel_blob_err:?}"), Some(vercel_blob_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[tokio::test] async fn test_parse_error() { let err_res = vec![( r#"{ "error": { "code": "forbidden", "message": "Invalid token" } }"#, ErrorKind::PermissionDenied, StatusCode::FORBIDDEN, )]; for res in err_res { let body = Buffer::from(res.0.as_bytes().to_vec()); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } let bs = bytes::Bytes::from( r#"{ "error": { "code": "forbidden", "message": "Invalid token" } }"#, ); let out: VercelBlobError = serde_json::from_reader(bs.reader()).expect("must success"); println!("{out:?}"); assert_eq!(out.error.code, "forbidden"); assert_eq!(out.error.message, Some("Invalid token".to_string())); } } opendal-0.52.0/src/services/vercel_blob/lister.rs000064400000000000000000000036651046102023000201040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use super::core::parse_blob; use super::core::VercelBlobCore; use crate::raw::oio::Entry; use crate::raw::*; use crate::Result; pub struct VercelBlobLister { core: Arc, path: String, limit: Option, } impl VercelBlobLister { pub(super) fn new(core: Arc, path: &str, limit: Option) -> Self { VercelBlobLister { core, path: path.to_string(), limit, } } } impl oio::PageList for VercelBlobLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let p = build_abs_path(&self.core.root, &self.path); let resp = self.core.list(&p, self.limit).await?; ctx.done = !resp.has_more; if let Some(cursor) = resp.cursor { ctx.token = cursor; } for blob in resp.blobs { let path = build_rel_path(&self.core.root, &blob.pathname); if path == self.path { continue; } let md = parse_blob(&blob)?; ctx.entries.push_back(Entry::new(&path, md)); } Ok(()) } } opendal-0.52.0/src/services/vercel_blob/mod.rs000064400000000000000000000022641046102023000173530ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-vercel-blob")] mod core; #[cfg(feature = "services-vercel-blob")] mod error; #[cfg(feature = "services-vercel-blob")] mod lister; #[cfg(feature = "services-vercel-blob")] mod writer; #[cfg(feature = "services-vercel-blob")] mod backend; #[cfg(feature = "services-vercel-blob")] pub use backend::VercelBlobBuilder as VercelBlob; mod config; pub use config::VercelBlobConfig; opendal-0.52.0/src/services/vercel_blob/writer.rs000064400000000000000000000101011046102023000200750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use http::StatusCode; use super::core::InitiateMultipartUploadResponse; use super::core::Part; use super::core::UploadPartResponse; use super::core::VercelBlobCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type VercelBlobWriters = oio::MultipartWriter; pub struct VercelBlobWriter { core: Arc, op: OpWrite, path: String, } impl VercelBlobWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { VercelBlobWriter { core, op, path } } } impl oio::MultipartWrite for VercelBlobWriter { async fn write_once(&self, size: u64, body: Buffer) -> Result { let req = self .core .get_put_request(&self.path, Some(size), &self.op, body)?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn initiate_part(&self) -> Result { let resp = self .core .initiate_multipart_upload(&self.path, &self.op) .await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: InitiateMultipartUploadResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; Ok(resp.upload_id) } _ => Err(parse_error(resp)), } } async fn write_part( &self, upload_id: &str, part_number: usize, size: u64, body: Buffer, ) -> Result { let part_number = part_number + 1; let resp = self .core .upload_part(&self.path, upload_id, part_number, size, body) .await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: UploadPartResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; Ok(oio::MultipartPart { part_number, etag: resp.etag, checksum: None, }) } _ => Err(parse_error(resp)), } } async fn complete_part( &self, upload_id: &str, parts: &[oio::MultipartPart], ) -> Result { let parts = parts .iter() .map(|p| Part { part_number: p.part_number, etag: p.etag.clone(), }) .collect::>(); let resp = self .core .complete_multipart_upload(&self.path, upload_id, parts) .await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn abort_part(&self, _upload_id: &str) -> Result<()> { Err(Error::new( ErrorKind::Unsupported, "VercelBlob does not support abort multipart upload", )) } } opendal-0.52.0/src/services/webdav/backend.rs000064400000000000000000000230621046102023000171540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::str::FromStr; use std::sync::Arc; use http::Response; use http::StatusCode; use log::debug; use super::core::*; use super::delete::WebdavDeleter; use super::error::parse_error; use super::lister::WebdavLister; use super::writer::WebdavWriter; use crate::raw::*; use crate::services::WebdavConfig; use crate::*; impl Configurator for WebdavConfig { type Builder = WebdavBuilder; fn into_builder(self) -> Self::Builder { WebdavBuilder { config: self, http_client: None, } } } /// [WebDAV](https://datatracker.ietf.org/doc/html/rfc4918) backend support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct WebdavBuilder { config: WebdavConfig, http_client: Option, } impl Debug for WebdavBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("WebdavBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl WebdavBuilder { /// Set endpoint for http backend. /// /// For example: `https://example.com` pub fn endpoint(mut self, endpoint: &str) -> Self { self.config.endpoint = if endpoint.is_empty() { None } else { Some(endpoint.to_string()) }; self } /// set the username for Webdav /// /// default: no username pub fn username(mut self, username: &str) -> Self { if !username.is_empty() { self.config.username = Some(username.to_owned()); } self } /// set the password for Webdav /// /// default: no password pub fn password(mut self, password: &str) -> Self { if !password.is_empty() { self.config.password = Some(password.to_owned()); } self } /// set the bearer token for Webdav /// /// default: no access token pub fn token(mut self, token: &str) -> Self { if !token.is_empty() { self.config.token = Some(token.to_string()); } self } /// Set root path of http backend. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for WebdavBuilder { const SCHEME: Scheme = Scheme::Webdav; type Config = WebdavConfig; fn build(self) -> Result { debug!("backend build started: {:?}", &self); let endpoint = match &self.config.endpoint { Some(v) => v, None => { return Err(Error::new(ErrorKind::ConfigInvalid, "endpoint is empty") .with_context("service", Scheme::Webdav)); } }; // Some services might return the path with suffix `/remote.php/webdav/`, we need to trim them. let server_path = http::Uri::from_str(endpoint) .map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "endpoint is invalid") .with_context("service", Scheme::Webdav) .set_source(err) })? .path() .trim_end_matches('/') .to_string(); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", root); let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::Webdav) })? }; let mut authorization = None; if let Some(username) = &self.config.username { authorization = Some(format_authorization_by_basic( username, self.config.password.as_deref().unwrap_or_default(), )?); } if let Some(token) = &self.config.token { authorization = Some(format_authorization_by_bearer(token)?) } let core = Arc::new(WebdavCore { endpoint: endpoint.to_string(), server_path, authorization, disable_copy: self.config.disable_copy, root, client, }); Ok(WebdavBackend { core }) } } /// Backend is used to serve `Accessor` support for http. #[derive(Clone)] pub struct WebdavBackend { core: Arc, } impl Debug for WebdavBackend { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("WebdavBackend") .field("core", &self.core) .finish() } } impl Access for WebdavBackend { type Reader = HttpBody; type Writer = oio::OneShotWriter; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut ma = AccessorInfo::default(); ma.set_scheme(Scheme::Webdav) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_content_type: true, stat_has_etag: true, stat_has_last_modified: true, read: true, write: true, write_can_empty: true, create_dir: true, delete: true, copy: !self.core.disable_copy, rename: true, list: true, list_has_content_length: true, list_has_content_type: true, list_has_etag: true, list_has_last_modified: true, // We already support recursive list but some details still need to polish. // list_with_recursive: true, shared: true, ..Default::default() }); ma.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { self.core.webdav_mkcol(path).await?; Ok(RpCreateDir::default()) } async fn stat(&self, path: &str, _: OpStat) -> Result { let metadata = self.core.webdav_stat(path).await?; Ok(RpStat::new(metadata)) } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.core.webdav_get(path, args.range(), &args).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { // Ensure parent path exists self.core.webdav_mkcol(get_parent(path)).await?; Ok(( RpWrite::default(), oio::OneShotWriter::new(WebdavWriter::new(self.core.clone(), args, path.to_string())), )) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(WebdavDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { Ok(( RpList::default(), oio::PageLister::new(WebdavLister::new(self.core.clone(), path, args)), )) } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { let resp = self.core.webdav_copy(from, to).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::NO_CONTENT => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { let resp = self.core.webdav_move(from, to).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::NO_CONTENT | StatusCode::OK => { Ok(RpRename::default()) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/webdav/config.rs000064400000000000000000000034441046102023000170340ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for [WebDAV](https://datatracker.ietf.org/doc/html/rfc4918) backend support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct WebdavConfig { /// endpoint of this backend pub endpoint: Option, /// username of this backend pub username: Option, /// password of this backend pub password: Option, /// token of this backend pub token: Option, /// root of this backend pub root: Option, /// WebDAV Service doesn't support copy. pub disable_copy: bool, } impl Debug for WebdavConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("WebdavConfig"); d.field("endpoint", &self.endpoint) .field("username", &self.username) .field("root", &self.root); d.finish_non_exhaustive() } } opendal-0.52.0/src/services/webdav/core.rs000064400000000000000000000673541046102023000165310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::VecDeque; use std::fmt; use std::fmt::Debug; use std::fmt::Formatter; use bytes::Bytes; use http::header; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use super::error::parse_error; use crate::raw::*; use crate::*; /// The request to query all properties of a file or directory. /// /// rfc4918 9.1: retrieve all properties define in specification static PROPFIND_REQUEST: &str = r#""#; /// The header to specify the depth of the query. /// /// Valid values are `0`, `1`, `infinity`. /// /// - `0`: only to the resource itself. /// - `1`: to the resource and its internal members only. /// - `infinity`: to the resource and all its members. /// /// reference: [RFC4918: 10.2. Depth Header](https://datatracker.ietf.org/doc/html/rfc4918#section-10.2) static HEADER_DEPTH: &str = "Depth"; /// The header to specify the destination of the query. /// /// The Destination request header specifies the URI that identifies a /// destination resource for methods such as COPY and MOVE, which take /// two URIs as parameters. /// /// reference: [RFC4918: 10.3. Destination Header](https://datatracker.ietf.org/doc/html/rfc4918#section-10.3) static HEADER_DESTINATION: &str = "Destination"; /// The header to specify the overwrite behavior of the query /// /// The Overwrite request header specifies whether the server should /// overwrite a resource mapped to the destination URL during a COPY or /// MOVE. /// /// Valid values are `T` and `F`. /// /// A value of "F" states that the server must not perform the COPY or MOVE operation /// if the destination URL does map to a resource. /// /// reference: [RFC4918: 10.6. Overwrite Header](https://datatracker.ietf.org/doc/html/rfc4918#section-10.6) static HEADER_OVERWRITE: &str = "Overwrite"; pub struct WebdavCore { pub endpoint: String, pub server_path: String, pub root: String, pub disable_copy: bool, pub authorization: Option, pub client: HttpClient, } impl Debug for WebdavCore { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { f.debug_struct("WebdavCore") .field("endpoint", &self.endpoint) .field("root", &self.root) .finish_non_exhaustive() } } impl WebdavCore { pub async fn webdav_stat(&self, path: &str) -> Result { let path = build_rooted_abs_path(&self.root, path); self.webdav_stat_rooted_abs_path(&path).await } /// Input path must be `rooted_abs_path`. async fn webdav_stat_rooted_abs_path(&self, rooted_abs_path: &str) -> Result { let url = format!("{}{}", self.endpoint, percent_encode_path(rooted_abs_path)); let mut req = Request::builder().method("PROPFIND").uri(url); req = req.header(header::CONTENT_TYPE, "application/xml"); req = req.header(header::CONTENT_LENGTH, PROPFIND_REQUEST.len()); if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth); } // Only stat the resource itself. req = req.header(HEADER_DEPTH, "0"); let req = req .body(Buffer::from(Bytes::from(PROPFIND_REQUEST))) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; if !resp.status().is_success() { return Err(parse_error(resp)); } let bs = resp.into_body(); let result: Multistatus = deserialize_multistatus(&bs.to_bytes())?; let propfind_resp = result.response.first().ok_or_else(|| { Error::new( ErrorKind::NotFound, "propfind response is empty, the resource is not exist", ) })?; let metadata = parse_propstat(&propfind_resp.propstat)?; Ok(metadata) } pub async fn webdav_get( &self, path: &str, range: BytesRange, _: &OpRead, ) -> Result> { let path = build_rooted_abs_path(&self.root, path); let url: String = format!("{}{}", self.endpoint, percent_encode_path(&path)); let mut req = Request::get(&url); if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth.clone()) } if !range.is_full() { req = req.header(header::RANGE, range.to_header()); } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.fetch(req).await } pub async fn webdav_put( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let path = build_rooted_abs_path(&self.root, path); let url = format!("{}{}", self.endpoint, percent_encode_path(&path)); let mut req = Request::put(&url); if let Some(v) = &self.authorization { req = req.header(header::AUTHORIZATION, v) } if let Some(v) = size { req = req.header(header::CONTENT_LENGTH, v) } if let Some(v) = args.content_type() { req = req.header(header::CONTENT_TYPE, v) } if let Some(v) = args.content_disposition() { req = req.header(header::CONTENT_DISPOSITION, v) } let req = req.body(body).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn webdav_delete(&self, path: &str) -> Result> { let path = build_rooted_abs_path(&self.root, path); let url = format!("{}{}", self.endpoint, percent_encode_path(&path)); let mut req = Request::delete(&url); if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth.clone()) } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn webdav_copy(&self, from: &str, to: &str) -> Result> { // Check if source file exists. let _ = self.webdav_stat(from).await?; // Make sure target's dir is exist. self.webdav_mkcol(get_parent(to)).await?; let source = build_rooted_abs_path(&self.root, from); let source_uri = format!("{}{}", self.endpoint, percent_encode_path(&source)); let target = build_rooted_abs_path(&self.root, to); let target_uri = format!("{}{}", self.endpoint, percent_encode_path(&target)); let mut req = Request::builder().method("COPY").uri(&source_uri); if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth); } req = req.header(HEADER_DESTINATION, target_uri); req = req.header(HEADER_OVERWRITE, "T"); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn webdav_move(&self, from: &str, to: &str) -> Result> { // Check if source file exists. let _ = self.webdav_stat(from).await?; // Make sure target's dir is exist. self.webdav_mkcol(get_parent(to)).await?; let source = build_rooted_abs_path(&self.root, from); let source_uri = format!("{}{}", self.endpoint, percent_encode_path(&source)); let target = build_rooted_abs_path(&self.root, to); let target_uri = format!("{}{}", self.endpoint, percent_encode_path(&target)); let mut req = Request::builder().method("MOVE").uri(&source_uri); if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth); } req = req.header(HEADER_DESTINATION, target_uri); req = req.header(HEADER_OVERWRITE, "T"); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.client.send(req).await } pub async fn webdav_list(&self, path: &str, args: &OpList) -> Result> { let path = build_rooted_abs_path(&self.root, path); let url = format!("{}{}", self.endpoint, percent_encode_path(&path)); let mut req = Request::builder().method("PROPFIND").uri(&url); req = req.header(header::CONTENT_TYPE, "application/xml"); req = req.header(header::CONTENT_LENGTH, PROPFIND_REQUEST.len()); if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth); } if args.recursive() { req = req.header(HEADER_DEPTH, "infinity"); } else { req = req.header(HEADER_DEPTH, "1"); } let req = req .body(Buffer::from(Bytes::from(PROPFIND_REQUEST))) .map_err(new_request_build_error)?; self.client.send(req).await } /// Create dir recursively for given path. /// /// # Notes /// /// We only expose this method to the backend since there are dependencies on input path. pub async fn webdav_mkcol(&self, path: &str) -> Result<()> { let path = build_rooted_abs_path(&self.root, path); let mut path = path.as_str(); let mut dirs = VecDeque::default(); loop { match self.webdav_stat_rooted_abs_path(path).await { // Dir exists, break the loop. Ok(_) => { break; } // Dir not found, keep going. Err(err) if err.kind() == ErrorKind::NotFound => { dirs.push_front(path); path = get_parent(path); } // Unexpected error found, return it. Err(err) => return Err(err), } if path == "/" { break; } } for dir in dirs { self.webdav_mkcol_rooted_abs_path(dir).await?; } Ok(()) } /// Create a dir /// /// Input path must be `rooted_abs_path` /// /// Reference: [RFC4918: 9.3.1. MKCOL Status Codes](https://datatracker.ietf.org/doc/html/rfc4918#section-9.3.1) async fn webdav_mkcol_rooted_abs_path(&self, rooted_abs_path: &str) -> Result<()> { let url = format!("{}{}", self.endpoint, percent_encode_path(rooted_abs_path)); let mut req = Request::builder().method("MKCOL").uri(&url); if let Some(auth) = &self.authorization { req = req.header(header::AUTHORIZATION, auth.clone()) } let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { // 201 (Created) - The collection was created. StatusCode::CREATED // 405 (Method Not Allowed) - MKCOL can only be executed on an unmapped URL. // // The MKCOL method can only be performed on a deleted or non-existent resource. // This error means the directory already exists which is allowed by create_dir. | StatusCode::METHOD_NOT_ALLOWED => { Ok(()) } _ => Err(parse_error(resp)), } } } pub fn deserialize_multistatus(bs: &[u8]) -> Result { let s = String::from_utf8_lossy(bs); // HACKS! HACKS! HACKS! // // Make sure the string is escaped. // Related to // // This is a temporary solution, we should find a better way to handle this. let s = s.replace("&()_+-=;", "%26%28%29_%2B-%3D%3B"); quick_xml::de::from_str(&s).map_err(new_xml_deserialize_error) } pub fn parse_propstat(propstat: &Propstat) -> Result { let Propstat { prop: Prop { getlastmodified, getcontentlength, getcontenttype, getetag, resourcetype, .. }, status, } = propstat; if let [_, code, text] = status.splitn(3, ' ').collect::>()[..3] { // As defined in https://tools.ietf.org/html/rfc2068#section-6.1 let code = code.parse::().unwrap(); if code >= 400 { return Err(Error::new( ErrorKind::Unexpected, format!("propfind response is unexpected: {} {}", code, text), )); } } let mode: EntryMode = if resourcetype.value == Some(ResourceType::Collection) { EntryMode::DIR } else { EntryMode::FILE }; let mut m = Metadata::new(mode); if let Some(v) = getcontentlength { m.set_content_length(v.parse::().unwrap()); } if let Some(v) = getcontenttype { m.set_content_type(v); } if let Some(v) = getetag { m.set_etag(v); } // https://www.rfc-editor.org/rfc/rfc4918#section-14.18 m.set_last_modified(parse_datetime_from_rfc2822(getlastmodified)?); // the storage services have returned all the properties Ok(m) } #[derive(Deserialize, Debug, PartialEq, Eq, Clone, Default)] #[serde(default)] pub struct Multistatus { pub response: Vec, } #[derive(Deserialize, Debug, PartialEq, Eq, Clone)] pub struct PropfindResponse { pub href: String, pub propstat: Propstat, } #[derive(Deserialize, Debug, PartialEq, Eq, Clone)] pub struct Propstat { pub status: String, pub prop: Prop, } #[derive(Deserialize, Debug, PartialEq, Eq, Clone)] pub struct Prop { pub getlastmodified: String, pub getetag: Option, pub getcontentlength: Option, pub getcontenttype: Option, pub resourcetype: ResourceTypeContainer, } #[derive(Deserialize, Debug, PartialEq, Eq, Clone)] pub struct ResourceTypeContainer { #[serde(rename = "$value")] pub value: Option, } #[derive(Deserialize, Debug, PartialEq, Eq, Clone)] #[serde(rename_all = "lowercase")] pub enum ResourceType { Collection, } #[cfg(test)] mod tests { use quick_xml::de::from_str; use super::*; #[test] fn test_propstat() { let xml = r#" / Tue, 01 May 2022 06:39:47 GMT HTTP/1.1 200 OK "#; let propstat = from_str::(xml).unwrap(); assert_eq!( propstat.prop.getlastmodified, "Tue, 01 May 2022 06:39:47 GMT" ); assert_eq!( propstat.prop.resourcetype.value.unwrap(), ResourceType::Collection ); assert_eq!(propstat.status, "HTTP/1.1 200 OK"); } #[test] fn test_response_simple() { let xml = r#" / / Tue, 01 May 2022 06:39:47 GMT HTTP/1.1 200 OK "#; let response = from_str::(xml).unwrap(); assert_eq!(response.href, "/"); assert_eq!( response.propstat.prop.getlastmodified, "Tue, 01 May 2022 06:39:47 GMT" ); assert_eq!( response.propstat.prop.resourcetype.value.unwrap(), ResourceType::Collection ); assert_eq!(response.propstat.status, "HTTP/1.1 200 OK"); } #[test] fn test_response_file() { let xml = r#" /test_file test_file 1 Tue, 07 May 2022 05:52:22 GMT HTTP/1.1 200 OK "#; let response = from_str::(xml).unwrap(); assert_eq!(response.href, "/test_file"); assert_eq!( response.propstat.prop.getlastmodified, "Tue, 07 May 2022 05:52:22 GMT" ); assert_eq!(response.propstat.prop.getcontentlength.unwrap(), "1"); assert_eq!(response.propstat.prop.resourcetype.value, None); assert_eq!(response.propstat.status, "HTTP/1.1 200 OK"); } #[test] fn test_with_multiple_items_simple() { let xml = r#" / / Tue, 01 May 2022 06:39:47 GMT HTTP/1.1 200 OK / / Tue, 01 May 2022 06:39:47 GMT HTTP/1.1 200 OK "#; let multistatus = from_str::(xml).unwrap(); let response = multistatus.response; assert_eq!(response.len(), 2); assert_eq!(response[0].href, "/"); assert_eq!( response[0].propstat.prop.getlastmodified, "Tue, 01 May 2022 06:39:47 GMT" ); } #[test] fn test_with_multiple_items_mixed() { let xml = r#" / / Tue, 07 May 2022 06:39:47 GMT HTTP/1.1 200 OK /testdir/ testdir Tue, 07 May 2022 06:40:10 GMT HTTP/1.1 200 OK /test_file test_file 1 Tue, 07 May 2022 05:52:22 GMT HTTP/1.1 200 OK "#; let multistatus = from_str::(xml).unwrap(); let response = multistatus.response; assert_eq!(response.len(), 3); let first_response = &response[0]; assert_eq!(first_response.href, "/"); assert_eq!( first_response.propstat.prop.getlastmodified, "Tue, 07 May 2022 06:39:47 GMT" ); let second_response = &response[1]; assert_eq!(second_response.href, "/testdir/"); assert_eq!( second_response.propstat.prop.getlastmodified, "Tue, 07 May 2022 06:40:10 GMT" ); let third_response = &response[2]; assert_eq!(third_response.href, "/test_file"); assert_eq!( third_response.propstat.prop.getlastmodified, "Tue, 07 May 2022 05:52:22 GMT" ); } #[test] fn test_with_multiple_items_mixed_nginx() { let xml = r#" / Fri, 17 Feb 2023 03:37:22 GMT HTTP/1.1 200 OK /test_file_75 1 Fri, 17 Feb 2023 03:36:54 GMT HTTP/1.1 200 OK /test_file_36 1 Fri, 17 Feb 2023 03:36:54 GMT HTTP/1.1 200 OK /test_file_38 1 Fri, 17 Feb 2023 03:36:54 GMT HTTP/1.1 200 OK /test_file_59 1 Fri, 17 Feb 2023 03:36:54 GMT HTTP/1.1 200 OK /test_file_9 1 Fri, 17 Feb 2023 03:36:54 GMT HTTP/1.1 200 OK /test_file_93 1 Fri, 17 Feb 2023 03:36:54 GMT HTTP/1.1 200 OK /test_file_43 1 Fri, 17 Feb 2023 03:36:54 GMT HTTP/1.1 200 OK /test_file_95 1 Fri, 17 Feb 2023 03:36:54 GMT HTTP/1.1 200 OK "#; let multistatus: Multistatus = from_str(xml).unwrap(); let response = multistatus.response; assert_eq!(response.len(), 9); let first_response = &response[0]; assert_eq!(first_response.href, "/"); assert_eq!( first_response.propstat.prop.getlastmodified, "Fri, 17 Feb 2023 03:37:22 GMT" ); } } opendal-0.52.0/src/services/webdav/delete.rs000064400000000000000000000026541046102023000170330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct WebdavDeleter { core: Arc, } impl WebdavDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for WebdavDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.webdav_delete(&path).await?; let status = resp.status(); match status { StatusCode::NO_CONTENT | StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/webdav/docs.md000064400000000000000000000015601046102023000164700ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [ ] ~~presign~~ - [ ] blocking ## Notes Bazel Remote Caching and Ccache HTTP Storage is also part of this service. Users can use `webdav` to connect those services. ## Configuration - `endpoint`: set the endpoint for webdav - `root`: Set the work directory for backend You can refer to [`WebdavBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::Webdav; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = Webdav::default() .endpoint("127.0.0.1") .username("xxx") .password("xxx"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/webdav/error.rs000064400000000000000000000035211046102023000167140ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::Response; use http::StatusCode; use crate::raw::*; use crate::*; /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), // Some services (like owncloud) return 403 while file locked. StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, true), // Allowing retry for resource locked. StatusCode::LOCKED => (ErrorKind::Unexpected, true), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = String::from_utf8_lossy(&bs); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } opendal-0.52.0/src/services/webdav/lister.rs000064400000000000000000000066311046102023000170720ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::*; use super::error::*; use crate::raw::*; use crate::*; pub struct WebdavLister { core: Arc, path: String, args: OpList, } impl WebdavLister { pub fn new(core: Arc, path: &str, args: OpList) -> Self { Self { core, path: path.to_string(), args, } } } impl oio::PageList for WebdavLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let resp = self.core.webdav_list(&self.path, &self.args).await?; // jfrog artifactory's webdav services have some strange behavior. // We add this flag to check if the server is jfrog artifactory. // // Example: `"x-jfrog-version": "Artifactory/7.77.5 77705900"` let is_jfrog_artifactory = if let Some(v) = resp.headers().get("x-jfrog-version") { v.to_str().unwrap_or_default().starts_with("Artifactory") } else { false }; let bs = if resp.status().is_success() { resp.into_body() } else if resp.status() == StatusCode::NOT_FOUND && self.path.ends_with('/') { ctx.done = true; return Ok(()); } else { return Err(parse_error(resp)); }; let result: Multistatus = deserialize_multistatus(&bs.to_bytes())?; for res in result.response { let mut path = res .href .strip_prefix(&self.core.server_path) .unwrap_or(&res.href) .to_string(); let meta = parse_propstat(&res.propstat)?; // Append `/` to path if it's a dir if !path.ends_with('/') && meta.is_dir() { path += "/" } let decoded_path = percent_decode_path(&path); // Ignore the root path itself. if self.core.root == decoded_path { continue; } let normalized_path = build_rel_path(&self.core.root, &decoded_path); // HACKS! HACKS! HACKS! // // jfrog artifactory will generate a virtual checksum file for each file. // The checksum file can't be stated, but can be listed and read. // We ignore the checksum files to avoid listing unexpected files. if is_jfrog_artifactory && meta.content_type() == Some("application/x-checksum") { continue; } ctx.entries .push_back(oio::Entry::new(&normalized_path, meta)) } ctx.done = true; Ok(()) } } opendal-0.52.0/src/services/webdav/mod.rs000064400000000000000000000022721046102023000163440ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-webdav")] mod core; #[cfg(feature = "services-webdav")] mod delete; #[cfg(feature = "services-webdav")] mod error; #[cfg(feature = "services-webdav")] mod lister; #[cfg(feature = "services-webdav")] mod writer; #[cfg(feature = "services-webdav")] mod backend; #[cfg(feature = "services-webdav")] pub use backend::WebdavBuilder as Webdav; mod config; pub use config::WebdavConfig; opendal-0.52.0/src/services/webdav/writer.rs000064400000000000000000000031771046102023000171060ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::StatusCode; use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; pub struct WebdavWriter { core: Arc, op: OpWrite, path: String, } impl WebdavWriter { pub fn new(core: Arc, op: OpWrite, path: String) -> Self { WebdavWriter { core, op, path } } } impl oio::OneShotWrite for WebdavWriter { async fn write_once(&self, bs: Buffer) -> Result { let resp = self .core .webdav_put(&self.path, Some(bs.len() as u64), &self.op, bs) .await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK | StatusCode::NO_CONTENT => { Ok(Metadata::default()) } _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/webhdfs/backend.rs000064400000000000000000000530351046102023000173310ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use core::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::header::CONTENT_LENGTH; use http::header::CONTENT_TYPE; use http::Request; use http::Response; use http::StatusCode; use log::debug; use serde::Deserialize; use tokio::sync::OnceCell; use super::delete::WebhdfsDeleter; use super::error::parse_error; use super::lister::WebhdfsLister; use super::message::BooleanResp; use super::message::FileStatusType; use super::message::FileStatusWrapper; use super::writer::WebhdfsWriter; use super::writer::WebhdfsWriters; use crate::raw::*; use crate::services::WebhdfsConfig; use crate::*; const WEBHDFS_DEFAULT_ENDPOINT: &str = "http://127.0.0.1:9870"; impl Configurator for WebhdfsConfig { type Builder = WebhdfsBuilder; fn into_builder(self) -> Self::Builder { WebhdfsBuilder { config: self } } } /// [WebHDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html)'s REST API support. #[doc = include_str!("docs.md")] #[derive(Default, Clone)] pub struct WebhdfsBuilder { config: WebhdfsConfig, } impl Debug for WebhdfsBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("WebhdfsBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl WebhdfsBuilder { /// Set the working directory of this backend /// /// All operations will happen under this root /// /// # Note /// /// The root will be automatically created if not exists. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// Set the remote address of this backend /// default to `http://127.0.0.1:9870` /// /// Endpoints should be full uri, e.g. /// /// - `https://webhdfs.example.com:9870` /// - `http://192.168.66.88:9870` /// /// If user inputs endpoint without scheme, we will /// prepend `http://` to it. pub fn endpoint(mut self, endpoint: &str) -> Self { if !endpoint.is_empty() { // trim tailing slash so we can accept `http://127.0.0.1:9870/` self.config.endpoint = Some(endpoint.trim_end_matches('/').to_string()); } self } /// Set the username of this backend, /// used for authentication /// pub fn user_name(mut self, user_name: &str) -> Self { if !user_name.is_empty() { self.config.user_name = Some(user_name.to_string()); } self } /// Set the delegation token of this backend, /// used for authentication /// /// # Note /// The builder prefers using delegation token over username. /// If both are set, delegation token will be used. pub fn delegation(mut self, delegation: &str) -> Self { if !delegation.is_empty() { self.config.delegation = Some(delegation.to_string()); } self } /// Disable batch listing /// /// # Note /// /// When listing a directory, the backend will default to use batch listing. /// If disabled, the backend will list all files/directories in one request. pub fn disable_list_batch(mut self) -> Self { self.config.disable_list_batch = true; self } /// Set temp dir for atomic write. /// /// # Notes /// /// If not set, write multi not support, eg: `.opendal_tmp/`. pub fn atomic_write_dir(mut self, dir: &str) -> Self { self.config.atomic_write_dir = if dir.is_empty() { None } else { Some(String::from(dir)) }; self } } impl Builder for WebhdfsBuilder { const SCHEME: Scheme = Scheme::Webhdfs; type Config = WebhdfsConfig; /// build the backend /// /// # Note /// /// when building backend, the built backend will check if the root directory /// exits. /// if the directory does not exit, the directory will be automatically created fn build(self) -> Result { debug!("start building backend: {:?}", self); let root = normalize_root(&self.config.root.unwrap_or_default()); debug!("backend use root {root}"); // check scheme let endpoint = match self.config.endpoint { Some(endpoint) => { if endpoint.starts_with("http") { endpoint } else { format!("http://{endpoint}") } } None => WEBHDFS_DEFAULT_ENDPOINT.to_string(), }; debug!("backend use endpoint {}", endpoint); let atomic_write_dir = self.config.atomic_write_dir; let auth = self.config.delegation.map(|dt| format!("delegation={dt}")); let client = HttpClient::new()?; let backend = WebhdfsBackend { root, endpoint, user_name: self.config.user_name, auth, client, root_checker: OnceCell::new(), atomic_write_dir, disable_list_batch: self.config.disable_list_batch, }; Ok(backend) } } /// Backend for WebHDFS service #[derive(Debug, Clone)] pub struct WebhdfsBackend { root: String, endpoint: String, user_name: Option, auth: Option, root_checker: OnceCell<()>, pub atomic_write_dir: Option, pub disable_list_batch: bool, pub client: HttpClient, } impl WebhdfsBackend { pub fn webhdfs_create_dir_request(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/webhdfs/v1/{}?op=MKDIRS&overwrite=true&noredirect=true", self.endpoint, percent_encode_path(&p), ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += format!("&{auth}").as_str(); } let req = Request::put(&url); req.body(Buffer::new()).map_err(new_request_build_error) } /// create object pub async fn webhdfs_create_object_request( &self, path: &str, size: Option, args: &OpWrite, body: Buffer, ) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/webhdfs/v1/{}?op=CREATE&overwrite=true&noredirect=true", self.endpoint, percent_encode_path(&p), ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += format!("&{auth}").as_str(); } let req = Request::put(&url); let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); if status != StatusCode::CREATED && status != StatusCode::OK { return Err(parse_error(resp)); } let bs = resp.into_body(); let resp: LocationResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let mut req = Request::put(&resp.location); if let Some(size) = size { req = req.header(CONTENT_LENGTH, size); }; if let Some(content_type) = args.content_type() { req = req.header(CONTENT_TYPE, content_type); }; let req = req.body(body).map_err(new_request_build_error)?; Ok(req) } pub async fn webhdfs_init_append_request(&self, path: &str) -> Result { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/webhdfs/v1/{}?op=APPEND&noredirect=true", self.endpoint, percent_encode_path(&p), ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += &format!("&{auth}"); } let req = Request::post(url) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let resp: LocationResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; Ok(resp.location) } _ => Err(parse_error(resp)), } } pub async fn webhdfs_rename_object(&self, from: &str, to: &str) -> Result> { let from = build_abs_path(&self.root, from); let to = build_rooted_abs_path(&self.root, to); let mut url = format!( "{}/webhdfs/v1/{}?op=RENAME&destination={}", self.endpoint, percent_encode_path(&from), percent_encode_path(&to) ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += &format!("&{auth}"); } let req = Request::put(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.client.send(req).await } pub fn webhdfs_append_request( &self, location: &str, size: u64, body: Buffer, ) -> Result> { let mut url = location.to_string(); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += &format!("&{auth}"); } let mut req = Request::post(&url); req = req.header(CONTENT_LENGTH, size.to_string()); req.body(body).map_err(new_request_build_error) } /// CONCAT will concat sources to the path pub fn webhdfs_concat_request( &self, path: &str, sources: Vec, ) -> Result> { let p = build_abs_path(&self.root, path); let sources = sources .iter() .map(|p| build_rooted_abs_path(&self.root, p)) .collect::>() .join(","); let mut url = format!( "{}/webhdfs/v1/{}?op=CONCAT&sources={}", self.endpoint, percent_encode_path(&p), percent_encode_path(&sources), ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += &format!("&{auth}"); } let req = Request::post(url); req.body(Buffer::new()).map_err(new_request_build_error) } fn webhdfs_open_request(&self, path: &str, range: &BytesRange) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/webhdfs/v1/{}?op=OPEN", self.endpoint, percent_encode_path(&p), ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += &format!("&{auth}"); } if !range.is_full() { url += &format!("&offset={}", range.offset()); if let Some(size) = range.size() { url += &format!("&length={size}") } } let req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; Ok(req) } pub async fn webhdfs_list_status_request(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/webhdfs/v1/{}?op=LISTSTATUS", self.endpoint, percent_encode_path(&p), ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += format!("&{auth}").as_str(); } let req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.client.send(req).await } pub async fn webhdfs_list_status_batch_request( &self, path: &str, start_after: &str, ) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/webhdfs/v1/{}?op=LISTSTATUS_BATCH", self.endpoint, percent_encode_path(&p), ); if !start_after.is_empty() { url += format!("&startAfter={}", start_after).as_str(); } if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += format!("&{auth}").as_str(); } let req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.client.send(req).await } pub async fn webhdfs_read_file( &self, path: &str, range: BytesRange, ) -> Result> { let req = self.webhdfs_open_request(path, &range)?; self.client.fetch(req).await } pub(super) async fn webhdfs_get_file_status(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/webhdfs/v1/{}?op=GETFILESTATUS", self.endpoint, percent_encode_path(&p), ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += format!("&{auth}").as_str(); } let req = Request::get(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.client.send(req).await } pub async fn webhdfs_delete(&self, path: &str) -> Result> { let p = build_abs_path(&self.root, path); let mut url = format!( "{}/webhdfs/v1/{}?op=DELETE&recursive=false", self.endpoint, percent_encode_path(&p), ); if let Some(user) = &self.user_name { url += format!("&user.name={user}").as_str(); } if let Some(auth) = &self.auth { url += format!("&{auth}").as_str(); } let req = Request::delete(&url) .body(Buffer::new()) .map_err(new_request_build_error)?; self.client.send(req).await } async fn check_root(&self) -> Result<()> { let resp = self.webhdfs_get_file_status("/").await?; match resp.status() { StatusCode::OK => { let bs = resp.into_body(); let file_status = serde_json::from_reader::<_, FileStatusWrapper>(bs.reader()) .map_err(new_json_deserialize_error)? .file_status; if file_status.ty == FileStatusType::File { return Err(Error::new( ErrorKind::ConfigInvalid, "root path must be dir", )); } } StatusCode::NOT_FOUND => { self.create_dir("/", OpCreateDir::new()).await?; } _ => return Err(parse_error(resp)), } Ok(()) } } impl Access for WebhdfsBackend { type Reader = HttpBody; type Writer = WebhdfsWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::Webhdfs) .set_root(&self.root) .set_native_capability(Capability { stat: true, stat_has_content_length: true, stat_has_last_modified: true, read: true, write: true, write_can_append: true, write_can_multi: self.atomic_write_dir.is_some(), create_dir: true, delete: true, list: true, list_has_content_length: true, list_has_last_modified: true, shared: true, ..Default::default() }); am.into() } /// Create a file or directory async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { let req = self.webhdfs_create_dir_request(path)?; let resp = self.client.send(req).await?; let status = resp.status(); // WebHDFS's has a two-step create/append to prevent clients to send out // data before creating it. // According to the redirect policy of `reqwest` HTTP Client we are using, // the redirection should be done automatically. match status { StatusCode::CREATED | StatusCode::OK => { let bs = resp.into_body(); let resp = serde_json::from_reader::<_, BooleanResp>(bs.reader()) .map_err(new_json_deserialize_error)?; if resp.boolean { Ok(RpCreateDir::default()) } else { Err(Error::new( ErrorKind::Unexpected, "webhdfs create dir failed", )) } } _ => Err(parse_error(resp)), } } async fn stat(&self, path: &str, _: OpStat) -> Result { // if root exists and is a directory, stat will be ok self.root_checker .get_or_try_init(|| async { self.check_root().await }) .await?; let resp = self.webhdfs_get_file_status(path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let file_status = serde_json::from_reader::<_, FileStatusWrapper>(bs.reader()) .map_err(new_json_deserialize_error)? .file_status; let meta = match file_status.ty { FileStatusType::Directory => Metadata::new(EntryMode::DIR), FileStatusType::File => Metadata::new(EntryMode::FILE) .with_content_length(file_status.length) .with_last_modified(parse_datetime_from_from_timestamp_millis( file_status.modification_time, )?), }; Ok(RpStat::new(meta)) } _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { let resp = self.webhdfs_read_file(path, args.range()).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => { Ok((RpRead::default(), resp.into_body())) } _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn write(&self, path: &str, args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let w = WebhdfsWriter::new(self.clone(), args.clone(), path.to_string()); let w = if args.append() { WebhdfsWriters::Two(oio::AppendWriter::new(w)) } else { WebhdfsWriters::One(oio::BlockWriter::new( w, args.executor().cloned(), args.concurrent(), )) }; Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(WebhdfsDeleter::new(Arc::new(self.clone()))), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { if args.recursive() { return Err(Error::new( ErrorKind::Unsupported, "WebHDFS doesn't support list with recursive", )); } let path = path.trim_end_matches('/'); let l = WebhdfsLister::new(self.clone(), path); Ok((RpList::default(), oio::PageLister::new(l))) } } #[derive(Debug, Deserialize)] #[serde(rename_all = "PascalCase")] pub(super) struct LocationResponse { pub location: String, } opendal-0.52.0/src/services/webhdfs/config.rs000064400000000000000000000034721046102023000172070ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for WebHDFS support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct WebhdfsConfig { /// Root for webhdfs. pub root: Option, /// Endpoint for webhdfs. pub endpoint: Option, /// Name of the user for webhdfs. pub user_name: Option, /// Delegation token for webhdfs. pub delegation: Option, /// Disable batch listing pub disable_list_batch: bool, /// atomic_write_dir of this backend pub atomic_write_dir: Option, } impl Debug for WebhdfsConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("WebhdfsConfig") .field("root", &self.root) .field("endpoint", &self.endpoint) .field("user_name", &self.user_name) .field("atomic_write_dir", &self.atomic_write_dir) .finish_non_exhaustive() } } opendal-0.52.0/src/services/webhdfs/delete.rs000064400000000000000000000026131046102023000172000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::backend::WebhdfsBackend; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct WebhdfsDeleter { core: Arc, } impl WebhdfsDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for WebhdfsDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.webhdfs_delete(&path).await?; match resp.status() { StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/webhdfs/docs.md000064400000000000000000000067671046102023000166600ustar 00000000000000There two implementations of WebHDFS REST API: - Native via HDFS Namenode and Datanode, data are transferred between nodes directly. - [HttpFS](https://hadoop.apache.org/docs/stable/hadoop-hdfs-httpfs/index.html) is a gateway before hdfs nodes, data are proxied. ## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [ ] copy - [ ] rename - [x] list - [ ] ~~presign~~ - [ ] blocking ## Differences with HDFS [Hdfs][crate::services::Hdfs] is powered by HDFS's native java client. Users need to set up the HDFS services correctly. But webhdfs can access from HTTP API and no extra setup needed. ## WebHDFS Compatibility Guidelines ### File Creation and Write For [File creation and write](https://hadoop.apache.org/docs/r3.1.3/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File) operations, OpenDAL WebHDFS is optimized for Hadoop Distributed File System (HDFS) versions 2.9 and later. This involves two API calls in webhdfs, where the initial `put` call to the namenode is redirected to the datanode handling the file data. The optional `noredirect` flag can be set to prevent redirection. If used, the API response body contains the datanode URL, which is then utilized for the subsequent `put` call with the actual file data. OpenDAL automatically sets the `noredirect` flag with the first `put` call. This feature is supported starting from HDFS version 2.9. ### Multi-Write Support OpenDAL WebHDFS supports multi-write operations by creating temporary files in the specified `atomic_write_dir`. The final concatenation of these temporary files occurs when the writer is closed. However, it's essential to be aware of HDFS concat restrictions for earlier versions, where the target file must not be empty, and its last block must be full. Due to these constraints, the concat operation might fail for HDFS 2.6. This issue, identified as [HDFS-6641](https://issues.apache.org/jira/browse/HDFS-6641), has been addressed in later versions of HDFS. In summary, OpenDAL WebHDFS is designed for optimal compatibility with HDFS, specifically versions 2.9 and later. ## Configurations - `root`: The root path of the WebHDFS service. - `endpoint`: The endpoint of the WebHDFS service. - `delegation`: The delegation token for WebHDFS. - `atomic_write_dir`: The tmp write dir of multi write for WebHDFS.Needs to be configured for multi write support. Refer to [`Builder`]'s public API docs for more information. ## Examples ### Via Builder ```rust,no_run use std::sync::Arc; use anyhow::Result; use opendal::services::Webhdfs; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { let mut builder = Webhdfs::default() // set the root for WebHDFS, all operations will happen under this root // // Note: // if the root exists, the builder will automatically create the // root directory for you // if the root exists and is a directory, the builder will continue working // if the root exists and is a file, the builder will fail on building backend .root("/path/to/dir") // set the endpoint of webhdfs namenode, controlled by dfs.namenode.http-address // default is http://127.0.0.1:9870 .endpoint("http://127.0.0.1:9870") // set the delegation_token for builder .delegation("delegation_token") // set atomic_write_dir for builder .atomic_write_dir(".opendal_tmp/"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/webhdfs/error.rs000064400000000000000000000076121046102023000170730ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use http::response::Parts; use http::Response; use http::StatusCode; use serde::Deserialize; use crate::raw::*; use crate::*; #[derive(Debug, Deserialize)] #[serde(rename_all = "PascalCase")] struct WebHdfsErrorWrapper { pub remote_exception: WebHdfsError, } /// WebHdfsError is the error message returned by WebHdfs service #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] #[allow(dead_code)] struct WebHdfsError { exception: String, message: String, java_class_name: String, } pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let s = String::from_utf8_lossy(&bs); parse_error_msg(parts, &s) } pub(super) fn parse_error_msg(parts: Parts, body: &str) -> Error { let (kind, retryable) = match parts.status { StatusCode::NOT_FOUND => (ErrorKind::NotFound, false), StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => (ErrorKind::PermissionDenied, false), // passing invalid arguments will return BAD_REQUEST // should be un-retryable StatusCode::BAD_REQUEST => (ErrorKind::Unexpected, false), StatusCode::INTERNAL_SERVER_ERROR | StatusCode::BAD_GATEWAY | StatusCode::SERVICE_UNAVAILABLE | StatusCode::GATEWAY_TIMEOUT => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let message = match serde_json::from_str::(body) { Ok(wh_error) => format!("{:?}", wh_error.remote_exception), Err(_) => body.to_owned(), }; let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod tests { use bytes::Buf; use serde_json::from_reader; use super::*; /// Error response example from https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Error%20Responses #[tokio::test] async fn test_parse_error() -> Result<()> { let ill_args = bytes::Bytes::from( r#" { "RemoteException": { "exception" : "IllegalArgumentException", "javaClassName": "java.lang.IllegalArgumentException", "message" : "Invalid value for webhdfs parameter \"permission\": ..." } } "#, ); let body = Buffer::from(ill_args.clone()); let resp = Response::builder() .status(StatusCode::BAD_REQUEST) .body(body) .unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), ErrorKind::Unexpected); assert!(!err.is_temporary()); let err_msg: WebHdfsError = from_reader::<_, WebHdfsErrorWrapper>(ill_args.reader()) .expect("must success") .remote_exception; assert_eq!(err_msg.exception, "IllegalArgumentException"); assert_eq!( err_msg.java_class_name, "java.lang.IllegalArgumentException" ); assert_eq!( err_msg.message, "Invalid value for webhdfs parameter \"permission\": ..." ); Ok(()) } } opendal-0.52.0/src/services/webhdfs/lister.rs000064400000000000000000000110431046102023000172350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::StatusCode; use super::backend::WebhdfsBackend; use super::error::parse_error; use super::message::*; use crate::raw::*; use crate::*; pub struct WebhdfsLister { backend: WebhdfsBackend, path: String, } impl WebhdfsLister { pub fn new(backend: WebhdfsBackend, path: &str) -> Self { Self { backend, path: path.to_string(), } } } impl oio::PageList for WebhdfsLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let file_status = if self.backend.disable_list_batch { let resp = self.backend.webhdfs_list_status_request(&self.path).await?; match resp.status() { StatusCode::OK => { ctx.done = true; ctx.entries.push_back(oio::Entry::new( format!("{}/", self.path).as_str(), Metadata::new(EntryMode::DIR), )); let bs = resp.into_body(); serde_json::from_reader::<_, FileStatusesWrapper>(bs.reader()) .map_err(new_json_deserialize_error)? .file_statuses .file_status } StatusCode::NOT_FOUND => { ctx.done = true; return Ok(()); } _ => return Err(parse_error(resp)), } } else { let resp = self .backend .webhdfs_list_status_batch_request(&self.path, &ctx.token) .await?; match resp.status() { StatusCode::OK => { let bs = resp.into_body(); let res: DirectoryListingWrapper = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; let directory_listing = res.directory_listing; let file_statuses = directory_listing.partial_listing.file_statuses.file_status; if directory_listing.remaining_entries == 0 { ctx.entries.push_back(oio::Entry::new( format!("{}/", self.path).as_str(), Metadata::new(EntryMode::DIR), )); ctx.done = true; } else if !file_statuses.is_empty() { ctx.token .clone_from(&file_statuses.last().unwrap().path_suffix); } file_statuses } StatusCode::NOT_FOUND => { ctx.done = true; return Ok(()); } _ => return Err(parse_error(resp)), } }; for status in file_status { let mut path = if self.path.is_empty() { status.path_suffix.to_string() } else { format!("{}/{}", self.path, status.path_suffix) }; let meta = match status.ty { FileStatusType::Directory => Metadata::new(EntryMode::DIR), FileStatusType::File => Metadata::new(EntryMode::FILE) .with_content_length(status.length) .with_last_modified(parse_datetime_from_from_timestamp_millis( status.modification_time, )?), }; if meta.mode().is_file() { path = path.trim_end_matches('/').to_string(); } if meta.mode().is_dir() { path += "/" } let entry = oio::Entry::new(&path, meta); ctx.entries.push_back(entry); } Ok(()) } } opendal-0.52.0/src/services/webhdfs/message.rs000064400000000000000000000163151046102023000173660ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! WebHDFS response messages use serde::Deserialize; #[derive(Debug, Deserialize)] pub(super) struct BooleanResp { pub boolean: bool, } #[derive(Debug, Deserialize)] #[serde(rename_all = "PascalCase")] pub(super) struct FileStatusWrapper { pub file_status: FileStatus, } #[derive(Debug, Deserialize)] #[serde(rename_all = "PascalCase")] pub(super) struct FileStatusesWrapper { pub file_statuses: FileStatuses, } #[derive(Debug, Deserialize)] #[serde(rename_all = "PascalCase")] pub(super) struct DirectoryListingWrapper { pub directory_listing: DirectoryListing, } #[derive(Debug, Default, Deserialize)] #[serde(rename_all = "camelCase")] pub(super) struct DirectoryListing { pub partial_listing: PartialListing, pub remaining_entries: u32, } #[derive(Debug, Default, Deserialize)] #[serde(rename_all = "PascalCase")] pub(super) struct PartialListing { pub file_statuses: FileStatuses, } #[derive(Debug, Default, Deserialize)] #[serde(rename_all = "PascalCase")] pub(super) struct FileStatuses { pub file_status: Vec, } #[derive(Debug, Default, Deserialize)] #[serde(rename_all = "camelCase")] pub struct FileStatus { pub length: u64, pub modification_time: i64, pub path_suffix: String, #[serde(rename = "type")] pub ty: FileStatusType, } #[derive(Debug, Default, Deserialize, PartialEq, Eq)] #[serde(rename_all = "UPPERCASE")] pub enum FileStatusType { Directory, #[default] File, } #[cfg(test)] mod test { use super::*; #[test] fn test_file_status() { let json = r#" { "FileStatus": { "accessTime" : 0, "blockSize" : 0, "group" : "supergroup", "length" : 0, "modificationTime": 1320173277227, "owner" : "webuser", "pathSuffix" : "", "permission" : "777", "replication" : 0, "type" : "DIRECTORY" } } "#; let status: FileStatusWrapper = serde_json::from_str(json).expect("must success"); assert_eq!(status.file_status.length, 0); assert_eq!(status.file_status.modification_time, 1320173277227); assert_eq!(status.file_status.path_suffix, ""); assert_eq!(status.file_status.ty, FileStatusType::Directory); } #[tokio::test] async fn test_list_empty() { let json = r#" { "FileStatuses": {"FileStatus":[]} } "#; let file_statuses = serde_json::from_str::(json) .expect("must success") .file_statuses .file_status; assert!(file_statuses.is_empty()); } #[tokio::test] async fn test_list_status() { let json = r#" { "FileStatuses": { "FileStatus": [ { "accessTime" : 1320171722771, "blockSize" : 33554432, "group" : "supergroup", "length" : 24930, "modificationTime": 1320171722771, "owner" : "webuser", "pathSuffix" : "a.patch", "permission" : "644", "replication" : 1, "type" : "FILE" }, { "accessTime" : 0, "blockSize" : 0, "group" : "supergroup", "length" : 0, "modificationTime": 1320895981256, "owner" : "szetszwo", "pathSuffix" : "bar", "permission" : "711", "replication" : 0, "type" : "DIRECTORY" } ] } } "#; let file_statuses = serde_json::from_str::(json) .expect("must success") .file_statuses .file_status; // we should check the value of FileStatusWrapper directly. assert_eq!(file_statuses.len(), 2); assert_eq!(file_statuses[0].length, 24930); assert_eq!(file_statuses[0].modification_time, 1320171722771); assert_eq!(file_statuses[0].path_suffix, "a.patch"); assert_eq!(file_statuses[0].ty, FileStatusType::File); assert_eq!(file_statuses[1].length, 0); assert_eq!(file_statuses[1].modification_time, 1320895981256); assert_eq!(file_statuses[1].path_suffix, "bar"); assert_eq!(file_statuses[1].ty, FileStatusType::Directory); } #[tokio::test] async fn test_list_status_batch() { let json = r#" { "DirectoryListing": { "partialListing": { "FileStatuses": { "FileStatus": [ { "accessTime": 0, "blockSize": 0, "childrenNum": 0, "fileId": 16387, "group": "supergroup", "length": 0, "modificationTime": 1473305882563, "owner": "andrew", "pathSuffix": "bardir", "permission": "755", "replication": 0, "storagePolicy": 0, "type": "DIRECTORY" }, { "accessTime": 1473305896945, "blockSize": 1024, "childrenNum": 0, "fileId": 16388, "group": "supergroup", "length": 0, "modificationTime": 1473305896965, "owner": "andrew", "pathSuffix": "bazfile", "permission": "644", "replication": 3, "storagePolicy": 0, "type": "FILE" } ] } }, "remainingEntries": 2 } } "#; let directory_listing = serde_json::from_str::(json) .expect("must success") .directory_listing; assert_eq!(directory_listing.remaining_entries, 2); assert_eq!( directory_listing .partial_listing .file_statuses .file_status .len(), 2 ); assert_eq!( directory_listing.partial_listing.file_statuses.file_status[0].path_suffix, "bardir" ); assert_eq!( directory_listing.partial_listing.file_statuses.file_status[1].path_suffix, "bazfile" ); } } opendal-0.52.0/src/services/webhdfs/mod.rs000064400000000000000000000023071046102023000165150ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-webhdfs")] mod delete; #[cfg(feature = "services-webhdfs")] mod error; #[cfg(feature = "services-webhdfs")] mod lister; #[cfg(feature = "services-webhdfs")] mod message; #[cfg(feature = "services-webhdfs")] mod writer; #[cfg(feature = "services-webhdfs")] mod backend; #[cfg(feature = "services-webhdfs")] pub use backend::WebhdfsBuilder as Webhdfs; mod config; pub use config::WebhdfsConfig; opendal-0.52.0/src/services/webhdfs/writer.rs000064400000000000000000000141241046102023000172520ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::StatusCode; use uuid::Uuid; use super::backend::WebhdfsBackend; use super::error::parse_error; use crate::raw::*; use crate::services::webhdfs::message::FileStatusWrapper; use crate::*; pub type WebhdfsWriters = TwoWays, oio::AppendWriter>; pub struct WebhdfsWriter { backend: WebhdfsBackend, op: OpWrite, path: String, } impl WebhdfsWriter { pub fn new(backend: WebhdfsBackend, op: OpWrite, path: String) -> Self { WebhdfsWriter { backend, op, path } } } impl oio::BlockWrite for WebhdfsWriter { async fn write_once(&self, size: u64, body: Buffer) -> Result { let req = self .backend .webhdfs_create_object_request(&self.path, Some(size), &self.op, body) .await?; let resp = self.backend.client.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn write_block(&self, block_id: Uuid, size: u64, body: Buffer) -> Result<()> { let Some(ref atomic_write_dir) = self.backend.atomic_write_dir else { return Err(Error::new( ErrorKind::Unsupported, "write multi is not supported when atomic is not set", )); }; let req = self .backend .webhdfs_create_object_request( &format!("{}{}", atomic_write_dir, block_id), Some(size), &self.op, body, ) .await?; let resp = self.backend.client.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(()), _ => Err(parse_error(resp)), } } async fn complete_block(&self, block_ids: Vec) -> Result { let Some(ref atomic_write_dir) = self.backend.atomic_write_dir else { return Err(Error::new( ErrorKind::Unsupported, "write multi is not supported when atomic is not set", )); }; let first_block_id = format!("{}{}", atomic_write_dir, block_ids[0].clone()); if block_ids.len() >= 2 { let sources: Vec = block_ids[1..] .iter() .map(|s| format!("{}{}", atomic_write_dir, s)) .collect(); // concat blocks let req = self .backend .webhdfs_concat_request(&first_block_id, sources)?; let resp = self.backend.client.send(req).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } } // delete the path file let resp = self.backend.webhdfs_delete(&self.path).await?; let status = resp.status(); if status != StatusCode::OK { return Err(parse_error(resp)); } // rename concat file to path let resp = self .backend .webhdfs_rename_object(&first_block_id, &self.path) .await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } async fn abort_block(&self, block_ids: Vec) -> Result<()> { for block_id in block_ids { let resp = self.backend.webhdfs_delete(&block_id.to_string()).await?; match resp.status() { StatusCode::OK => {} _ => return Err(parse_error(resp)), } } Ok(()) } } impl oio::AppendWrite for WebhdfsWriter { async fn offset(&self) -> Result { let resp = self.backend.webhdfs_get_file_status(&self.path).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let file_status = serde_json::from_reader::<_, FileStatusWrapper>(bs.reader()) .map_err(new_json_deserialize_error)? .file_status; Ok(file_status.length) } StatusCode::NOT_FOUND => { let req = self .backend .webhdfs_create_object_request(&self.path, None, &self.op, Buffer::new()) .await?; let resp = self.backend.client.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::OK => Ok(0), _ => Err(parse_error(resp)), } } _ => Err(parse_error(resp)), } } async fn append(&self, _offset: u64, size: u64, body: Buffer) -> Result { let location = self.backend.webhdfs_init_append_request(&self.path).await?; let req = self.backend.webhdfs_append_request(&location, size, body)?; let resp = self.backend.client.send(req).await?; let status = resp.status(); match status { StatusCode::OK => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/yandex_disk/backend.rs000064400000000000000000000211471046102023000202100ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::sync::Arc; use bytes::Buf; use http::header; use http::Request; use http::Response; use http::StatusCode; use log::debug; use super::core::*; use super::delete::YandexDiskDeleter; use super::error::parse_error; use super::lister::YandexDiskLister; use super::writer::YandexDiskWriter; use super::writer::YandexDiskWriters; use crate::raw::*; use crate::services::YandexDiskConfig; use crate::*; impl Configurator for YandexDiskConfig { type Builder = YandexDiskBuilder; fn into_builder(self) -> Self::Builder { YandexDiskBuilder { config: self, http_client: None, } } } /// [YandexDisk](https://360.yandex.com/disk/) services support. #[doc = include_str!("docs.md")] #[derive(Default)] pub struct YandexDiskBuilder { config: YandexDiskConfig, http_client: Option, } impl Debug for YandexDiskBuilder { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut d = f.debug_struct("YandexDiskBuilder"); d.field("config", &self.config); d.finish_non_exhaustive() } } impl YandexDiskBuilder { /// Set root of this backend. /// /// All operations will happen under this root. pub fn root(mut self, root: &str) -> Self { self.config.root = if root.is_empty() { None } else { Some(root.to_string()) }; self } /// yandex disk oauth access_token. /// The valid token will looks like `y0_XXXXXXqihqIWAADLWwAAAAD3IXXXXXX0gtVeSPeIKM0oITMGhXXXXXX`. /// We can fetch the debug token from . /// To use it in production, please register an app at instead. pub fn access_token(mut self, access_token: &str) -> Self { self.config.access_token = access_token.to_string(); self } /// Specify the http client that used by this service. /// /// # Notes /// /// This API is part of OpenDAL's Raw API. `HttpClient` could be changed /// during minor updates. pub fn http_client(mut self, client: HttpClient) -> Self { self.http_client = Some(client); self } } impl Builder for YandexDiskBuilder { const SCHEME: Scheme = Scheme::YandexDisk; type Config = YandexDiskConfig; /// Builds the backend and returns the result of YandexDiskBackend. fn build(self) -> Result { debug!("backend build started: {:?}", &self); let root = normalize_root(&self.config.root.clone().unwrap_or_default()); debug!("backend use root {}", &root); // Handle oauth access_token. if self.config.access_token.is_empty() { return Err( Error::new(ErrorKind::ConfigInvalid, "access_token is empty") .with_operation("Builder::build") .with_context("service", Scheme::YandexDisk), ); } let client = if let Some(client) = self.http_client { client } else { HttpClient::new().map_err(|err| { err.with_operation("Builder::build") .with_context("service", Scheme::YandexDisk) })? }; Ok(YandexDiskBackend { core: Arc::new(YandexDiskCore { root, access_token: self.config.access_token.clone(), client, }), }) } } /// Backend for YandexDisk services. #[derive(Debug, Clone)] pub struct YandexDiskBackend { core: Arc, } impl Access for YandexDiskBackend { type Reader = HttpBody; type Writer = YandexDiskWriters; type Lister = oio::PageLister; type Deleter = oio::OneShotDeleter; type BlockingReader = (); type BlockingWriter = (); type BlockingLister = (); type BlockingDeleter = (); fn info(&self) -> Arc { let mut am = AccessorInfo::default(); am.set_scheme(Scheme::YandexDisk) .set_root(&self.core.root) .set_native_capability(Capability { stat: true, stat_has_last_modified: true, stat_has_content_md5: true, stat_has_content_type: true, stat_has_content_length: true, create_dir: true, read: true, write: true, write_can_empty: true, delete: true, rename: true, copy: true, list: true, list_with_limit: true, list_has_last_modified: true, list_has_content_md5: true, list_has_content_type: true, list_has_content_length: true, shared: true, ..Default::default() }); am.into() } async fn create_dir(&self, path: &str, _: OpCreateDir) -> Result { self.core.ensure_dir_exists(path).await?; Ok(RpCreateDir::default()) } async fn rename(&self, from: &str, to: &str, _args: OpRename) -> Result { self.core.ensure_dir_exists(to).await?; let resp = self.core.move_object(from, to).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => Ok(RpRename::default()), _ => Err(parse_error(resp)), } } async fn copy(&self, from: &str, to: &str, _args: OpCopy) -> Result { self.core.ensure_dir_exists(to).await?; let resp = self.core.copy(from, to).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::CREATED => Ok(RpCopy::default()), _ => Err(parse_error(resp)), } } async fn read(&self, path: &str, args: OpRead) -> Result<(RpRead, Self::Reader)> { // TODO: move this out of reader. let download_url = self.core.get_download_url(path).await?; let req = Request::get(download_url) .header(header::RANGE, args.range().to_header()) .body(Buffer::new()) .map_err(new_request_build_error)?; let resp = self.core.client.fetch(req).await?; let status = resp.status(); match status { StatusCode::OK | StatusCode::PARTIAL_CONTENT => Ok((RpRead::new(), resp.into_body())), _ => { let (part, mut body) = resp.into_parts(); let buf = body.to_buffer().await?; Err(parse_error(Response::from_parts(part, buf))) } } } async fn stat(&self, path: &str, _args: OpStat) -> Result { let resp = self.core.metainformation(path, None, None).await?; let status = resp.status(); match status { StatusCode::OK => { let bs = resp.into_body(); let mf: MetainformationResponse = serde_json::from_reader(bs.reader()).map_err(new_json_deserialize_error)?; parse_info(mf).map(RpStat::new) } _ => Err(parse_error(resp)), } } async fn write(&self, path: &str, _args: OpWrite) -> Result<(RpWrite, Self::Writer)> { let writer = YandexDiskWriter::new(self.core.clone(), path.to_string()); let w = oio::OneShotWriter::new(writer); Ok((RpWrite::default(), w)) } async fn delete(&self) -> Result<(RpDelete, Self::Deleter)> { Ok(( RpDelete::default(), oio::OneShotDeleter::new(YandexDiskDeleter::new(self.core.clone())), )) } async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Lister)> { let l = YandexDiskLister::new(self.core.clone(), path, args.limit()); Ok((RpList::default(), oio::PageLister::new(l))) } } opendal-0.52.0/src/services/yandex_disk/config.rs000064400000000000000000000027041046102023000200640ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use serde::Deserialize; use serde::Serialize; /// Config for YandexDisk services support. #[derive(Default, Serialize, Deserialize, Clone, PartialEq, Eq)] #[serde(default)] #[non_exhaustive] pub struct YandexDiskConfig { /// root of this backend. /// /// All operations will happen under this root. pub root: Option, /// yandex disk oauth access_token. pub access_token: String, } impl Debug for YandexDiskConfig { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut ds = f.debug_struct("Config"); ds.field("root", &self.root); ds.finish() } } opendal-0.52.0/src/services/yandex_disk/core.rs000064400000000000000000000202021046102023000175400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use bytes::Buf; use http::header; use http::request; use http::Request; use http::Response; use http::StatusCode; use serde::Deserialize; use super::error::parse_error; use crate::raw::*; use crate::*; #[derive(Clone)] pub struct YandexDiskCore { /// The root of this core. pub root: String, /// Yandex Disk oauth access_token. pub access_token: String, pub client: HttpClient, } impl Debug for YandexDiskCore { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("Backend") .field("root", &self.root) .finish_non_exhaustive() } } impl YandexDiskCore { #[inline] pub async fn send(&self, req: Request) -> Result> { self.client.send(req).await } #[inline] pub fn sign(&self, req: request::Builder) -> request::Builder { req.header( header::AUTHORIZATION, format!("OAuth {}", self.access_token), ) } } impl YandexDiskCore { /// Get upload url. pub async fn get_upload_url(&self, path: &str) -> Result { let path = build_rooted_abs_path(&self.root, path); let url = format!( "https://cloud-api.yandex.net/v1/disk/resources/upload?path={}&overwrite=true", percent_encode_path(&path) ); let req = Request::get(url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let bytes = resp.into_body(); let resp: GetUploadUrlResponse = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; Ok(resp.href) } _ => Err(parse_error(resp)), } } pub async fn get_download_url(&self, path: &str) -> Result { let path = build_rooted_abs_path(&self.root, path); let url = format!( "https://cloud-api.yandex.net/v1/disk/resources/download?path={}&overwrite=true", percent_encode_path(&path) ); let req = Request::get(url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; let resp = self.send(req).await?; let status = resp.status(); match status { StatusCode::OK => { let bytes = resp.into_body(); let resp: GetUploadUrlResponse = serde_json::from_reader(bytes.reader()).map_err(new_json_deserialize_error)?; Ok(resp.href) } _ => Err(parse_error(resp)), } } pub async fn ensure_dir_exists(&self, path: &str) -> Result<()> { let path = build_abs_path(&self.root, path); let paths = path.split('/').collect::>(); for i in 0..paths.len() - 1 { let path = paths[..i + 1].join("/"); let resp = self.create_dir(&path).await?; let status = resp.status(); match status { StatusCode::CREATED | StatusCode::CONFLICT => {} _ => return Err(parse_error(resp)), } } Ok(()) } pub async fn create_dir(&self, path: &str) -> Result> { let url = format!( "https://cloud-api.yandex.net/v1/disk/resources?path=/{}", percent_encode_path(path), ); let req = Request::put(url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn copy(&self, from: &str, to: &str) -> Result> { let from = build_rooted_abs_path(&self.root, from); let to = build_rooted_abs_path(&self.root, to); let url = format!( "https://cloud-api.yandex.net/v1/disk/resources/copy?from={}&path={}&overwrite=true", percent_encode_path(&from), percent_encode_path(&to) ); let req = Request::post(url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn move_object(&self, from: &str, to: &str) -> Result> { let from = build_rooted_abs_path(&self.root, from); let to = build_rooted_abs_path(&self.root, to); let url = format!( "https://cloud-api.yandex.net/v1/disk/resources/move?from={}&path={}&overwrite=true", percent_encode_path(&from), percent_encode_path(&to) ); let req = Request::post(url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn delete(&self, path: &str) -> Result> { let path = build_rooted_abs_path(&self.root, path); let url = format!( "https://cloud-api.yandex.net/v1/disk/resources?path={}&permanently=true", percent_encode_path(&path), ); let req = Request::delete(url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } pub async fn metainformation( &self, path: &str, limit: Option, offset: Option, ) -> Result> { let path = build_rooted_abs_path(&self.root, path); let mut url = format!( "https://cloud-api.yandex.net/v1/disk/resources?path={}", percent_encode_path(&path), ); if let Some(limit) = limit { url = format!("{}&limit={}", url, limit); } if let Some(offset) = offset { url = format!("{}&offset={}", url, offset); } let req = Request::get(url); let req = self.sign(req); // Set body let req = req.body(Buffer::new()).map_err(new_request_build_error)?; self.send(req).await } } pub(super) fn parse_info(mf: MetainformationResponse) -> Result { let mode = if mf.ty == "file" { EntryMode::FILE } else { EntryMode::DIR }; let mut m = Metadata::new(mode); m.set_last_modified(parse_datetime_from_rfc3339(&mf.modified)?); if let Some(md5) = mf.md5 { m.set_content_md5(&md5); } if let Some(mime_type) = mf.mime_type { m.set_content_type(&mime_type); } if let Some(size) = mf.size { m.set_content_length(size); } Ok(m) } #[derive(Debug, Deserialize)] pub struct GetUploadUrlResponse { pub href: String, } #[derive(Debug, Deserialize)] pub struct MetainformationResponse { #[serde(rename = "type")] pub ty: String, pub path: String, pub modified: String, pub md5: Option, pub mime_type: Option, pub size: Option, #[serde(rename = "_embedded")] pub embedded: Option, } #[derive(Debug, Deserialize)] pub struct Embedded { pub total: usize, pub items: Vec, } opendal-0.52.0/src/services/yandex_disk/delete.rs000064400000000000000000000034331046102023000200610ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::core::*; use super::error::parse_error; use crate::raw::*; use crate::*; use http::StatusCode; use std::sync::Arc; pub struct YandexDiskDeleter { core: Arc, } impl YandexDiskDeleter { pub fn new(core: Arc) -> Self { Self { core } } } impl oio::OneShotDelete for YandexDiskDeleter { async fn delete_once(&self, path: String, _: OpDelete) -> Result<()> { let resp = self.core.delete(&path).await?; let status = resp.status(); match status { StatusCode::OK => Ok(()), StatusCode::NO_CONTENT => Ok(()), // Yandex Disk deleting a non-empty folder can take an unknown amount of time, // So the API responds with the code 202 Accepted (the deletion process has started). StatusCode::ACCEPTED => Ok(()), // Allow 404 when deleting a non-existing object StatusCode::NOT_FOUND => Ok(()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/services/yandex_disk/docs.md000064400000000000000000000014571046102023000175270ustar 00000000000000## Capabilities This service can be used to: - [x] stat - [x] read - [x] write - [x] create_dir - [x] delete - [x] copy - [x] rename - [x] list - [ ] presign - [ ] blocking ## Configuration - `root`: Set the work directory for backend - `access_token` YandexDisk oauth access_token You can refer to [`YandexDiskBuilder`]'s docs for more information ## Example ### Via Builder ```rust,no_run use anyhow::Result; use opendal::services::YandexDisk; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // create backend builder let mut builder = YandexDisk::default() // set the storage bucket for OpenDAL .root("/") // set the access_token for OpenDAL .access_token("test"); let op: Operator = Operator::new(builder)?.finish(); Ok(()) } ``` opendal-0.52.0/src/services/yandex_disk/error.rs000064400000000000000000000063301046102023000177470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use bytes::Buf; use http::Response; use quick_xml::de; use serde::Deserialize; use crate::raw::*; use crate::*; /// YandexDiskError is the error returned by YandexDisk service. #[derive(Default, Debug, Deserialize)] #[allow(unused)] struct YandexDiskError { message: String, description: String, error: String, } /// Parse error response into Error. pub(super) fn parse_error(resp: Response) -> Error { let (parts, body) = resp.into_parts(); let bs = body.to_bytes(); let (kind, retryable) = match parts.status.as_u16() { 410 | 403 => (ErrorKind::PermissionDenied, false), 404 => (ErrorKind::NotFound, false), // We should retry it when we get 423 error. 423 => (ErrorKind::RateLimited, true), 499 => (ErrorKind::Unexpected, true), 503 | 507 => (ErrorKind::Unexpected, true), _ => (ErrorKind::Unexpected, false), }; let (message, _yandex_disk_err) = de::from_reader::<_, YandexDiskError>(bs.clone().reader()) .map(|yandex_disk_err| (format!("{yandex_disk_err:?}"), Some(yandex_disk_err))) .unwrap_or_else(|_| (String::from_utf8_lossy(&bs).into_owned(), None)); let mut err = Error::new(kind, message); err = with_error_response_context(err, parts); if retryable { err = err.set_temporary(); } err } #[cfg(test)] mod test { use http::StatusCode; use super::*; #[tokio::test] async fn test_parse_error() { let err_res = vec![ ( r#"{ "message": "Не удалось найти запрошенный ресурс.", "description": "Resource not found.", "error": "DiskNotFoundError" }"#, ErrorKind::NotFound, StatusCode::NOT_FOUND, ), ( r#"{ "message": "Не авторизован.", "description": "Unauthorized", "error": "UnauthorizedError" }"#, ErrorKind::PermissionDenied, StatusCode::FORBIDDEN, ), ]; for res in err_res { let bs = bytes::Bytes::from(res.0); let body = Buffer::from(bs); let resp = Response::builder().status(res.2).body(body).unwrap(); let err = parse_error(resp); assert_eq!(err.kind(), res.1); } } } opendal-0.52.0/src/services/yandex_disk/lister.rs000064400000000000000000000064751046102023000201320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use super::core::parse_info; use super::core::MetainformationResponse; use super::core::YandexDiskCore; use super::error::parse_error; use crate::raw::oio::Entry; use crate::raw::*; use crate::Result; pub struct YandexDiskLister { core: Arc, path: String, limit: Option, } impl YandexDiskLister { pub(super) fn new(core: Arc, path: &str, limit: Option) -> Self { YandexDiskLister { core, path: path.to_string(), limit, } } } impl oio::PageList for YandexDiskLister { async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> { let offset = if ctx.token.is_empty() { None } else { Some(ctx.token.clone()) }; let resp = self .core .metainformation(&self.path, self.limit, offset) .await?; if resp.status() == http::StatusCode::NOT_FOUND { ctx.done = true; return Ok(()); } match resp.status() { http::StatusCode::OK => { let body = resp.into_body(); let resp: MetainformationResponse = serde_json::from_reader(body.reader()).map_err(new_json_deserialize_error)?; if let Some(embedded) = resp.embedded { let n = embedded.items.len(); for mf in embedded.items { let path = mf.path.strip_prefix("disk:"); if let Some(path) = path { let mut path = build_rel_path(&self.core.root, path); let md = parse_info(mf)?; if md.mode().is_dir() { path = format!("{}/", path); } ctx.entries.push_back(Entry::new(&path, md)); }; } let current_len = ctx.token.parse::().unwrap_or(0) + n; if current_len >= embedded.total { ctx.done = true; } ctx.token = current_len.to_string(); return Ok(()); } } http::StatusCode::NOT_FOUND => { ctx.done = true; return Ok(()); } _ => { return Err(parse_error(resp)); } } Ok(()) } } opendal-0.52.0/src/services/yandex_disk/mod.rs000064400000000000000000000023511046102023000173740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[cfg(feature = "services-yandex-disk")] mod core; #[cfg(feature = "services-yandex-disk")] mod delete; #[cfg(feature = "services-yandex-disk")] mod error; #[cfg(feature = "services-yandex-disk")] mod lister; #[cfg(feature = "services-yandex-disk")] mod writer; #[cfg(feature = "services-yandex-disk")] mod backend; #[cfg(feature = "services-yandex-disk")] pub use backend::YandexDiskBuilder as YandexDisk; mod config; pub use config::YandexDiskConfig; opendal-0.52.0/src/services/yandex_disk/writer.rs000064400000000000000000000034501046102023000201320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use http::Request; use http::StatusCode; use super::core::YandexDiskCore; use super::error::parse_error; use crate::raw::*; use crate::*; pub type YandexDiskWriters = oio::OneShotWriter; pub struct YandexDiskWriter { core: Arc, path: String, } impl YandexDiskWriter { pub fn new(core: Arc, path: String) -> Self { YandexDiskWriter { core, path } } } impl oio::OneShotWrite for YandexDiskWriter { async fn write_once(&self, bs: Buffer) -> Result { self.core.ensure_dir_exists(&self.path).await?; let upload_url = self.core.get_upload_url(&self.path).await?; let req = Request::put(upload_url) .body(bs) .map_err(new_request_build_error)?; let resp = self.core.send(req).await?; let status = resp.status(); match status { StatusCode::CREATED => Ok(Metadata::default()), _ => Err(parse_error(resp)), } } } opendal-0.52.0/src/types/blocking_read/blocking_reader.rs000064400000000000000000000113571046102023000215370ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::Bound; use std::ops::Range; use std::ops::RangeBounds; use std::sync::Arc; use bytes::BufMut; use crate::raw::*; use crate::*; /// BlockingReader is designed to read data from given path in an blocking /// manner. #[derive(Clone)] pub struct BlockingReader { ctx: Arc, /// Total size of the reader. size: Arc, } impl BlockingReader { /// Create a new blocking reader. /// /// Create will use internal information to decide the most suitable /// implementation for users. /// /// We don't want to expose those details to users so keep this function /// in crate only. pub(crate) fn new(ctx: ReadContext) -> Self { BlockingReader { ctx: Arc::new(ctx), size: Arc::new(AtomicContentLength::new()), } } /// Parse users input range bounds into valid `Range`. /// /// To avoid duplicated stat call, we will cache the size of the reader. fn parse_range(&self, range: impl RangeBounds) -> Result> { let start = match range.start_bound() { Bound::Included(v) => *v, Bound::Excluded(v) => v + 1, Bound::Unbounded => 0, }; let end = match range.end_bound() { Bound::Included(v) => v + 1, Bound::Excluded(v) => *v, Bound::Unbounded => match self.size.load() { Some(v) => v, None => { let size = self .ctx .accessor() .blocking_stat(self.ctx.path(), OpStat::new())? .into_metadata() .content_length(); self.size.store(size); size } }, }; Ok(start..end) } /// Read give range from reader into [`Buffer`]. /// /// This operation is zero-copy, which means it keeps the [`bytes::Bytes`] returned by underlying /// storage services without any extra copy or intensive memory allocations. /// /// # Notes /// /// - Buffer length smaller than range means we have reached the end of file. pub fn read(&self, range: impl RangeBounds) -> Result { let mut bufs = vec![]; for buf in self.clone().into_iterator(range)? { bufs.push(buf?) } Ok(bufs.into_iter().flatten().collect()) } /// /// This operation will copy and write bytes into given [`BufMut`]. Allocation happens while /// [`BufMut`] doesn't have enough space. /// /// # Notes /// /// - Returning length smaller than range means we have reached the end of file. pub fn read_into(&self, buf: &mut impl BufMut, range: impl RangeBounds) -> Result { let mut iter = self.clone().into_iterator(range)?; let mut read = 0; loop { let Some(bs) = iter.next().transpose()? else { return Ok(read); }; read += bs.len(); buf.put(bs); } } /// Create a buffer iterator to read specific range from given reader. fn into_iterator(self, range: impl RangeBounds) -> Result { let range = self.parse_range(range)?; Ok(BufferIterator::new(self.ctx, range)) } /// Convert reader into [`StdReader`] which implements [`futures::AsyncRead`], /// [`futures::AsyncSeek`] and [`futures::AsyncBufRead`]. #[inline] pub fn into_std_read(self, range: impl RangeBounds) -> Result { let range = self.parse_range(range)?; Ok(StdReader::new(self.ctx, range)) } /// Convert reader into [`StdBytesIterator`] which implements [`Iterator`]. #[inline] pub fn into_bytes_iterator(self, range: impl RangeBounds) -> Result { let range = self.parse_range(range)?; Ok(StdBytesIterator::new(self.ctx, range)) } } opendal-0.52.0/src/types/blocking_read/buffer_iterator.rs000064400000000000000000000052121046102023000216000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ops::RangeBounds; use std::sync::Arc; use crate::raw::*; use crate::Buffer; use crate::*; struct IteratingReader { generator: ReadGenerator, reader: Option, } impl IteratingReader { /// Create a new iterating reader. #[inline] fn new(ctx: Arc, range: BytesRange) -> Self { let generator = ReadGenerator::new(ctx.clone(), range.offset(), range.size()); Self { generator, reader: None, } } } impl oio::BlockingRead for IteratingReader { fn read(&mut self) -> Result { loop { if self.reader.is_none() { self.reader = self.generator.next_blocking_reader()?; } let Some(r) = self.reader.as_mut() else { return Ok(Buffer::new()); }; let buf = r.read()?; // Reset reader to None if this reader returns empty buffer. if buf.is_empty() { self.reader = None; continue; } else { return Ok(buf); } } } } /// BufferIterator is an iterator of buffers. /// /// # TODO /// /// We can support chunked reader for concurrent read in the future. pub struct BufferIterator { inner: IteratingReader, } impl BufferIterator { /// Create a new buffer iterator. #[inline] pub fn new(ctx: Arc, range: impl RangeBounds) -> Self { Self { inner: IteratingReader::new(ctx, range.into()), } } } impl Iterator for BufferIterator { type Item = Result; fn next(&mut self) -> Option { use oio::BlockingRead; match self.inner.read() { Ok(buf) if buf.is_empty() => None, Ok(buf) => Some(Ok(buf)), Err(err) => Some(Err(err)), } } } opendal-0.52.0/src/types/blocking_read/mod.rs000064400000000000000000000020701046102023000171740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[allow(clippy::module_inception)] mod blocking_reader; pub use blocking_reader::BlockingReader; mod buffer_iterator; pub use buffer_iterator::BufferIterator; mod std_bytes_iterator; pub use std_bytes_iterator::StdBytesIterator; mod std_reader; pub use std_reader::StdReader; opendal-0.52.0/src/types/blocking_read/std_bytes_iterator.rs000064400000000000000000000041061046102023000223300ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io; use std::ops::Range; use std::sync::Arc; use bytes::Bytes; use crate::raw::*; use crate::Buffer; use crate::BufferIterator; use crate::*; /// StdIterator is the adapter of [`Iterator`] for [`BlockingReader`][crate::BlockingReader]. /// /// Users can use this adapter in cases where they need to use [`Iterator`] trait. /// /// StdIterator also implements [`Send`] and [`Sync`]. pub struct StdBytesIterator { iter: BufferIterator, buf: Buffer, } impl StdBytesIterator { /// NOTE: don't allow users to create StdIterator directly. #[inline] pub(crate) fn new(ctx: Arc, range: Range) -> Self { let iter = BufferIterator::new(ctx, range); StdBytesIterator { iter, buf: Buffer::new(), } } } impl Iterator for StdBytesIterator { type Item = io::Result; fn next(&mut self) -> Option { loop { // Consume current buffer if let Some(bs) = Iterator::next(&mut self.buf) { return Some(Ok(bs)); } self.buf = match self.iter.next() { Some(Ok(buf)) => buf, Some(Err(err)) => return Some(Err(format_std_io_error(err))), None => return None, }; } } } opendal-0.52.0/src/types/blocking_read/std_reader.rs000064400000000000000000000101201046102023000205240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io; use std::io::BufRead; use std::io::Read; use std::io::Seek; use std::io::SeekFrom; use std::ops::Range; use std::sync::Arc; use bytes::Buf; use crate::raw::*; use crate::*; /// StdReader is the adapter of [`Read`], [`Seek`] and [`BufRead`] for [`BlockingReader`][crate::BlockingReader]. /// /// Users can use this adapter in cases where they need to use [`Read`] or [`BufRead`] trait. /// /// StdReader also implements [`Send`] and [`Sync`]. pub struct StdReader { ctx: Arc, iter: BufferIterator, buf: Buffer, start: u64, end: u64, pos: u64, } impl StdReader { /// NOTE: don't allow users to create StdReader directly. #[inline] pub(super) fn new(ctx: Arc, range: Range) -> Self { let (start, end) = (range.start, range.end); let iter = BufferIterator::new(ctx.clone(), range); Self { ctx, iter, buf: Buffer::new(), start, end, pos: 0, } } } impl BufRead for StdReader { fn fill_buf(&mut self) -> io::Result<&[u8]> { loop { if self.buf.has_remaining() { return Ok(self.buf.chunk()); } self.buf = match self.iter.next().transpose().map_err(format_std_io_error)? { Some(buf) => buf, None => return Ok(&[]), }; } } fn consume(&mut self, amt: usize) { self.buf.advance(amt); // Make sure buf has been dropped before starting new request. // Otherwise, we will hold those bytes in memory until next // buffer reaching. if self.buf.is_empty() { self.buf = Buffer::new(); } self.pos += amt as u64; } } impl Read for StdReader { #[inline] fn read(&mut self, buf: &mut [u8]) -> io::Result { loop { if self.buf.remaining() > 0 { let size = self.buf.remaining().min(buf.len()); self.buf.copy_to_slice(&mut buf[..size]); self.pos += size as u64; return Ok(size); } self.buf = match self.iter.next() { Some(Ok(buf)) => buf, Some(Err(err)) => return Err(format_std_io_error(err)), None => return Ok(0), }; } } } impl Seek for StdReader { #[inline] fn seek(&mut self, pos: SeekFrom) -> io::Result { let new_pos = match pos { SeekFrom::Start(pos) => pos as i64, SeekFrom::End(pos) => self.end as i64 - self.start as i64 + pos, SeekFrom::Current(pos) => self.pos as i64 + pos, }; // Check if new_pos is negative. if new_pos < 0 { return Err(io::Error::new( io::ErrorKind::InvalidInput, "invalid seek to a negative position", )); } let new_pos = new_pos as u64; if (self.pos..self.pos + self.buf.remaining() as u64).contains(&new_pos) { let cnt = new_pos - self.pos; self.buf.advance(cnt as _); } else { self.buf = Buffer::new(); self.iter = BufferIterator::new(self.ctx.clone(), new_pos + self.start..self.end); } self.pos = new_pos; Ok(self.pos) } } opendal-0.52.0/src/types/blocking_write/blocking_writer.rs000064400000000000000000000060101046102023000220160ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use crate::raw::*; use crate::*; /// BlockingWriter is designed to write data into given path in an blocking /// manner. pub struct BlockingWriter { /// Keep a reference to write context in writer. _ctx: Arc, inner: WriteGenerator, } impl BlockingWriter { /// Create a new writer. /// /// Create will use internal information to decide the most suitable /// implementation for users. /// /// We don't want to expose those details to users so keep this function /// in crate only. pub(crate) fn new(ctx: WriteContext) -> Result { let ctx = Arc::new(ctx); let inner = WriteGenerator::blocking_create(ctx.clone())?; Ok(Self { _ctx: ctx, inner }) } /// Write [`Buffer`] into writer. /// /// This operation will write all data in given buffer into writer. /// /// ## Examples /// /// ``` /// use bytes::Bytes; /// use opendal::BlockingOperator; /// use opendal::Result; /// /// async fn test(op: BlockingOperator) -> Result<()> { /// let mut w = op.writer("hello.txt")?; /// // Buffer can be created from continues bytes. /// w.write("hello, world")?; /// // Buffer can also be created from non-continues bytes. /// w.write(vec![Bytes::from("hello,"), Bytes::from("world!")])?; /// /// // Make sure file has been written completely. /// w.close()?; /// Ok(()) /// } /// ``` pub fn write(&mut self, bs: impl Into) -> Result<()> { let mut bs = bs.into(); while !bs.is_empty() { let n = self.inner.write(bs.clone())?; bs.advance(n); } Ok(()) } /// Close the writer and make sure all data have been committed. /// /// ## Notes /// /// Close should only be called when the writer is not closed or /// aborted, otherwise an unexpected error could be returned. pub fn close(&mut self) -> Result { self.inner.close() } /// Convert writer into [`StdWriter`] which implements [`std::io::Write`], pub fn into_std_write(self) -> StdWriter { StdWriter::new(self.inner) } } opendal-0.52.0/src/types/blocking_write/mod.rs000064400000000000000000000016201046102023000174130ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod blocking_writer; pub use blocking_writer::BlockingWriter; mod std_writer; pub use std_writer::StdWriter; opendal-0.52.0/src/types/blocking_write/std_writer.rs000064400000000000000000000071571046102023000210350ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io::Write; use crate::raw::*; use crate::*; /// StdWriter is the adapter of [`std::io::Write`] for [`BlockingWriter`]. /// /// Users can use this adapter in cases where they need to use [`std::io::Write`] related trait. /// /// # Notes /// /// Files are automatically closed when they go out of scope. Errors detected on closing are ignored /// by the implementation of Drop. Use the method `close` if these errors must be manually handled. pub struct StdWriter { w: Option>, buf: oio::FlexBuf, } impl StdWriter { /// NOTE: don't allow users to create directly. #[inline] pub(crate) fn new(w: WriteGenerator) -> Self { StdWriter { w: Some(w), buf: oio::FlexBuf::new(256 * 1024), } } /// Close the internal writer and make sure all data have been stored. pub fn close(&mut self) -> std::io::Result<()> { // Make sure all cache has been flushed. self.flush()?; let Some(w) = &mut self.w else { return Err(std::io::Error::new( std::io::ErrorKind::Other, "writer has been closed", )); }; w.close() .map_err(|err| std::io::Error::new(std::io::ErrorKind::Other, err))?; // Drop writer after close succeed; self.w = None; Ok(()) } } impl Write for StdWriter { fn write(&mut self, buf: &[u8]) -> std::io::Result { let Some(w) = &mut self.w else { return Err(std::io::Error::new( std::io::ErrorKind::Other, "writer has been closed", )); }; loop { let n = self.buf.put(buf); if n > 0 { return Ok(n); } let bs = self.buf.get().expect("frozen buffer must be valid"); let n = w .write(Buffer::from(bs)) .map_err(|err| std::io::Error::new(std::io::ErrorKind::Other, err))?; self.buf.advance(n); } } fn flush(&mut self) -> std::io::Result<()> { let Some(w) = &mut self.w else { return Err(std::io::Error::new( std::io::ErrorKind::Other, "writer has been closed", )); }; loop { // Make sure buf has been frozen. self.buf.freeze(); let Some(bs) = self.buf.get() else { return Ok(()); }; w.write(Buffer::from(bs)) .map_err(|err| std::io::Error::new(std::io::ErrorKind::Other, err))?; self.buf.clean(); } } } impl Drop for StdWriter { fn drop(&mut self) { if let Some(mut w) = self.w.take() { // Ignore error happens in close. let _ = w.close(); } } } opendal-0.52.0/src/types/buffer.rs000064400000000000000000000536731046102023000151220ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::VecDeque; use std::convert::Infallible; use std::fmt::Debug; use std::fmt::Formatter; use std::io::IoSlice; use std::mem; use std::ops::Bound; use std::ops::RangeBounds; use std::pin::Pin; use std::sync::Arc; use std::task::Context; use std::task::Poll; use bytes::Buf; use bytes::BufMut; use bytes::Bytes; use bytes::BytesMut; use futures::Stream; use crate::*; /// Buffer is a wrapper of contiguous `Bytes` and non-contiguous `[Bytes]`. /// /// We designed buffer to allow underlying storage to return non-contiguous bytes. For example, /// http based storage like s3 could generate non-contiguous bytes by stream. /// /// ## Features /// /// - [`Buffer`] can be used as [`Buf`], [`Iterator`], [`Stream`] directly. /// - [`Buffer`] is cheap to clone like [`Bytes`], only update reference count, no allocation. /// - [`Buffer`] is vectorized write friendly, you can convert it to [`IoSlice`] for vectored write. /// /// ## Examples /// /// ### As `Buf` /// /// `Buffer` implements `Buf` trait: /// /// ```rust /// use bytes::Buf; /// use opendal::Buffer; /// use serde_json; /// /// fn test(mut buf: Buffer) -> Vec { /// serde_json::from_reader(buf.reader()).unwrap() /// } /// ``` /// /// ### As Bytes `Iterator` /// /// `Buffer` implements `Iterator` trait: /// /// ```rust /// use bytes::Bytes; /// use opendal::Buffer; /// /// fn test(mut buf: Buffer) -> Vec { /// buf.into_iter().collect() /// } /// ``` /// /// ### As Bytes `Stream` /// /// `Buffer` implements `Stream>` trait: /// /// ```rust /// use bytes::Bytes; /// use futures::TryStreamExt; /// use opendal::Buffer; /// /// async fn test(mut buf: Buffer) -> Vec { /// buf.into_iter().try_collect().await.unwrap() /// } /// ``` /// /// ### As one contiguous Bytes /// /// `Buffer` can make contiguous by transform into `Bytes` or `Vec`. /// Please keep in mind that this operation involves new allocation and bytes copy, and we can't /// reuse the same memory region anymore. /// /// ```rust /// use bytes::Bytes; /// use opendal::Buffer; /// /// fn test_to_vec(buf: Buffer) -> Vec { /// buf.to_vec() /// } /// /// fn test_to_bytes(buf: Buffer) -> Bytes { /// buf.to_bytes() /// } /// ``` #[derive(Clone)] pub struct Buffer(Inner); #[derive(Clone)] enum Inner { Contiguous(Bytes), NonContiguous { parts: Arc<[Bytes]>, size: usize, idx: usize, offset: usize, }, } impl Debug for Buffer { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { let mut b = f.debug_struct("Buffer"); match &self.0 { Inner::Contiguous(bs) => { b.field("type", &"contiguous"); b.field("size", &bs.len()); } Inner::NonContiguous { parts, size, idx, offset, } => { b.field("type", &"non_contiguous"); b.field("parts", &parts); b.field("size", &size); b.field("idx", &idx); b.field("offset", &offset); } } b.finish_non_exhaustive() } } impl Default for Buffer { fn default() -> Self { Self::new() } } impl Buffer { /// Create a new empty buffer. /// /// This operation is const and no allocation will be performed. #[inline] pub const fn new() -> Self { Self(Inner::Contiguous(Bytes::new())) } /// Get the length of the buffer. #[inline] pub fn len(&self) -> usize { match &self.0 { Inner::Contiguous(b) => b.remaining(), Inner::NonContiguous { size, .. } => *size, } } /// Check if buffer is empty. #[inline] pub fn is_empty(&self) -> bool { self.len() == 0 } /// Number of [`Bytes`] in [`Buffer`]. /// /// For contiguous buffer, it's always 1. For non-contiguous buffer, it's number of bytes /// available for use. pub fn count(&self) -> usize { match &self.0 { Inner::Contiguous(_) => 1, Inner::NonContiguous { parts, idx, size, offset, } => { parts .iter() .skip(*idx) .fold((0, size + offset), |(count, size), bytes| { if size == 0 { (count, 0) } else { (count + 1, size.saturating_sub(bytes.len())) } }) .0 } } } /// Get current [`Bytes`]. pub fn current(&self) -> Bytes { match &self.0 { Inner::Contiguous(inner) => inner.clone(), Inner::NonContiguous { parts, idx, offset, size, } => { let chunk = &parts[*idx]; let n = (chunk.len() - *offset).min(*size); chunk.slice(*offset..*offset + n) } } } /// Shortens the buffer, keeping the first `len` bytes and dropping the rest. /// /// If `len` is greater than the buffer’s current length, this has no effect. #[inline] pub fn truncate(&mut self, len: usize) { match &mut self.0 { Inner::Contiguous(bs) => bs.truncate(len), Inner::NonContiguous { size, .. } => { *size = (*size).min(len); } } } /// Returns a slice of self for the provided range. /// /// This will increment the reference count for the underlying memory and return a new Buffer handle set to the slice. /// /// This operation is O(1). pub fn slice(&self, range: impl RangeBounds) -> Self { let len = self.len(); let begin = match range.start_bound() { Bound::Included(&n) => n, Bound::Excluded(&n) => n.checked_add(1).expect("out of range"), Bound::Unbounded => 0, }; let end = match range.end_bound() { Bound::Included(&n) => n.checked_add(1).expect("out of range"), Bound::Excluded(&n) => n, Bound::Unbounded => len, }; assert!( begin <= end, "range start must not be greater than end: {:?} <= {:?}", begin, end, ); assert!( end <= len, "range end out of bounds: {:?} <= {:?}", end, len, ); if end == begin { return Buffer::new(); } let mut ret = self.clone(); ret.truncate(end); ret.advance(begin); ret } /// Combine all bytes together into one single [`Bytes`]. /// /// This operation is zero copy if the underlying bytes are contiguous. /// Otherwise, it will copy all bytes into one single [`Bytes`]. /// Please use API from [`Buf`], [`Iterator`] or [`Stream`] whenever possible. #[inline] pub fn to_bytes(&self) -> Bytes { match &self.0 { Inner::Contiguous(bytes) => bytes.clone(), Inner::NonContiguous { parts, size, idx: _, offset, } => { if parts.len() == 1 { parts[0].slice(*offset..(*offset + *size)) } else { let mut ret = BytesMut::with_capacity(self.len()); ret.put(self.clone()); ret.freeze() } } } } /// Combine all bytes together into one single [`Vec`]. /// /// This operation is not zero copy, it will copy all bytes into one single [`Vec`]. /// Please use API from [`Buf`], [`Iterator`] or [`Stream`] whenever possible. #[inline] pub fn to_vec(&self) -> Vec { let mut ret = Vec::with_capacity(self.len()); ret.put(self.clone()); ret } /// Convert buffer into a slice of [`IoSlice`] for vectored write. #[inline] pub fn to_io_slice(&self) -> Vec> { match &self.0 { Inner::Contiguous(bs) => vec![IoSlice::new(bs.chunk())], Inner::NonContiguous { parts, idx, offset, .. } => { let mut ret = Vec::with_capacity(parts.len() - *idx); let mut new_offset = *offset; for part in parts.iter().skip(*idx) { ret.push(IoSlice::new(&part[new_offset..])); new_offset = 0; } ret } } } } impl From> for Buffer { #[inline] fn from(bs: Vec) -> Self { Self(Inner::Contiguous(bs.into())) } } impl From for Buffer { #[inline] fn from(bs: Bytes) -> Self { Self(Inner::Contiguous(bs)) } } impl From for Buffer { #[inline] fn from(s: String) -> Self { Self(Inner::Contiguous(Bytes::from(s))) } } impl From<&'static [u8]> for Buffer { #[inline] fn from(s: &'static [u8]) -> Self { Self(Inner::Contiguous(Bytes::from_static(s))) } } impl From<&'static str> for Buffer { #[inline] fn from(s: &'static str) -> Self { Self(Inner::Contiguous(Bytes::from_static(s.as_bytes()))) } } impl FromIterator for Buffer { #[inline] fn from_iter>(iter: T) -> Self { Self(Inner::Contiguous(Bytes::from_iter(iter))) } } impl From> for Buffer { #[inline] fn from(bs: VecDeque) -> Self { let size = bs.iter().map(Bytes::len).sum(); Self(Inner::NonContiguous { parts: Vec::from(bs).into(), size, idx: 0, offset: 0, }) } } impl From> for Buffer { #[inline] fn from(bs: Vec) -> Self { let size = bs.iter().map(Bytes::len).sum(); Self(Inner::NonContiguous { parts: bs.into(), size, idx: 0, offset: 0, }) } } impl From> for Buffer { #[inline] fn from(bs: Arc<[Bytes]>) -> Self { let size = bs.iter().map(Bytes::len).sum(); Self(Inner::NonContiguous { parts: bs, size, idx: 0, offset: 0, }) } } impl FromIterator for Buffer { #[inline] fn from_iter>(iter: T) -> Self { let mut size = 0; let bs = iter.into_iter().inspect(|v| size += v.len()); // This operation only needs one allocation from iterator to `Arc<[Bytes]>` instead // of iterator -> `Vec` -> `Arc<[Bytes]>`. let parts = Arc::from_iter(bs); Self(Inner::NonContiguous { parts, size, idx: 0, offset: 0, }) } } impl Buf for Buffer { #[inline] fn remaining(&self) -> usize { self.len() } #[inline] fn chunk(&self) -> &[u8] { match &self.0 { Inner::Contiguous(b) => b.chunk(), Inner::NonContiguous { parts, size, idx, offset, } => { if *size == 0 { return &[]; } let chunk = &parts[*idx]; let n = (chunk.len() - *offset).min(*size); &parts[*idx][*offset..*offset + n] } } } #[inline] fn chunks_vectored<'a>(&'a self, dst: &mut [IoSlice<'a>]) -> usize { match &self.0 { Inner::Contiguous(b) => { if dst.is_empty() { return 0; } dst[0] = IoSlice::new(b.chunk()); 1 } Inner::NonContiguous { parts, idx, offset, .. } => { if dst.is_empty() { return 0; } let mut new_offset = *offset; parts .iter() .skip(*idx) .zip(dst.iter_mut()) .map(|(part, dst)| { *dst = IoSlice::new(&part[new_offset..]); new_offset = 0; }) .count() } } } #[inline] fn advance(&mut self, cnt: usize) { match &mut self.0 { Inner::Contiguous(b) => b.advance(cnt), Inner::NonContiguous { parts, size, idx, offset, } => { assert!( cnt <= *size, "cannot advance past {cnt} bytes, only {size} bytes left" ); let mut new_idx = *idx; let mut new_offset = *offset; let mut remaining_cnt = cnt; while remaining_cnt > 0 { let part_len = parts[new_idx].len(); let remaining_in_part = part_len - new_offset; if remaining_cnt < remaining_in_part { new_offset += remaining_cnt; break; } remaining_cnt -= remaining_in_part; new_idx += 1; new_offset = 0; } *idx = new_idx; *offset = new_offset; *size -= cnt; } } } } impl Iterator for Buffer { type Item = Bytes; fn next(&mut self) -> Option { match &mut self.0 { Inner::Contiguous(bs) => { if bs.is_empty() { None } else { Some(mem::take(bs)) } } Inner::NonContiguous { parts, size, idx, offset, } => { if *size == 0 { return None; } let chunk = &parts[*idx]; let n = (chunk.len() - *offset).min(*size); let buf = chunk.slice(*offset..*offset + n); *size -= n; *offset += n; if *offset == chunk.len() { *idx += 1; *offset = 0; } Some(buf) } } } fn size_hint(&self) -> (usize, Option) { match &self.0 { Inner::Contiguous(bs) => { if bs.is_empty() { (0, Some(0)) } else { (1, Some(1)) } } Inner::NonContiguous { parts, idx, .. } => { let remaining = parts.len().saturating_sub(*idx); (remaining, Some(remaining)) } } } } impl Stream for Buffer { type Item = Result; fn poll_next(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll> { Poll::Ready(self.get_mut().next().map(Ok)) } fn size_hint(&self) -> (usize, Option) { Iterator::size_hint(self) } } #[cfg(test)] mod tests { use pretty_assertions::assert_eq; use rand::prelude::*; use super::*; const EMPTY_SLICE: &[u8] = &[]; #[test] fn test_contiguous_buffer() { let mut buf = Buffer::new(); assert_eq!(buf.remaining(), 0); assert_eq!(buf.chunk(), EMPTY_SLICE); assert_eq!(buf.next(), None); } #[test] fn test_empty_non_contiguous_buffer() { let mut buf = Buffer::from(vec![Bytes::new()]); assert_eq!(buf.remaining(), 0); assert_eq!(buf.chunk(), EMPTY_SLICE); assert_eq!(buf.next(), None); } #[test] fn test_non_contiguous_buffer_with_empty_chunks() { let mut buf = Buffer::from(vec![Bytes::from("a")]); assert_eq!(buf.remaining(), 1); assert_eq!(buf.chunk(), b"a"); buf.advance(1); assert_eq!(buf.remaining(), 0); assert_eq!(buf.chunk(), EMPTY_SLICE); } #[test] fn test_non_contiguous_buffer_with_next() { let mut buf = Buffer::from(vec![Bytes::from("a")]); assert_eq!(buf.remaining(), 1); assert_eq!(buf.chunk(), b"a"); let bs = buf.next(); assert_eq!(bs, Some(Bytes::from("a"))); assert_eq!(buf.remaining(), 0); assert_eq!(buf.chunk(), EMPTY_SLICE); } #[test] fn test_buffer_advance() { let mut buf = Buffer::from(vec![Bytes::from("a"), Bytes::from("b"), Bytes::from("c")]); assert_eq!(buf.remaining(), 3); assert_eq!(buf.chunk(), b"a"); buf.advance(1); assert_eq!(buf.remaining(), 2); assert_eq!(buf.chunk(), b"b"); buf.advance(1); assert_eq!(buf.remaining(), 1); assert_eq!(buf.chunk(), b"c"); buf.advance(1); assert_eq!(buf.remaining(), 0); assert_eq!(buf.chunk(), EMPTY_SLICE); buf.advance(0); assert_eq!(buf.remaining(), 0); assert_eq!(buf.chunk(), EMPTY_SLICE); } #[test] fn test_buffer_truncate() { let mut buf = Buffer::from(vec![Bytes::from("a"), Bytes::from("b"), Bytes::from("c")]); assert_eq!(buf.remaining(), 3); assert_eq!(buf.chunk(), b"a"); buf.truncate(100); assert_eq!(buf.remaining(), 3); assert_eq!(buf.chunk(), b"a"); buf.truncate(2); assert_eq!(buf.remaining(), 2); assert_eq!(buf.chunk(), b"a"); buf.truncate(0); assert_eq!(buf.remaining(), 0); assert_eq!(buf.chunk(), EMPTY_SLICE); } /// This setup will return /// /// - A buffer /// - Total size of this buffer. /// - Total content of this buffer. fn setup_buffer() -> (Buffer, usize, Bytes) { let mut rng = thread_rng(); let bs = (0..100) .map(|_| { let len = rng.gen_range(1..100); let mut buf = vec![0; len]; rng.fill(&mut buf[..]); Bytes::from(buf) }) .collect::>(); let total_size = bs.iter().map(|b| b.len()).sum::(); let total_content = bs.iter().flatten().copied().collect::(); let buf = Buffer::from(bs); (buf, total_size, total_content) } #[test] fn fuzz_buffer_advance() { let mut rng = thread_rng(); let (mut buf, total_size, total_content) = setup_buffer(); assert_eq!(buf.remaining(), total_size); assert_eq!(buf.to_bytes(), total_content); let mut cur = 0; // Loop at most 10000 times. let mut times = 10000; while !buf.is_empty() && times > 0 { times -= 1; let cnt = rng.gen_range(0..total_size - cur); cur += cnt; buf.advance(cnt); assert_eq!(buf.remaining(), total_size - cur); assert_eq!(buf.to_bytes(), total_content.slice(cur..)); } } #[test] fn fuzz_buffer_iter() { let mut rng = thread_rng(); let (mut buf, total_size, total_content) = setup_buffer(); assert_eq!(buf.remaining(), total_size); assert_eq!(buf.to_bytes(), total_content); let mut cur = 0; while buf.is_empty() { let cnt = rng.gen_range(0..total_size - cur); cur += cnt; buf.advance(cnt); // Before next assert_eq!(buf.remaining(), total_size - cur); assert_eq!(buf.to_bytes(), total_content.slice(cur..)); if let Some(bs) = buf.next() { assert_eq!(bs, total_content.slice(cur..cur + bs.len())); cur += bs.len(); } // After next assert_eq!(buf.remaining(), total_size - cur); assert_eq!(buf.to_bytes(), total_content.slice(cur..)); } } #[test] fn fuzz_buffer_truncate() { let mut rng = thread_rng(); let (mut buf, total_size, total_content) = setup_buffer(); assert_eq!(buf.remaining(), total_size); assert_eq!(buf.to_bytes(), total_content); let mut cur = 0; while buf.is_empty() { let cnt = rng.gen_range(0..total_size - cur); cur += cnt; buf.advance(cnt); // Before truncate assert_eq!(buf.remaining(), total_size - cur); assert_eq!(buf.to_bytes(), total_content.slice(cur..)); let truncate_size = rng.gen_range(0..total_size - cur); buf.truncate(truncate_size); // After truncate assert_eq!(buf.remaining(), truncate_size); assert_eq!( buf.to_bytes(), total_content.slice(cur..cur + truncate_size) ); // Try next after truncate if let Some(bs) = buf.next() { assert_eq!(bs, total_content.slice(cur..cur + bs.len())); cur += bs.len(); } // After next assert_eq!(buf.remaining(), total_size - cur); assert_eq!(buf.to_bytes(), total_content.slice(cur..)); } } } opendal-0.52.0/src/types/builder.rs000064400000000000000000000111231046102023000152570ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use serde::de::DeserializeOwned; use serde::Serialize; use crate::raw::*; use crate::*; /// Builder is used to set up underlying services. /// /// This trait allows the developer to define a builder struct that can: /// /// - build a service via builder style API. /// - configure in-memory options like `http_client` or `customized_credential_load`. /// /// Usually, users don't need to use or import this trait directly, they can use `Operator` API instead. /// /// For example: /// /// ``` /// # use anyhow::Result; /// use opendal::services::Fs; /// use opendal::Operator; /// async fn test() -> Result<()> { /// // Create fs backend builder. /// let mut builder = Fs::default().root("/tmp"); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::new(builder)?.finish(); /// /// Ok(()) /// } /// ``` pub trait Builder: Default + 'static { /// Associated scheme for this builder. It indicates what underlying service is. const SCHEME: Scheme; /// Associated configuration for this builder. type Config: Configurator; /// Consume the accessor builder to build a service. fn build(self) -> Result; } /// Dummy implementation of builder impl Builder for () { const SCHEME: Scheme = Scheme::Custom("dummy"); type Config = (); fn build(self) -> Result { Ok(()) } } /// Configurator is used to configure the underlying service. /// /// This trait allows the developer to define a configuration struct that can: /// /// - deserialize from an iterator like hashmap or vector. /// - convert into a service builder and finally build the underlying services. /// /// Usually, users don't need to use or import this trait directly, they can use `Operator` API instead. /// /// For example: /// /// ``` /// # use anyhow::Result; /// use std::collections::HashMap; /// /// use opendal::services::MemoryConfig; /// use opendal::Operator; /// async fn test() -> Result<()> { /// let mut cfg = MemoryConfig::default(); /// cfg.root = Some("/".to_string()); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::from_config(cfg)?.finish(); /// /// Ok(()) /// } /// ``` /// /// Some service builder might contain in memory options like `http_client` . Users can call /// `into_builder` to convert the configuration into a builder instead. /// /// ``` /// # use anyhow::Result; /// use std::collections::HashMap; /// /// use opendal::raw::HttpClient; /// use opendal::services::S3Config; /// use opendal::Configurator; /// use opendal::Operator; /// /// async fn test() -> Result<()> { /// let mut cfg = S3Config::default(); /// cfg.root = Some("/".to_string()); /// cfg.bucket = "test".to_string(); /// /// let builder = cfg.into_builder(); /// let builder = builder.http_client(HttpClient::new()?); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::new(builder)?.finish(); /// /// Ok(()) /// } /// ``` pub trait Configurator: Serialize + DeserializeOwned + Debug + 'static { /// Associated builder for this configuration. type Builder: Builder; /// Deserialize from an iterator. /// /// This API is provided by opendal, developer should not implement it. fn from_iter(iter: impl IntoIterator) -> Result { let cfg = ConfigDeserializer::new(iter.into_iter().collect()); Self::deserialize(cfg).map_err(|err| { Error::new(ErrorKind::ConfigInvalid, "failed to deserialize config").set_source(err) }) } /// Convert this configuration into a service builder. fn into_builder(self) -> Self::Builder; } impl Configurator for () { type Builder = (); fn into_builder(self) -> Self::Builder {} } opendal-0.52.0/src/types/capability.rs000064400000000000000000000275511046102023000157660ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; /// Capability defines the supported operations and their constraints for a storage Operator. /// /// # Overview /// /// This structure provides a comprehensive description of an Operator's capabilities, /// including: /// /// - Basic operations support (read, write, delete, etc.) /// - Advanced operation variants (conditional operations, metadata handling) /// - Operational constraints (size limits, batch limitations) /// /// # Capability Types /// /// Every operator maintains two capability sets: /// /// 1. [`OperatorInfo::native_capability`][crate::OperatorInfo::native_capability]: /// Represents operations natively supported by the storage backend. /// /// 2. [`OperatorInfo::full_capability`][crate::OperatorInfo::full_capability]: /// Represents all available operations, including those implemented through /// alternative mechanisms. /// /// # Implementation Details /// /// Some operations might be available even when not natively supported by the /// backend. For example: /// /// - Blocking operations are provided through the BlockingLayer /// /// Developers should: /// - Use `full_capability` to determine available operations /// - Use `native_capability` to identify optimized operations /// /// # Field Naming Conventions /// /// Fields follow these naming patterns: /// /// - Basic operations: Simple lowercase (e.g., `read`, `write`) /// - Compound operations: Underscore-separated (e.g., `presign_read`) /// - Variants: Capability description (e.g., `write_can_empty`) /// - Parameterized operations: With-style (e.g., `read_with_if_match`) /// - Limitations: Constraint description (e.g., `write_multi_max_size`) /// - Metadata Results: Returning metadata capabilities (e.g., `stat_has_content_length`) /// /// All capability fields are public and can be accessed directly. #[derive(Copy, Clone, Default)] pub struct Capability { /// Indicates if the operator supports metadata retrieval operations. pub stat: bool, /// Indicates if conditional stat operations using If-Match are supported. pub stat_with_if_match: bool, /// Indicates if conditional stat operations using If-None-Match are supported. pub stat_with_if_none_match: bool, /// Indicates if conditional stat operations using If-Modified-Since are supported. pub stat_with_if_modified_since: bool, /// Indicates if conditional stat operations using If-Unmodified-Since are supported. pub stat_with_if_unmodified_since: bool, /// Indicates if Cache-Control header override is supported during stat operations. pub stat_with_override_cache_control: bool, /// Indicates if Content-Disposition header override is supported during stat operations. pub stat_with_override_content_disposition: bool, /// Indicates if Content-Type header override is supported during stat operations. pub stat_with_override_content_type: bool, /// Indicates if versions stat operations are supported. pub stat_with_version: bool, /// Indicates whether cache control information is available in stat response pub stat_has_cache_control: bool, /// Indicates whether content disposition information is available in stat response pub stat_has_content_disposition: bool, /// Indicates whether content length information is available in stat response pub stat_has_content_length: bool, /// Indicates whether content MD5 checksum is available in stat response pub stat_has_content_md5: bool, /// Indicates whether content range information is available in stat response pub stat_has_content_range: bool, /// Indicates whether content type information is available in stat response pub stat_has_content_type: bool, /// Indicates whether content encoding information is available in stat response pub stat_has_content_encoding: bool, /// Indicates whether entity tag is available in stat response pub stat_has_etag: bool, /// Indicates whether last modified timestamp is available in stat response pub stat_has_last_modified: bool, /// Indicates whether version information is available in stat response pub stat_has_version: bool, /// Indicates whether user-defined metadata is available in stat response pub stat_has_user_metadata: bool, /// Indicates if the operator supports read operations. pub read: bool, /// Indicates if conditional read operations using If-Match are supported. pub read_with_if_match: bool, /// Indicates if conditional read operations using If-None-Match are supported. pub read_with_if_none_match: bool, /// Indicates if conditional read operations using If-Modified-Since are supported. pub read_with_if_modified_since: bool, /// Indicates if conditional read operations using If-Unmodified-Since are supported. pub read_with_if_unmodified_since: bool, /// Indicates if Cache-Control header override is supported during read operations. pub read_with_override_cache_control: bool, /// Indicates if Content-Disposition header override is supported during read operations. pub read_with_override_content_disposition: bool, /// Indicates if Content-Type header override is supported during read operations. pub read_with_override_content_type: bool, /// Indicates if versions read operations are supported. pub read_with_version: bool, /// Indicates if the operator supports write operations. pub write: bool, /// Indicates if multiple write operations can be performed on the same object. pub write_can_multi: bool, /// Indicates if writing empty content is supported. pub write_can_empty: bool, /// Indicates if append operations are supported. pub write_can_append: bool, /// Indicates if Content-Type can be specified during write operations. pub write_with_content_type: bool, /// Indicates if Content-Disposition can be specified during write operations. pub write_with_content_disposition: bool, /// Indicates if Content-Encoding can be specified during write operations. pub write_with_content_encoding: bool, /// Indicates if Cache-Control can be specified during write operations. pub write_with_cache_control: bool, /// Indicates if conditional write operations using If-Match are supported. pub write_with_if_match: bool, /// Indicates if conditional write operations using If-None-Match are supported. pub write_with_if_none_match: bool, /// Indicates if write operations can be conditional on object non-existence. pub write_with_if_not_exists: bool, /// Indicates if custom user metadata can be attached during write operations. pub write_with_user_metadata: bool, /// Maximum size supported for multipart uploads. /// For example, AWS S3 supports up to 5GiB per part in multipart uploads. pub write_multi_max_size: Option, /// Minimum size required for multipart uploads (except for the last part). /// For example, AWS S3 requires at least 5MiB per part. pub write_multi_min_size: Option, /// Maximum total size supported for write operations. /// For example, Cloudflare D1 has a 1MB total size limit. pub write_total_max_size: Option, /// Indicates whether content length information is available in write response. /// by default, it should be `true` for all write operations. pub write_has_content_length: bool, /// Indicates whether last modified timestamp is available in write response pub write_has_last_modified: bool, /// Indicates whether entity tag is available in write response pub write_has_etag: bool, /// Indicates whether version information is available in write response pub write_has_version: bool, /// Indicates if directory creation is supported. pub create_dir: bool, /// Indicates if delete operations are supported. pub delete: bool, /// Indicates if versions delete operations are supported. pub delete_with_version: bool, /// Maximum size supported for single delete operations. pub delete_max_size: Option, /// Indicates if copy operations are supported. pub copy: bool, /// Indicates if rename operations are supported. pub rename: bool, /// Indicates if list operations are supported. pub list: bool, /// Indicates if list operations support result limiting. pub list_with_limit: bool, /// Indicates if list operations support continuation from a specific point. pub list_with_start_after: bool, /// Indicates if recursive listing is supported. pub list_with_recursive: bool, /// Indicates if versions listing is supported. #[deprecated(since = "0.51.1", note = "use with_versions instead")] pub list_with_version: bool, /// Indicates if listing with versions included is supported. pub list_with_versions: bool, /// Indicates if listing with deleted files included is supported. pub list_with_deleted: bool, /// Indicates whether cache control information is available in list response pub list_has_cache_control: bool, /// Indicates whether content disposition information is available in list response pub list_has_content_disposition: bool, /// Indicates whether content length information is available in list response pub list_has_content_length: bool, /// Indicates whether content MD5 checksum is available in list response pub list_has_content_md5: bool, /// Indicates whether content range information is available in list response pub list_has_content_range: bool, /// Indicates whether content type information is available in list response pub list_has_content_type: bool, /// Indicates whether entity tag is available in list response pub list_has_etag: bool, /// Indicates whether last modified timestamp is available in list response pub list_has_last_modified: bool, /// Indicates whether version information is available in list response pub list_has_version: bool, /// Indicates whether user-defined metadata is available in list response pub list_has_user_metadata: bool, /// Indicates if presigned URL generation is supported. pub presign: bool, /// Indicates if presigned URLs for read operations are supported. pub presign_read: bool, /// Indicates if presigned URLs for stat operations are supported. pub presign_stat: bool, /// Indicates if presigned URLs for write operations are supported. pub presign_write: bool, /// Indicate if the operator supports shared access. pub shared: bool, /// Indicates if blocking operations are supported. pub blocking: bool, } impl Debug for Capability { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { // NOTE: All services in opendal are readable. if self.read { f.write_str("Read")?; } if self.write { f.write_str("| Write")?; } if self.list { f.write_str("| List")?; } if self.presign { f.write_str("| Presign")?; } if self.shared { f.write_str("| Shared")?; } if self.blocking { f.write_str("| Blocking")?; } Ok(()) } } opendal-0.52.0/src/types/context/mod.rs000064400000000000000000000015331046102023000161000ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod read; pub use read::*; mod write; pub use write::*; opendal-0.52.0/src/types/context/read.rs000064400000000000000000000167471046102023000162510ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ops::{Bound, Range, RangeBounds}; use std::sync::Arc; use crate::raw::*; use crate::*; /// ReadContext holds the immutable context for give read operation. pub struct ReadContext { /// The accessor to the storage services. acc: Accessor, /// Path to the file. path: String, /// Arguments for the read operation. args: OpRead, /// Options for the reader. options: OpReader, } impl ReadContext { /// Create a new ReadContext. #[inline] pub fn new(acc: Accessor, path: String, args: OpRead, options: OpReader) -> Self { Self { acc, path, args, options, } } /// Get the accessor. #[inline] pub fn accessor(&self) -> &Accessor { &self.acc } /// Get the path. #[inline] pub fn path(&self) -> &str { &self.path } /// Get the arguments. #[inline] pub fn args(&self) -> &OpRead { &self.args } /// Get the options. #[inline] pub fn options(&self) -> &OpReader { &self.options } /// Parse the range bounds into a range. pub(crate) async fn parse_into_range( &self, range: impl RangeBounds, ) -> Result> { let start = match range.start_bound() { Bound::Included(v) => *v, Bound::Excluded(v) => v + 1, Bound::Unbounded => 0, }; let end = match range.end_bound() { Bound::Included(v) => v + 1, Bound::Excluded(v) => *v, Bound::Unbounded => { let mut op_stat = OpStat::new(); if let Some(v) = self.args().version() { op_stat = op_stat.with_version(v); } self.accessor() .stat(self.path(), op_stat) .await? .into_metadata() .content_length() } }; Ok(start..end) } } /// ReadGenerator is used to generate new readers. /// /// If chunk is None, ReaderGenerator will only return one reader. /// Otherwise, ReaderGenerator will return multiple readers, each with size /// of chunk. /// /// It's design that we didn't implement the generator as a stream, because /// we don't expose the generator to the user. Instead, we use the async method /// directly to keep it simple and easy to understand. pub struct ReadGenerator { ctx: Arc, offset: u64, size: Option, } impl ReadGenerator { /// Create a new ReadGenerator. #[inline] pub fn new(ctx: Arc, offset: u64, size: Option) -> Self { Self { ctx, offset, size } } /// Generate next range to read. fn next_range(&mut self) -> Option { if self.size == Some(0) { return None; } let next_offset = self.offset; let next_size = match self.size { // Given size is None, read all data. None => { // Update size to Some(0) to indicate that there is no more data to read. self.size = Some(0); None } Some(remaining) => { // If chunk is set, read data in chunks. let read_size = self .ctx .options .chunk() .map_or(remaining, |chunk| remaining.min(chunk as u64)); // Update (offset, size) before building future. self.offset += read_size; self.size = Some(remaining - read_size); Some(read_size) } }; Some(BytesRange::new(next_offset, next_size)) } /// Generate next reader. pub async fn next_reader(&mut self) -> Result> { let Some(range) = self.next_range() else { return Ok(None); }; let args = self.ctx.args.clone().with_range(range); let (_, r) = self.ctx.acc.read(&self.ctx.path, args).await?; Ok(Some(r)) } /// Generate next blocking reader. pub fn next_blocking_reader(&mut self) -> Result> { let Some(range) = self.next_range() else { return Ok(None); }; let args = self.ctx.args.clone().with_range(range); let (_, r) = self.ctx.acc.blocking_read(&self.ctx.path, args)?; Ok(Some(r)) } } #[cfg(test)] mod tests { use bytes::Bytes; use super::*; #[tokio::test] async fn test_next_reader() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new().with_chunk(3), )); let mut generator = ReadGenerator::new(ctx, 0, Some(10)); let mut readers = vec![]; while let Some(r) = generator.next_reader().await? { readers.push(r); } pretty_assertions::assert_eq!(readers.len(), 4); Ok(()) } #[tokio::test] async fn test_next_reader_without_size() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new().with_chunk(3), )); let mut generator = ReadGenerator::new(ctx, 0, None); let mut readers = vec![]; while let Some(r) = generator.next_reader().await? { readers.push(r); } pretty_assertions::assert_eq!(readers.len(), 1); Ok(()) } #[test] fn test_next_blocking_reader() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.blocking().write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), )?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new().with_chunk(3), )); let mut generator = ReadGenerator::new(ctx, 0, Some(10)); let mut readers = vec![]; while let Some(r) = generator.next_blocking_reader()? { readers.push(r); } pretty_assertions::assert_eq!(readers.len(), 4); Ok(()) } } opendal-0.52.0/src/types/context/write.rs000064400000000000000000000364761046102023000164710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use crate::raw::oio::Write; use crate::raw::*; use crate::*; /// WriteContext holds the immutable context for give write operation. pub struct WriteContext { /// The accessor to the storage services. acc: Accessor, /// Path to the file. path: String, /// Arguments for the write operation. args: OpWrite, /// Options for the writer. options: OpWriter, } impl WriteContext { /// Create a new WriteContext. #[inline] pub fn new(acc: Accessor, path: String, args: OpWrite, options: OpWriter) -> Self { Self { acc, path, args, options, } } /// Get the accessor. #[inline] pub fn accessor(&self) -> &Accessor { &self.acc } /// Get the path. #[inline] pub fn path(&self) -> &str { &self.path } /// Get the arguments. #[inline] pub fn args(&self) -> &OpWrite { &self.args } /// Get the options. #[inline] pub fn options(&self) -> &OpWriter { &self.options } /// Calculate the chunk size for this write process. /// /// Returns the chunk size and if the chunk size is exact. fn calculate_chunk_size(&self) -> (Option, bool) { let cap = self.accessor().info().full_capability(); let exact = self.options().chunk().is_some(); let chunk_size = self .options() .chunk() .or(cap.write_multi_min_size) .map(|mut size| { if let Some(v) = cap.write_multi_max_size { size = size.min(v); } if let Some(v) = cap.write_multi_min_size { size = size.max(v); } size }); (chunk_size, exact) } } pub struct WriteGenerator { w: W, /// The size for buffer, we will flush the underlying storage at the size of this buffer. chunk_size: Option, /// If `exact` is true, the size of the data written to the underlying storage is /// exactly `chunk_size` bytes. exact: bool, buffer: oio::QueueBuf, } impl WriteGenerator { /// Create a new exact buf writer. pub async fn create(ctx: Arc) -> Result { let (chunk_size, exact) = ctx.calculate_chunk_size(); let (_, w) = ctx.acc.write(ctx.path(), ctx.args().clone()).await?; Ok(Self { w, chunk_size, exact, buffer: oio::QueueBuf::new(), }) } /// Allow building from existing oio::Writer for easier testing. #[cfg(test)] fn new(w: oio::Writer, chunk_size: Option, exact: bool) -> Self { Self { w, chunk_size, exact, buffer: oio::QueueBuf::new(), } } } impl WriteGenerator { /// Write the entire buffer into writer. pub async fn write(&mut self, mut bs: Buffer) -> Result { let Some(chunk_size) = self.chunk_size else { let size = bs.len(); self.w.write_dyn(bs).await?; return Ok(size); }; if self.buffer.len() + bs.len() < chunk_size { let size = bs.len(); self.buffer.push(bs); return Ok(size); } // Condition: // - exact is false // - buffer + bs is larger than chunk_size. // Action: // - write buffer + bs directly. if !self.exact { let fill_size = bs.len(); self.buffer.push(bs); let buf = self.buffer.take().collect(); self.w.write_dyn(buf).await?; return Ok(fill_size); } // Condition: // - exact is true: we need write buffer in exact chunk size. // - buffer is larger than chunk_size // - in exact mode, the size must be chunk_size, use `>=` just for safe coding. // Action: // - write existing buffer in chunk_size to make more rooms for writing data. if self.buffer.len() >= chunk_size { let buf = self.buffer.take().collect(); self.w.write_dyn(buf).await?; } // Condition // - exact is true. // - buffer size must lower than chunk_size. // Action: // - write bs to buffer with remaining size. let remaining = chunk_size - self.buffer.len(); bs.truncate(remaining); let n = bs.len(); self.buffer.push(bs); Ok(n) } /// Finish the write process. pub async fn close(&mut self) -> Result { loop { if self.buffer.is_empty() { break; } let buf = self.buffer.take().collect(); self.w.write_dyn(buf).await?; } self.w.close().await } /// Abort the write process. pub async fn abort(&mut self) -> Result<()> { self.buffer.clear(); self.w.abort().await } } impl WriteGenerator { /// Create a new exact buf writer. pub fn blocking_create(ctx: Arc) -> Result { let (chunk_size, exact) = ctx.calculate_chunk_size(); let (_, w) = ctx.acc.blocking_write(ctx.path(), ctx.args().clone())?; Ok(Self { w, chunk_size, exact, buffer: oio::QueueBuf::new(), }) } } impl WriteGenerator { /// Write the entire buffer into writer. pub fn write(&mut self, mut bs: Buffer) -> Result { let Some(chunk_size) = self.chunk_size else { let size = bs.len(); self.w.write(bs)?; return Ok(size); }; if self.buffer.len() + bs.len() < chunk_size { let size = bs.len(); self.buffer.push(bs); return Ok(size); } // Condition: // - exact is false // - buffer + bs is larger than chunk_size. // Action: // - write buffer + bs directly. if !self.exact { let fill_size = bs.len(); self.buffer.push(bs); let buf = self.buffer.take().collect(); self.w.write(buf)?; return Ok(fill_size); } // Condition: // - exact is true: we need write buffer in exact chunk size. // - buffer is larger than chunk_size // - in exact mode, the size must be chunk_size, use `>=` just for safe coding. // Action: // - write existing buffer in chunk_size to make more rooms for writing data. if self.buffer.len() >= chunk_size { let buf = self.buffer.take().collect(); self.w.write(buf)?; } // Condition // - exact is true. // - buffer size must lower than chunk_size. // Action: // - write bs to buffer with remaining size. let remaining = chunk_size - self.buffer.len(); bs.truncate(remaining); let n = bs.len(); self.buffer.push(bs); Ok(n) } /// Finish the write process. pub fn close(&mut self) -> Result { loop { if self.buffer.is_empty() { break; } let buf = self.buffer.take().collect(); self.w.write(buf)?; } self.w.close() } } #[cfg(test)] mod tests { use bytes::Buf; use bytes::BufMut; use bytes::Bytes; use log::debug; use pretty_assertions::assert_eq; use rand::thread_rng; use rand::Rng; use rand::RngCore; use sha2::Digest; use sha2::Sha256; use tokio::sync::Mutex; use super::*; use crate::raw::oio::Write; struct MockWriter { buf: Arc>>, } impl Write for MockWriter { async fn write(&mut self, bs: Buffer) -> Result<()> { debug!("test_fuzz_exact_buf_writer: flush size: {}", &bs.len()); let mut buf = self.buf.lock().await; buf.put(bs); Ok(()) } async fn close(&mut self) -> Result { Ok(Metadata::default()) } async fn abort(&mut self) -> Result<()> { Ok(()) } } #[tokio::test] async fn test_exact_buf_writer_short_write() -> Result<()> { let _ = tracing_subscriber::fmt() .pretty() .with_test_writer() .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) .try_init(); let mut rng = thread_rng(); let mut expected = vec![0; 5]; rng.fill_bytes(&mut expected); let buf = Arc::new(Mutex::new(vec![])); let mut w = WriteGenerator::new(Box::new(MockWriter { buf: buf.clone() }), Some(10), true); let mut bs = Bytes::from(expected.clone()); while !bs.is_empty() { let n = w.write(bs.clone().into()).await?; bs.advance(n); } w.close().await?; let buf = buf.lock().await; assert_eq!(buf.len(), expected.len()); assert_eq!( format!("{:x}", Sha256::digest(&*buf)), format!("{:x}", Sha256::digest(&expected)) ); Ok(()) } #[tokio::test] async fn test_inexact_buf_writer_large_write() -> Result<()> { let _ = tracing_subscriber::fmt() .pretty() .with_test_writer() .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) .try_init(); let buf = Arc::new(Mutex::new(vec![])); let mut w = WriteGenerator::new(Box::new(MockWriter { buf: buf.clone() }), Some(10), false); let mut rng = thread_rng(); let mut expected = vec![0; 15]; rng.fill_bytes(&mut expected); let bs = Bytes::from(expected.clone()); // The MockWriter always returns the first chunk size. let n = w.write(bs.into()).await?; assert_eq!(expected.len(), n); w.close().await?; let buf = buf.lock().await; assert_eq!(buf.len(), expected.len()); assert_eq!( format!("{:x}", Sha256::digest(&*buf)), format!("{:x}", Sha256::digest(&expected)) ); Ok(()) } #[tokio::test] async fn test_inexact_buf_writer_combine_small() -> Result<()> { let _ = tracing_subscriber::fmt() .pretty() .with_test_writer() .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) .try_init(); let buf = Arc::new(Mutex::new(vec![])); let mut w = WriteGenerator::new(Box::new(MockWriter { buf: buf.clone() }), Some(10), false); let mut rng = thread_rng(); let mut expected = vec![]; let mut new_content = |size| { let mut content = vec![0; size]; rng.fill_bytes(&mut content); expected.extend_from_slice(&content); Bytes::from(content) }; // content > chunk size. let content = new_content(15); assert_eq!(15, w.write(content.into()).await?); // content < chunk size. let content = new_content(5); assert_eq!(5, w.write(content.into()).await?); // content > chunk size, but 5 bytes in queue. let content = new_content(15); // The MockWriter can send all 15 bytes together, so we can only advance 5 bytes. assert_eq!(15, w.write(content.clone().into()).await?); w.close().await?; let buf = buf.lock().await; assert_eq!(buf.len(), expected.len()); assert_eq!( format!("{:x}", Sha256::digest(&*buf)), format!("{:x}", Sha256::digest(&expected)) ); Ok(()) } #[tokio::test] async fn test_inexact_buf_writer_queue_remaining() -> Result<()> { let _ = tracing_subscriber::fmt() .pretty() .with_test_writer() .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) .try_init(); let buf = Arc::new(Mutex::new(vec![])); let mut w = WriteGenerator::new(Box::new(MockWriter { buf: buf.clone() }), Some(10), false); let mut rng = thread_rng(); let mut expected = vec![]; let mut new_content = |size| { let mut content = vec![0; size]; rng.fill_bytes(&mut content); expected.extend_from_slice(&content); Bytes::from(content) }; // content > chunk size. let content = new_content(15); assert_eq!(15, w.write(content.into()).await?); // content < chunk size. let content = new_content(5); assert_eq!(5, w.write(content.into()).await?); // content < chunk size. let content = new_content(3); assert_eq!(3, w.write(content.into()).await?); // content > chunk size, but can send all chunks in the queue. let content = new_content(15); assert_eq!(15, w.write(content.clone().into()).await?); w.close().await?; let buf = buf.lock().await; assert_eq!(buf.len(), expected.len()); assert_eq!( format!("{:x}", Sha256::digest(&*buf)), format!("{:x}", Sha256::digest(&expected)) ); Ok(()) } #[tokio::test] async fn test_fuzz_exact_buf_writer() -> Result<()> { let _ = tracing_subscriber::fmt() .pretty() .with_test_writer() .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) .try_init(); let mut rng = thread_rng(); let mut expected = vec![]; let buf = Arc::new(Mutex::new(vec![])); let buffer_size = rng.gen_range(1..10); let mut writer = WriteGenerator::new( Box::new(MockWriter { buf: buf.clone() }), Some(buffer_size), true, ); debug!("test_fuzz_exact_buf_writer: buffer size: {buffer_size}"); for _ in 0..1000 { let size = rng.gen_range(1..20); debug!("test_fuzz_exact_buf_writer: write size: {size}"); let mut content = vec![0; size]; rng.fill_bytes(&mut content); expected.extend_from_slice(&content); let mut bs = Bytes::from(content.clone()); while !bs.is_empty() { let n = writer.write(bs.clone().into()).await?; bs.advance(n); } } writer.close().await?; let buf = buf.lock().await; assert_eq!(buf.len(), expected.len()); assert_eq!( format!("{:x}", Sha256::digest(&*buf)), format!("{:x}", Sha256::digest(&expected)) ); Ok(()) } } opendal-0.52.0/src/types/delete/blocking_deleter.rs000064400000000000000000000063411046102023000203750ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; /// BlockingDeleter is designed to continuously remove content from storage. /// /// It leverages batch deletion capabilities provided by storage services for efficient removal. pub struct BlockingDeleter { deleter: oio::BlockingDeleter, max_size: usize, cur_size: usize, } impl BlockingDeleter { pub(crate) fn create(acc: Accessor) -> Result { let max_size = acc.info().full_capability().delete_max_size.unwrap_or(1); let (_, deleter) = acc.blocking_delete()?; Ok(Self { deleter, max_size, cur_size: 0, }) } /// Delete a path. pub fn delete(&mut self, input: impl IntoDeleteInput) -> Result<()> { if self.cur_size >= self.max_size { let deleted = self.deleter.flush()?; self.cur_size -= deleted; } let input = input.into_delete_input(); let mut op = OpDelete::default(); if let Some(version) = &input.version { op = op.with_version(version); } self.deleter.delete(&input.path, op)?; self.cur_size += 1; Ok(()) } /// Delete an infallible iterator of paths. /// /// Also see: /// /// - [`BlockingDeleter::delete_try_iter`]: delete an fallible iterator of paths. pub fn delete_iter(&mut self, iter: I) -> Result<()> where I: IntoIterator, D: IntoDeleteInput, { for entry in iter { self.delete(entry)?; } Ok(()) } /// Delete an fallible iterator of paths. /// /// Also see: /// /// - [`BlockingDeleter::delete_iter`]: delete an infallible iterator of paths. pub fn delete_try_iter(&mut self, try_iter: I) -> Result<()> where I: IntoIterator>, D: IntoDeleteInput, { for entry in try_iter { self.delete(entry?)?; } Ok(()) } /// Flush the deleter, returns the number of deleted paths. pub fn flush(&mut self) -> Result { let deleted = self.deleter.flush()?; self.cur_size -= deleted; Ok(deleted) } /// Close the deleter, this will flush the deleter and wait until all paths are deleted. pub fn close(&mut self) -> Result<()> { loop { self.flush()?; if self.cur_size == 0 { break; } } Ok(()) } } opendal-0.52.0/src/types/delete/deleter.rs000064400000000000000000000143221046102023000165230ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::oio::DeleteDyn; use crate::raw::*; use crate::*; use futures::{Stream, StreamExt}; use std::pin::pin; /// Deleter is designed to continuously remove content from storage. /// /// It leverages batch deletion capabilities provided by storage services for efficient removal. /// /// # Usage /// /// [`Deleter`] provides several ways to delete files: /// /// ## Direct Deletion /// /// Use the `delete` method to remove a single file: /// /// ```rust /// use opendal::Operator; /// use opendal::Result; /// /// async fn example(op: Operator) -> Result<()> { /// let mut d = op.deleter().await?; /// d.delete("path/to/file").await?; /// d.close().await?; /// Ok(()) /// } /// ``` /// /// Delete multiple files via a stream: /// /// ```rust /// use opendal::Operator; /// use opendal::Result; /// use futures::stream; /// /// async fn example(op: Operator) -> Result<()> { /// let mut d = op.deleter().await?; /// d.delete_stream(stream::iter(vec!["path/to/file"])).await?; /// d.close().await?; /// Ok(()) /// } /// ``` /// /// ## Using as a Sink /// /// Deleter can be used as a Sink for file deletion: /// /// ```rust /// use opendal::Operator; /// use opendal::Result; /// use futures::{stream, Sink}; /// use futures::SinkExt; /// /// async fn example(op: Operator) -> Result<()> { /// let mut sink = op.deleter().await?.into_sink(); /// sink.send("path/to/file").await?; /// sink.close().await?; /// Ok(()) /// } /// ``` pub struct Deleter { deleter: oio::Deleter, max_size: usize, cur_size: usize, } impl Deleter { pub(crate) async fn create(acc: Accessor) -> Result { let max_size = acc.info().full_capability().delete_max_size.unwrap_or(1); let (_, deleter) = acc.delete().await?; Ok(Self { deleter, max_size, cur_size: 0, }) } /// Delete a path. pub async fn delete(&mut self, input: impl IntoDeleteInput) -> Result<()> { if self.cur_size >= self.max_size { let deleted = self.deleter.flush_dyn().await?; self.cur_size -= deleted; } let input = input.into_delete_input(); let mut op = OpDelete::default(); if let Some(version) = &input.version { op = op.with_version(version); } self.deleter.delete_dyn(&input.path, op)?; self.cur_size += 1; Ok(()) } /// Delete an infallible iterator of paths. /// /// Also see: /// /// - [`Deleter::delete_try_iter`]: delete an fallible iterator of paths. /// - [`Deleter::delete_stream`]: delete an infallible stream of paths. /// - [`Deleter::delete_try_stream`]: delete an fallible stream of paths. pub async fn delete_iter(&mut self, iter: I) -> Result<()> where I: IntoIterator, D: IntoDeleteInput, { for entry in iter { self.delete(entry).await?; } Ok(()) } /// Delete an fallible iterator of paths. /// /// Also see: /// /// - [`Deleter::delete_iter`]: delete an infallible iterator of paths. /// - [`Deleter::delete_stream`]: delete an infallible stream of paths. /// - [`Deleter::delete_try_stream`]: delete an fallible stream of paths. pub async fn delete_try_iter(&mut self, try_iter: I) -> Result<()> where I: IntoIterator>, D: IntoDeleteInput, { for entry in try_iter { self.delete(entry?).await?; } Ok(()) } /// Delete an infallible stream of paths. /// /// Also see: /// /// - [`Deleter::delete_iter`]: delete an infallible iterator of paths. /// - [`Deleter::delete_try_iter`]: delete an fallible iterator of paths. /// - [`Deleter::delete_try_stream`]: delete an fallible stream of paths. pub async fn delete_stream(&mut self, mut stream: S) -> Result<()> where S: Stream, D: IntoDeleteInput, { let mut stream = pin!(stream); while let Some(entry) = stream.next().await { self.delete(entry).await?; } Ok(()) } /// Delete an fallible stream of paths. /// /// Also see: /// /// - [`Deleter::delete_iter`]: delete an infallible iterator of paths. /// - [`Deleter::delete_try_iter`]: delete an fallible iterator of paths. /// - [`Deleter::delete_stream`]: delete an infallible stream of paths. pub async fn delete_try_stream(&mut self, mut try_stream: S) -> Result<()> where S: Stream>, D: IntoDeleteInput, { let mut stream = pin!(try_stream); while let Some(entry) = stream.next().await.transpose()? { self.delete(entry).await?; } Ok(()) } /// Flush the deleter, returns the number of deleted paths. pub async fn flush(&mut self) -> Result { let deleted = self.deleter.flush_dyn().await?; self.cur_size -= deleted; Ok(deleted) } /// Close the deleter, this will flush the deleter and wait until all paths are deleted. pub async fn close(&mut self) -> Result<()> { loop { self.flush().await?; if self.cur_size == 0 { break; } } Ok(()) } /// Convert the deleter into a sink. pub fn into_sink(self) -> FuturesDeleteSink { FuturesDeleteSink::new(self) } } opendal-0.52.0/src/types/delete/futures_delete_sink.rs000064400000000000000000000144321046102023000211440ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; use futures::Sink; use std::marker::PhantomData; use std::pin::Pin; use std::task::{ready, Context, Poll}; /// FuturesDeleteSink is a sink that generated by [`Deleter`] pub struct FuturesDeleteSink { state: State, _phantom: PhantomData, } enum State { Idle(Option), Delete(BoxedStaticFuture<(Deleter, Result<()>)>), Flush(BoxedStaticFuture<(Deleter, Result)>), Close(BoxedStaticFuture<(Deleter, Result<()>)>), } impl FuturesDeleteSink { #[inline] pub(super) fn new(deleter: Deleter) -> Self { Self { state: State::Idle(Some(deleter)), _phantom: PhantomData, } } } impl Sink for FuturesDeleteSink { type Error = Error; fn poll_ready(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { match &mut self.state { State::Idle(_) => Poll::Ready(Ok(())), State::Delete(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); Poll::Ready(res.map(|_| ())) } State::Flush(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); Poll::Ready(res.map(|_| ())) } State::Close(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); Poll::Ready(res.map(|_| ())) } } } fn start_send(mut self: Pin<&mut Self>, item: T) -> Result<()> { match &mut self.state { State::Idle(deleter) => { let mut deleter = deleter.take().ok_or_else(|| { Error::new( ErrorKind::Unexpected, "FuturesDeleteSink has been closed or errored", ) })?; let input = item.into_delete_input(); let fut = async move { let res = deleter.delete(input).await; (deleter, res) }; self.state = State::Delete(Box::pin(fut)); Ok(()) } _ => Err(Error::new( ErrorKind::Unexpected, "FuturesDeleteSink is not ready to send, please poll_ready first", )), } } fn poll_flush(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { loop { match &mut self.state { State::Idle(deleter) => { let mut deleter = deleter.take().ok_or_else(|| { Error::new( ErrorKind::Unexpected, "FuturesDeleteSink has been closed or errored", ) })?; let fut = async move { let res = deleter.flush().await; (deleter, res) }; self.state = State::Flush(Box::pin(fut)); return Poll::Ready(Ok(())); } State::Delete(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); res?; continue; } State::Flush(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); let _ = res?; return Poll::Ready(Ok(())); } State::Close(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); res?; continue; } }; } } fn poll_close(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { loop { match &mut self.state { State::Idle(deleter) => { let mut deleter = deleter.take().ok_or_else(|| { Error::new( ErrorKind::Unexpected, "FuturesDeleteSink has been closed or errored", ) })?; let fut = async move { let res = deleter.close().await; (deleter, res) }; self.state = State::Close(Box::pin(fut)); return Poll::Ready(Ok(())); } State::Delete(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); res?; continue; } State::Flush(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); res?; continue; } State::Close(fut) => { let (deleter, res) = ready!(fut.as_mut().poll(cx)); self.state = State::Idle(Some(deleter)); return Poll::Ready(res); } }; } } } opendal-0.52.0/src/types/delete/input.rs000064400000000000000000000056731046102023000162470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::OpDelete; use crate::Entry; /// DeleteInput is the input for delete operations. #[non_exhaustive] #[derive(Default, Debug)] pub struct DeleteInput { /// The path of the path to delete. pub path: String, /// The version of the path to delete. pub version: Option, } /// IntoDeleteInput is a helper trait that makes it easier for users to play with `Deleter`. pub trait IntoDeleteInput: Send + Sync + Unpin { /// Convert `self` into a `DeleteInput`. fn into_delete_input(self) -> DeleteInput; } /// Implement `IntoDeleteInput` for `DeleteInput` self. impl IntoDeleteInput for DeleteInput { fn into_delete_input(self) -> DeleteInput { self } } /// Implement `IntoDeleteInput` for `&str` so we can use `&str` as a DeleteInput. impl IntoDeleteInput for &str { fn into_delete_input(self) -> DeleteInput { DeleteInput { path: self.to_string(), ..Default::default() } } } /// Implement `IntoDeleteInput` for `String` so we can use `Vec` as a DeleteInput stream. impl IntoDeleteInput for String { fn into_delete_input(self) -> DeleteInput { DeleteInput { path: self, ..Default::default() } } } /// Implement `IntoDeleteInput` for `(String, OpDelete)` so we can use `(String, OpDelete)` /// as a DeleteInput stream. impl IntoDeleteInput for (String, OpDelete) { fn into_delete_input(self) -> DeleteInput { let (path, args) = self; let mut input = DeleteInput { path, ..Default::default() }; if let Some(version) = args.version() { input.version = Some(version.to_string()); } input } } /// Implement `IntoDeleteInput` for `Entry` so we can use `Lister` as a DeleteInput stream. impl IntoDeleteInput for Entry { fn into_delete_input(self) -> DeleteInput { let (path, meta) = self.into_parts(); let mut input = DeleteInput { path, ..Default::default() }; if let Some(version) = meta.version() { input.version = Some(version.to_string()); } input } } opendal-0.52.0/src/types/delete/mod.rs000064400000000000000000000020351046102023000156540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod input; pub use input::DeleteInput; pub use input::IntoDeleteInput; mod deleter; pub use deleter::Deleter; mod futures_delete_sink; pub use futures_delete_sink::FuturesDeleteSink; mod blocking_deleter; pub use blocking_deleter::BlockingDeleter; opendal-0.52.0/src/types/entry.rs000064400000000000000000000041621046102023000147770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::*; use crate::*; /// Entry returned by [`Lister`] or [`BlockingLister`] to represent a path and it's relative metadata. #[derive(Clone, Debug)] pub struct Entry { /// Path of this entry. path: String, /// Metadata of this entry. metadata: Metadata, } impl Entry { /// Create an entry with metadata. /// /// # Notes /// /// The only way to get an entry with associated cached metadata /// is `Operator::list`. pub(crate) fn new(path: String, metadata: Metadata) -> Self { Self { path, metadata } } /// Path of entry. Path is relative to operator's root. /// /// Only valid in current operator. /// /// If this entry is a dir, `path` MUST end with `/` /// Otherwise, `path` MUST NOT end with `/`. pub fn path(&self) -> &str { &self.path } /// Name of entry. Name is the last segment of path. /// /// If this entry is a dir, `name` MUST end with `/` /// Otherwise, `name` MUST NOT end with `/`. pub fn name(&self) -> &str { get_basename(&self.path) } /// Fetch metadata of this entry. pub fn metadata(&self) -> &Metadata { &self.metadata } /// Consume this entry to get its path and metadata. pub fn into_parts(self) -> (String, Metadata) { (self.path, self.metadata) } } opendal-0.52.0/src/types/error.rs000064400000000000000000000340171046102023000147710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Errors that returned by OpenDAL //! //! # Examples //! //! ``` //! # use anyhow::Result; //! # use opendal::EntryMode; //! # use opendal::Operator; //! use opendal::ErrorKind; //! # async fn test(op: Operator) -> Result<()> { //! if let Err(e) = op.stat("test_file").await { //! if e.kind() == ErrorKind::NotFound { //! println!("entry not exist") //! } //! } //! # Ok(()) //! # } //! ``` use std::backtrace::Backtrace; use std::backtrace::BacktraceStatus; use std::fmt; use std::fmt::Debug; use std::fmt::Display; use std::fmt::Formatter; use std::io; /// Result that is a wrapper of `Result` pub type Result = std::result::Result; /// ErrorKind is all kinds of Error of opendal. #[derive(Clone, Copy, Debug, PartialEq, Eq)] #[non_exhaustive] pub enum ErrorKind { /// OpenDAL don't know what happened here, and no actions other than just /// returning it back. For example, s3 returns an internal service error. Unexpected, /// Underlying service doesn't support this operation. Unsupported, /// The config for backend is invalid. ConfigInvalid, /// The given path is not found. NotFound, /// The given path doesn't have enough permission for this operation PermissionDenied, /// The given path is a directory. IsADirectory, /// The given path is not a directory. NotADirectory, /// The given path already exists thus we failed to the specified operation on it. AlreadyExists, /// Requests that sent to this path is over the limit, please slow down. RateLimited, /// The given file paths are same. IsSameFile, /// The condition of this operation is not match. /// /// The `condition` itself is context based. /// /// For example, in S3, the `condition` can be: /// 1. writing a file with If-Match header but the file's ETag is not match (will get a 412 Precondition Failed). /// 2. reading a file with If-None-Match header but the file's ETag is match (will get a 304 Not Modified). /// /// As OpenDAL cannot handle the `condition not match` error, it will always return this error to users. /// So users could to handle this error by themselves. ConditionNotMatch, /// The range of the content is not satisfied. /// /// OpenDAL returns this error to indicate that the range of the read request is not satisfied. RangeNotSatisfied, } impl ErrorKind { /// Convert self into static str. pub fn into_static(self) -> &'static str { self.into() } /// Capturing a backtrace can be a quite expensive runtime operation. /// For some kinds of errors, backtrace is not useful and we can skip it (e.g., check if a file exists). /// /// See fn disable_backtrace(&self) -> bool { matches!(self, ErrorKind::NotFound) } } impl Display for ErrorKind { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { write!(f, "{}", self.into_static()) } } impl From for &'static str { fn from(v: ErrorKind) -> &'static str { match v { ErrorKind::Unexpected => "Unexpected", ErrorKind::Unsupported => "Unsupported", ErrorKind::ConfigInvalid => "ConfigInvalid", ErrorKind::NotFound => "NotFound", ErrorKind::PermissionDenied => "PermissionDenied", ErrorKind::IsADirectory => "IsADirectory", ErrorKind::NotADirectory => "NotADirectory", ErrorKind::AlreadyExists => "AlreadyExists", ErrorKind::RateLimited => "RateLimited", ErrorKind::IsSameFile => "IsSameFile", ErrorKind::ConditionNotMatch => "ConditionNotMatch", ErrorKind::RangeNotSatisfied => "RangeNotSatisfied", } } } #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum ErrorStatus { /// Permanent means without external changes, the error never changes. /// /// For example, underlying services returns a not found error. /// /// Users SHOULD never retry this operation. Permanent, /// Temporary means this error is returned for temporary. /// /// For example, underlying services is rate limited or unavailable for temporary. /// /// Users CAN retry the operation to resolve it. Temporary, /// Persistent means this error used to be temporary but still failed after retry. /// /// For example, underlying services kept returning network errors. /// /// Users MAY retry this operation but it's highly possible to error again. Persistent, } impl Display for ErrorStatus { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { match self { ErrorStatus::Permanent => write!(f, "permanent"), ErrorStatus::Temporary => write!(f, "temporary"), ErrorStatus::Persistent => write!(f, "persistent"), } } } /// Error is the error struct returned by all opendal functions. /// /// ## Display /// /// Error can be displayed in two ways: /// /// - Via `Display`: like `err.to_string()` or `format!("{err}")` /// /// Error will be printed in a single line: /// /// ```shell /// Unexpected, context: { path: /path/to/file, called: send_async } => something wrong happened, source: networking error" /// ``` /// /// - Via `Debug`: like `format!("{err:?}")` /// /// Error will be printed in multi lines with more details and backtraces (if captured): /// /// ```shell /// Unexpected => something wrong happened /// /// Context: /// path: /path/to/file /// called: send_async /// /// Source: networking error /// /// Backtrace: /// 0: opendal::error::Error::new /// at ./src/error.rs:197:24 /// 1: opendal::error::tests::generate_error /// at ./src/error.rs:241:9 /// 2: opendal::error::tests::test_error_debug_with_backtrace::{{closure}} /// at ./src/error.rs:305:41 /// ... /// ``` /// /// - For conventional struct-style Debug representation, like `format!("{err:#?}")`: /// /// ```shell /// Error { /// kind: Unexpected, /// message: "something wrong happened", /// status: Permanent, /// operation: "Read", /// context: [ /// ( /// "path", /// "/path/to/file", /// ), /// ( /// "called", /// "send_async", /// ), /// ], /// source: Some( /// "networking error", /// ), /// } /// ``` pub struct Error { kind: ErrorKind, message: String, status: ErrorStatus, operation: &'static str, context: Vec<(&'static str, String)>, source: Option, backtrace: Backtrace, } impl Display for Error { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { write!(f, "{} ({}) at {}", self.kind, self.status, self.operation)?; if !self.context.is_empty() { write!(f, ", context: {{ ")?; write!( f, "{}", self.context .iter() .map(|(k, v)| format!("{k}: {v}")) .collect::>() .join(", ") )?; write!(f, " }}")?; } if !self.message.is_empty() { write!(f, " => {}", self.message)?; } if let Some(source) = &self.source { write!(f, ", source: {source}")?; } Ok(()) } } impl Debug for Error { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { // If alternate has been specified, we will print like Debug. if f.alternate() { let mut de = f.debug_struct("Error"); de.field("kind", &self.kind); de.field("message", &self.message); de.field("status", &self.status); de.field("operation", &self.operation); de.field("context", &self.context); de.field("source", &self.source); return de.finish(); } write!(f, "{} ({}) at {}", self.kind, self.status, self.operation)?; if !self.message.is_empty() { write!(f, " => {}", self.message)?; } writeln!(f)?; if !self.context.is_empty() { writeln!(f)?; writeln!(f, "Context:")?; for (k, v) in self.context.iter() { writeln!(f, " {k}: {v}")?; } } if let Some(source) = &self.source { writeln!(f)?; writeln!(f, "Source:")?; writeln!(f, " {source:#}")?; } if self.backtrace.status() == BacktraceStatus::Captured { writeln!(f)?; writeln!(f, "Backtrace:")?; writeln!(f, "{}", self.backtrace)?; } Ok(()) } } impl std::error::Error for Error { fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { self.source.as_ref().map(|v| v.as_ref()) } } impl Error { /// Create a new Error with error kind and message. pub fn new(kind: ErrorKind, message: impl Into) -> Self { Self { kind, message: message.into(), status: ErrorStatus::Permanent, operation: "", context: Vec::default(), source: None, // `Backtrace::capture()` will check if backtrace has been enabled // internally. It's zero cost if backtrace is disabled. backtrace: if kind.disable_backtrace() { Backtrace::disabled() } else { Backtrace::capture() }, } } /// Update error's operation. /// /// # Notes /// /// If the error already carries an operation, we will push a new context /// `(called, operation)`. pub fn with_operation(mut self, operation: impl Into<&'static str>) -> Self { if !self.operation.is_empty() { self.context.push(("called", self.operation.to_string())); } self.operation = operation.into(); self } /// Add more context in error. pub fn with_context(mut self, key: &'static str, value: impl ToString) -> Self { self.context.push((key, value.to_string())); self } /// Set source for error. /// /// # Notes /// /// If the source has been set, we will raise a panic here. pub fn set_source(mut self, src: impl Into) -> Self { debug_assert!(self.source.is_none(), "the source error has been set"); self.source = Some(src.into()); self } /// Operate on error with map. pub fn map(self, f: F) -> Self where F: FnOnce(Self) -> Self, { f(self) } /// Set permanent status for error. pub fn set_permanent(mut self) -> Self { self.status = ErrorStatus::Permanent; self } /// Set temporary status for error. /// /// By set temporary, we indicate this error is retryable. pub fn set_temporary(mut self) -> Self { self.status = ErrorStatus::Temporary; self } /// Set temporary status for error by given temporary. /// /// By set temporary, we indicate this error is retryable. pub(crate) fn with_temporary(mut self, temporary: bool) -> Self { if temporary { self.status = ErrorStatus::Temporary; } self } /// Set persistent status for error. /// /// By setting persistent, we indicate the retry should be stopped. pub fn set_persistent(mut self) -> Self { self.status = ErrorStatus::Persistent; self } /// Return error's kind. pub fn kind(&self) -> ErrorKind { self.kind } /// Check if this error is temporary. pub fn is_temporary(&self) -> bool { self.status == ErrorStatus::Temporary } } impl From for io::Error { fn from(err: Error) -> Self { let kind = match err.kind() { ErrorKind::NotFound => io::ErrorKind::NotFound, ErrorKind::PermissionDenied => io::ErrorKind::PermissionDenied, _ => io::ErrorKind::Other, }; io::Error::new(kind, err) } } #[cfg(test)] mod tests { use anyhow::anyhow; use once_cell::sync::Lazy; use pretty_assertions::assert_eq; use super::*; static TEST_ERROR: Lazy = Lazy::new(|| Error { kind: ErrorKind::Unexpected, message: "something wrong happened".to_string(), status: ErrorStatus::Permanent, operation: "Read", context: vec![ ("path", "/path/to/file".to_string()), ("called", "send_async".to_string()), ], source: Some(anyhow!("networking error")), backtrace: Backtrace::disabled(), }); #[test] fn test_error_display() { let s = format!("{}", Lazy::force(&TEST_ERROR)); assert_eq!( s, r#"Unexpected (permanent) at Read, context: { path: /path/to/file, called: send_async } => something wrong happened, source: networking error"# ); println!("{:#?}", Lazy::force(&TEST_ERROR)); } #[test] fn test_error_debug() { let s = format!("{:?}", Lazy::force(&TEST_ERROR)); assert_eq!( s, r#"Unexpected (permanent) at Read => something wrong happened Context: path: /path/to/file called: send_async Source: networking error "# ) } } opendal-0.52.0/src/types/execute/api.rs000064400000000000000000000074451046102023000160600ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use std::pin::Pin; use std::task::Context; use std::task::Poll; use futures::future::RemoteHandle; use futures::FutureExt; use crate::raw::BoxedStaticFuture; /// Execute trait is used to execute task in background. /// /// # Notes about Timeout Implementation /// /// Implementing a correct and elegant timeout mechanism is challenging for us. /// /// The `Execute` trait must be object safe, allowing us to use `Arc`. Consequently, /// we cannot introduce a generic type parameter to `Execute`. We utilize [`RemoteHandle`] to /// implement the [`Execute::execute`] method. [`RemoteHandle`] operates by transmitting /// `Future::Output` through a channel, enabling the spawning of [`BoxedStaticFuture<()>`]. /// /// However, for timeouts, we need to spawn a future that resolves after a specified duration. /// Simply wrapping the future within another timeout future is not feasible because if the timeout /// is reached and the original future has not completed, it will be dropped—causing any held `Task` /// to panic. /// /// As an alternative solution, we developed a `timeout` API. Users of the `Executor` should invoke /// this API when they require a timeout and combine it with their own futures using /// [`futures::select`]. /// /// This approach may seem inelegant but it allows us flexibility without being tied specifically /// to the Tokio runtime. /// /// PLEASE raising an issue if you have a better solution. pub trait Execute: Send + Sync + 'static { /// Execute async task in background. /// /// # Behavior /// /// - Implementor MUST manage the executing futures and keep making progress. /// - Implementor MUST NOT drop futures until it's resolved. fn execute(&self, f: BoxedStaticFuture<()>); /// Return a future that will be resolved after the given timeout. /// /// Default implementation returns None. fn timeout(&self) -> Option> { None } } impl Execute for () { fn execute(&self, _: BoxedStaticFuture<()>) { panic!("concurrent tasks executed with no executor has been enabled") } } /// Task is generated by Executor that represents an executing task. /// /// Users can fetch the results by calling `poll` or `.await` on this task. /// Or, users can cancel the task by `drop` this task handle. /// /// # Notes /// /// Users don't need to call `poll` to make progress. All tasks are running in /// the background. pub struct Task { handle: RemoteHandle, } impl Task { /// Create a new task. #[inline] pub fn new(handle: RemoteHandle) -> Self { Self { handle } } /// Replace the task with a new task. /// /// The old task will be dropped directly. #[inline] pub fn replace(&mut self, new_task: Self) { self.handle = new_task.handle; } } impl Future for Task { type Output = T; fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { self.handle.poll_unpin(cx) } } opendal-0.52.0/src/types/execute/executor.rs000064400000000000000000000056071046102023000171430ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Debug; use std::fmt::Formatter; use std::future::Future; use std::sync::Arc; use futures::FutureExt; use super::*; use crate::raw::BoxedStaticFuture; use crate::raw::MaybeSend; /// Executor that runs futures in background. /// /// Executor is created by users and used by opendal. So it's by design that Executor only /// expose constructor methods. /// /// Executor will run futures in background and return a `Task` as handle to the future. Users /// can call `task.await` to wait for the future to complete or drop the `Task` to cancel it. #[derive(Clone)] pub struct Executor { executor: Arc, } impl Debug for Executor { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { write!(f, "Executor") } } impl Default for Executor { fn default() -> Self { Self::new() } } impl Executor { /// Create a default executor. /// /// The default executor is enabled by feature flags. If no feature flags enabled, the default /// executor will always return error if users try to perform concurrent tasks. pub fn new() -> Self { #[cfg(feature = "executors-tokio")] { Self::with(executors::TokioExecutor::default()) } #[cfg(not(feature = "executors-tokio"))] { Self::with(()) } } /// Create a new executor with given execute impl. pub fn with(exec: impl Execute) -> Self { Self { executor: Arc::new(exec), } } /// Return the inner executor. pub(crate) fn into_inner(self) -> Arc { self.executor } /// Return a future that will be resolved after the given timeout. pub(crate) fn timeout(&self) -> Option> { self.executor.timeout() } /// Run given future in background immediately. pub(crate) fn execute(&self, f: F) -> Task where F: Future + MaybeSend + 'static, F::Output: MaybeSend + 'static, { let (fut, handle) = f.remote_handle(); self.executor.execute(Box::pin(fut)); Task::new(handle) } } opendal-0.52.0/src/types/execute/executors/mod.rs000064400000000000000000000024621046102023000201010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! executors module provides implementations for the [`Execute`](crate::Execute) trait for widely used runtimes. //! //! Every executor will be hide behind the feature like `executors-xxx`. Users can switch or enable //! the executors they want by enabling the corresponding feature. Also, users can provide their //! own executor by implementing the [`Execute`](crate::Execute) trait directly. #[cfg(feature = "executors-tokio")] mod tokio_executor; #[cfg(feature = "executors-tokio")] pub use tokio_executor::TokioExecutor; opendal-0.52.0/src/types/execute/executors/tokio_executor.rs000064400000000000000000000037171046102023000223710ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use crate::raw::BoxedStaticFuture; use crate::*; /// Executor that uses the [`tokio::task::spawn`] to execute futures. #[derive(Default)] pub struct TokioExecutor {} impl Execute for TokioExecutor { /// Tokio's JoinHandle has its own `abort` support, so dropping handle won't cancel the task. fn execute(&self, f: BoxedStaticFuture<()>) { let _handle = tokio::task::spawn(f); } } #[cfg(test)] mod tests { use std::sync::atomic::AtomicBool; use std::sync::atomic::Ordering; use std::sync::Arc; use std::time::Duration; use tokio::time::sleep; use super::*; use crate::Executor; #[tokio::test] async fn test_tokio_executor() { let executor = Executor::with(TokioExecutor::default()); let finished = Arc::new(AtomicBool::new(false)); let finished_clone = finished.clone(); let _task = executor.execute(async move { sleep(Duration::from_secs(1)).await; finished_clone.store(true, Ordering::Relaxed); }); sleep(Duration::from_secs(2)).await; // Task must have been finished even without await task. assert!(finished.load(Ordering::Relaxed)) } } opendal-0.52.0/src/types/execute/mod.rs000064400000000000000000000016321046102023000160560ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod api; pub use api::Execute; pub(crate) use api::Task; mod executor; pub use executor::Executor; pub mod executors; opendal-0.52.0/src/types/list.rs000064400000000000000000000122021046102023000146030ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::pin::Pin; use std::task::ready; use std::task::Context; use std::task::Poll; use futures::Stream; use crate::raw::*; use crate::*; /// Lister is designed to list entries at given path in an asynchronous /// manner. /// /// - Lister implements `Stream>`. /// - Lister will return `None` if there is no more entries or error has been returned. pub struct Lister { lister: Option, fut: Option>)>>, errored: bool, } /// # Safety /// /// Lister will only be accessed by `&mut Self` unsafe impl Sync for Lister {} impl Lister { /// Create a new lister. pub(crate) async fn create(acc: Accessor, path: &str, args: OpList) -> Result { let (_, lister) = acc.list(path, args).await?; Ok(Self { lister: Some(lister), fut: None, errored: false, }) } } impl Stream for Lister { type Item = Result; fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { // Returns `None` if we have errored. if self.errored { return Poll::Ready(None); } if let Some(mut lister) = self.lister.take() { let fut = async move { let res = lister.next_dyn().await; (lister, res) }; self.fut = Some(Box::pin(fut)); } if let Some(fut) = self.fut.as_mut() { let (lister, entry) = ready!(fut.as_mut().poll(cx)); self.lister = Some(lister); self.fut = None; return match entry { Ok(Some(oe)) => Poll::Ready(Some(Ok(oe.into_entry()))), Ok(None) => { self.lister = None; Poll::Ready(None) } Err(err) => { self.errored = true; Poll::Ready(Some(Err(err))) } }; } Poll::Ready(None) } } /// BlockingLister is designed to list entries at given path in a blocking /// manner. /// /// Users can construct Lister by [`BlockingOperator::lister`] or [`BlockingOperator::lister_with`]. /// /// - Lister implements `Iterator>`. /// - Lister will return `None` if there is no more entries or error has been returned. pub struct BlockingLister { lister: oio::BlockingLister, errored: bool, } /// # Safety /// /// BlockingLister will only be accessed by `&mut Self` unsafe impl Sync for BlockingLister {} impl BlockingLister { /// Create a new lister. pub(crate) fn create(acc: Accessor, path: &str, args: OpList) -> Result { let (_, lister) = acc.blocking_list(path, args)?; Ok(Self { lister, errored: false, }) } } impl Iterator for BlockingLister { type Item = Result; fn next(&mut self) -> Option { // Returns `None` if we have errored. if self.errored { return None; } match self.lister.next() { Ok(Some(entry)) => Some(Ok(entry.into_entry())), Ok(None) => None, Err(err) => { self.errored = true; Some(Err(err)) } } } } #[cfg(test)] #[cfg(feature = "services-azblob")] mod tests { use futures::future; use futures::StreamExt; use super::*; use crate::services::Azblob; /// Inspired by /// /// Invalid lister should not panic nor endless loop. #[tokio::test] async fn test_invalid_lister() -> Result<()> { let _ = tracing_subscriber::fmt().try_init(); let builder = Azblob::default() .container("container") .account_name("account_name") .account_key("account_key") .endpoint("https://account_name.blob.core.windows.net"); let operator = Operator::new(builder)?.finish(); let lister = operator.lister("/").await?; lister .filter_map(|entry| { dbg!(&entry); future::ready(entry.ok()) }) .for_each(|entry| { println!("{:?}", entry); future::ready(()) }) .await; Ok(()) } } opendal-0.52.0/src/types/metadata.rs000064400000000000000000000443171046102023000154240ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use chrono::prelude::*; use crate::raw::*; use crate::*; /// Metadata contains all the information related to a specific path. /// /// Depending on the context of the requests, the metadata for the same path may vary. For example, two /// versions of the same path might have different content lengths. Keep in mind that metadata is always /// tied to the given context and is not a global state. /// /// ## File Versions /// /// In systems that support versioning, such as AWS S3, the metadata may represent a specific version /// of a file. /// /// Users can access [`Metadata::version`] to retrieve the file's version, if available. They can also /// use [`Metadata::is_current`] and [`Metadata::is_deleted`] to determine whether the metadata /// corresponds to the latest version or a deleted one. /// /// The all possible combinations of `is_current` and `is_deleted` are as follows: /// /// | `is_current` | `is_deleted` | description | /// |---------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| /// | `Some(true)` | `false` | **The metadata's associated version is the latest, current version.** This is the normal state, indicating that this version is the most up-to-date and accessible version. | /// | `Some(true)` | `true` | **The metadata's associated version is the latest, deleted version (Latest Delete Marker or Soft Deleted).** This is particularly important in object storage systems like S3. It signifies that this version is the **most recent delete marker**, indicating the object has been deleted. Subsequent GET requests will return 404 errors unless a specific version ID is provided. | /// | `Some(false)` | `false` | **The metadata's associated version is neither the latest version nor deleted.** This indicates that this version is a previous version, still accessible by specifying its version ID. | /// | `Some(false)` | `true` | **The metadata's associated version is not the latest version and is deleted.** This represents a historical version that has been marked for deletion. Users will need to specify the version ID to access it, and accessing it may be subject to specific delete marker behavior (e.g., in S3, it might not return actual data but a specific delete marker response). | /// | `None` | `false` | **The metadata's associated file is not deleted, but its version status is either unknown or it is not the latest version.** This likely indicates that versioning is not enabled for this file, or versioning information is unavailable. | /// | `None` | `true` | **The metadata's associated file is deleted, but its version status is either unknown or it is not the latest version.** This typically means the file was deleted without versioning enabled, or its versioning information is unavailable. This may represent an actual data deletion operation rather than an S3 delete marker. | #[derive(Debug, Clone, Eq, PartialEq, Default)] pub struct Metadata { mode: EntryMode, is_current: Option, is_deleted: bool, cache_control: Option, content_disposition: Option, content_length: Option, content_md5: Option, content_range: Option, content_type: Option, content_encoding: Option, etag: Option, last_modified: Option>, version: Option, user_metadata: Option>, } impl Metadata { /// Create a new metadata pub fn new(mode: EntryMode) -> Self { Self { mode, is_current: None, is_deleted: false, cache_control: None, content_length: None, content_md5: None, content_type: None, content_encoding: None, content_range: None, last_modified: None, etag: None, content_disposition: None, version: None, user_metadata: None, } } /// mode represent this entry's mode. pub fn mode(&self) -> EntryMode { self.mode } /// Set mode for entry. pub fn set_mode(&mut self, v: EntryMode) -> &mut Self { self.mode = v; self } /// Set mode for entry. pub fn with_mode(mut self, v: EntryMode) -> Self { self.mode = v; self } /// Returns `true` if this metadata is for a file. pub fn is_file(&self) -> bool { matches!(self.mode, EntryMode::FILE) } /// Returns `true` if this metadata is for a directory. pub fn is_dir(&self) -> bool { matches!(self.mode, EntryMode::DIR) } /// Checks whether the metadata corresponds to the most recent version of the file. /// /// This function is particularly useful when working with versioned objects, /// such as those stored in systems like AWS S3 with versioning enabled. It helps /// determine if the retrieved metadata represents the current state of the file /// or an older version. /// /// Refer to docs in [`Metadata`] for more information about file versions. /// /// # Return Value /// /// The function returns an `Option` which can have the following values: /// /// - `Some(true)`: Indicates that the metadata **is** associated with the latest version of the file. /// The metadata is current and reflects the most up-to-date state. /// - `Some(false)`: Indicates that the metadata **is not** associated with the latest version of the file. /// The metadata belongs to an older version, and there might be a more recent version available. /// - `None`: Indicates that the currency of the metadata **cannot be determined**. This might occur if /// versioning is not supported or enabled, or if there is insufficient information to ascertain the version status. pub fn is_current(&self) -> Option { self.is_current } /// Set the `is_current` status of this entry. /// /// By default, this value will be `None`. Please avoid using this API if it's unclear whether the entry is current. /// Set it to `true` if it is known to be the latest; otherwise, set it to `false`. pub fn set_is_current(&mut self, is_current: bool) -> &mut Self { self.is_current = Some(is_current); self } /// Set the `is_current` status of this entry. /// /// By default, this value will be `None`. Please avoid using this API if it's unclear whether the entry is current. /// Set it to `true` if it is known to be the latest; otherwise, set it to `false`. pub fn with_is_current(mut self, is_current: Option) -> Self { self.is_current = is_current; self } /// Checks if the file (or version) associated with this metadata has been deleted. /// /// This function returns `true` if the file represented by this metadata has been marked for /// deletion or has been permanently deleted. /// It returns `false` otherwise, indicating that the file (or version) is still present and accessible. /// /// Refer to docs in [`Metadata`] for more information about file versions. /// /// # Returns /// /// `bool`: `true` if the object is considered deleted, `false` otherwise. pub fn is_deleted(&self) -> bool { self.is_deleted } /// Set the deleted status of this entry. pub fn set_is_deleted(&mut self, v: bool) -> &mut Self { self.is_deleted = v; self } /// Set the deleted status of this entry. pub fn with_is_deleted(mut self, v: bool) -> Self { self.is_deleted = v; self } /// Cache control of this entry. /// /// Cache-Control is defined by [RFC 7234](https://httpwg.org/specs/rfc7234.html#header.cache-control) /// Refer to [MDN Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) for more information. pub fn cache_control(&self) -> Option<&str> { self.cache_control.as_deref() } /// Set cache control of this entry. /// /// Cache-Control is defined by [RFC 7234](https://httpwg.org/specs/rfc7234.html#header.cache-control) /// Refer to [MDN Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) for more information. pub fn set_cache_control(&mut self, v: &str) -> &mut Self { self.cache_control = Some(v.to_string()); self } /// Set cache control of this entry. /// /// Cache-Control is defined by [RFC 7234](https://httpwg.org/specs/rfc7234.html#header.cache-control) /// Refer to [MDN Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) for more information. pub fn with_cache_control(mut self, v: String) -> Self { self.cache_control = Some(v); self } /// Content length of this entry. /// /// `Content-Length` is defined by [RFC 7230](https://httpwg.org/specs/rfc7230.html#header.content-length) /// /// Refer to [MDN Content-Length](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Length) for more information. /// /// # Returns /// /// Content length of this entry. It will be `0` if the content length is not set by the storage services. pub fn content_length(&self) -> u64 { self.content_length.unwrap_or_default() } /// Set content length of this entry. pub fn set_content_length(&mut self, v: u64) -> &mut Self { self.content_length = Some(v); self } /// Set content length of this entry. pub fn with_content_length(mut self, v: u64) -> Self { self.content_length = Some(v); self } /// Content MD5 of this entry. /// /// Content MD5 is defined by [RFC 2616](http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html). /// And removed by [RFC 7231](https://www.rfc-editor.org/rfc/rfc7231). /// /// OpenDAL will try its best to set this value, but not guarantee this value is the md5 of content. pub fn content_md5(&self) -> Option<&str> { self.content_md5.as_deref() } /// Set content MD5 of this entry. pub fn set_content_md5(&mut self, v: &str) -> &mut Self { self.content_md5 = Some(v.to_string()); self } /// Set content MD5 of this entry. pub fn with_content_md5(mut self, v: String) -> Self { self.content_md5 = Some(v); self } /// Content Type of this entry. /// /// Content Type is defined by [RFC 9110](https://httpwg.org/specs/rfc9110.html#field.content-type). /// /// Refer to [MDN Content-Type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type) for more information. pub fn content_type(&self) -> Option<&str> { self.content_type.as_deref() } /// Set Content Type of this entry. pub fn set_content_type(&mut self, v: &str) -> &mut Self { self.content_type = Some(v.to_string()); self } /// Set Content Type of this entry. pub fn with_content_type(mut self, v: String) -> Self { self.content_type = Some(v); self } /// Content Encoding of this entry. /// /// Content Encoding is defined by [RFC 7231](https://httpwg.org/specs/rfc7231.html#header.content-encoding) /// /// Refer to [MDN Content-Encoding](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding) for more information. pub fn content_encoding(&self) -> Option<&str> { self.content_encoding.as_deref() } /// Set Content Encoding of this entry. pub fn set_content_encoding(&mut self, v: &str) -> &mut Self { self.content_encoding = Some(v.to_string()); self } /// Content Range of this entry. /// /// Content Range is defined by [RFC 9110](https://httpwg.org/specs/rfc9110.html#field.content-range). /// /// Refer to [MDN Content-Range](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Range) for more information. pub fn content_range(&self) -> Option { self.content_range } /// Set Content Range of this entry. pub fn set_content_range(&mut self, v: BytesContentRange) -> &mut Self { self.content_range = Some(v); self } /// Set Content Range of this entry. pub fn with_content_range(mut self, v: BytesContentRange) -> Self { self.content_range = Some(v); self } /// Last modified of this entry. /// /// `Last-Modified` is defined by [RFC 7232](https://httpwg.org/specs/rfc7232.html#header.last-modified) /// /// Refer to [MDN Last-Modified](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified) for more information. pub fn last_modified(&self) -> Option> { self.last_modified } /// Set Last modified of this entry. pub fn set_last_modified(&mut self, v: DateTime) -> &mut Self { self.last_modified = Some(v); self } /// Set Last modified of this entry. pub fn with_last_modified(mut self, v: DateTime) -> Self { self.last_modified = Some(v); self } /// ETag of this entry. /// /// `ETag` is defined by [RFC 7232](https://httpwg.org/specs/rfc7232.html#header.etag) /// /// Refer to [MDN ETag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag) for more information. /// /// OpenDAL will return this value AS-IS like the following: /// /// - `"33a64df551425fcc55e4d42a148795d9f25f89d4"` /// - `W/"0815"` /// /// `"` is part of etag. pub fn etag(&self) -> Option<&str> { self.etag.as_deref() } /// Set ETag of this entry. pub fn set_etag(&mut self, v: &str) -> &mut Self { self.etag = Some(v.to_string()); self } /// Set ETag of this entry. pub fn with_etag(mut self, v: String) -> Self { self.etag = Some(v); self } /// Content-Disposition of this entry /// /// `Content-Disposition` is defined by [RFC 2616](https://www.rfc-editor/rfcs/2616) and /// clarified usage in [RFC 6266](https://www.rfc-editor/6266). /// /// Refer to [MDN Content-Disposition](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition) for more information. /// /// OpenDAL will return this value AS-IS like the following: /// /// - "inline" /// - "attachment" /// - "attachment; filename=\"filename.jpg\"" pub fn content_disposition(&self) -> Option<&str> { self.content_disposition.as_deref() } /// Set Content-Disposition of this entry pub fn set_content_disposition(&mut self, v: &str) -> &mut Self { self.content_disposition = Some(v.to_string()); self } /// Set Content-Disposition of this entry pub fn with_content_disposition(mut self, v: String) -> Self { self.content_disposition = Some(v); self } /// Retrieves the `version` of the file, if available. /// /// The version is typically used in systems that support object versioning, such as AWS S3. /// /// # Returns /// /// - `Some(&str)`: If the file has a version associated with it, /// this function returns `Some` containing a reference to the version ID string. /// - `None`: If the file does not have a version, or if versioning is /// not supported or enabled for the underlying storage system, this function /// returns `None`. pub fn version(&self) -> Option<&str> { self.version.as_deref() } /// Set the version of the file pub fn set_version(&mut self, v: &str) -> &mut Self { self.version = Some(v.to_string()); self } /// With the version of the file. pub fn with_version(mut self, v: String) -> Self { self.version = Some(v); self } /// User defined metadata of this entry /// /// The prefix of the user defined metadata key(for example: in oss, it's x-oss-meta-) /// is remove from the key pub fn user_metadata(&self) -> Option<&HashMap> { self.user_metadata.as_ref() } /// Set user defined metadata of this entry pub fn with_user_metadata(&mut self, data: HashMap) -> &mut Self { self.user_metadata = Some(data); self } } opendal-0.52.0/src/types/mod.rs000064400000000000000000000033051046102023000144130ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod mode; pub use mode::EntryMode; mod buffer; pub use buffer::Buffer; mod entry; pub use entry::Entry; mod metadata; pub use metadata::Metadata; mod read; pub use read::*; mod blocking_read; pub use blocking_read::*; mod write; pub use write::*; mod blocking_write; pub use blocking_write::*; mod list; pub use list::BlockingLister; pub use list::Lister; mod delete; pub use delete::*; mod execute; pub use execute::*; mod operator; pub use operator::operator_functions; pub use operator::operator_futures; pub use operator::BlockingOperator; pub use operator::Operator; pub use operator::OperatorBuilder; pub use operator::OperatorInfo; mod builder; pub use builder::Builder; pub use builder::Configurator; mod error; pub use error::Error; pub use error::ErrorKind; pub use error::Result; mod scheme; pub use scheme::Scheme; mod capability; pub use capability::Capability; mod context; pub(crate) use context::*; opendal-0.52.0/src/types/mode.rs000064400000000000000000000037031046102023000145620ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::Display; use std::fmt::Formatter; /// EntryMode represents the mode. #[derive(Copy, Clone, Debug, Eq, PartialEq)] pub enum EntryMode { /// FILE means the path has data to read. FILE, /// DIR means the path can be listed. DIR, /// Unknown means we don't know what we can do on this path. Unknown, } impl EntryMode { /// Check if this mode is FILE. pub fn is_file(self) -> bool { self == EntryMode::FILE } /// Check if this mode is DIR. pub fn is_dir(self) -> bool { self == EntryMode::DIR } /// Create entry mode from given path. #[allow(dead_code)] pub(crate) fn from_path(path: &str) -> Self { if path.ends_with('/') { EntryMode::DIR } else { EntryMode::FILE } } } impl Default for EntryMode { fn default() -> Self { Self::Unknown } } impl Display for EntryMode { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { match self { EntryMode::FILE => write!(f, "file"), EntryMode::DIR => write!(f, "dir"), EntryMode::Unknown => write!(f, "unknown"), } } } opendal-0.52.0/src/types/operator/blocking_operator.rs000064400000000000000000001052621046102023000211770ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use super::operator_functions::*; use crate::raw::oio::BlockingDelete; use crate::raw::*; use crate::*; /// BlockingOperator is the entry for all public blocking APIs. /// /// Read [`concepts`][docs::concepts] for know more about [`Operator`]. /// /// # Examples /// /// ## Init backends /// /// Read more backend init examples in [`services`] /// /// ```rust,no_run /// # use anyhow::Result; /// use opendal::services::Fs; /// use opendal::BlockingOperator; /// use opendal::Operator; /// /// fn main() -> Result<()> { /// // Create fs backend builder. /// let builder = Fs::default().root("/tmp"); /// /// // Build an `BlockingOperator` to start operating the storage. /// let _: BlockingOperator = Operator::new(builder)?.finish().blocking(); /// /// Ok(()) /// } /// ``` /// /// ## Init backends with blocking layer /// /// Some services like s3, gcs doesn't have native blocking supports, we can use [`layers::BlockingLayer`] /// to wrap the async operator to make it blocking. #[cfg_attr(feature = "layers-blocking", doc = "```rust")] #[cfg_attr(not(feature = "layers-blocking"), doc = "```ignore")] /// # use anyhow::Result; /// use opendal::layers::BlockingLayer; /// use opendal::services::S3; /// use opendal::BlockingOperator; /// use opendal::Operator; /// /// async fn test() -> Result<()> { /// // Create fs backend builder. /// let mut builder = S3::default().bucket("test").region("us-east-1"); /// /// // Build an `BlockingOperator` with blocking layer to start operating the storage. /// let _: BlockingOperator = Operator::new(builder)? /// .layer(BlockingLayer::create()?) /// .finish() /// .blocking(); /// /// Ok(()) /// } /// ``` #[derive(Clone, Debug)] pub struct BlockingOperator { accessor: Accessor, } impl BlockingOperator { pub(super) fn inner(&self) -> &Accessor { &self.accessor } /// create a new blocking operator from inner accessor. /// /// # Note /// default batch limit is 1000. pub(crate) fn from_inner(accessor: Accessor) -> Self { Self { accessor } } /// Get current operator's limit #[deprecated(note = "limit is no-op for now", since = "0.52.0")] pub fn limit(&self) -> usize { 0 } /// Specify the batch limit. /// /// Default: 1000 #[deprecated(note = "limit is no-op for now", since = "0.52.0")] pub fn with_limit(&self, _: usize) -> Self { self.clone() } /// Get information of underlying accessor. /// /// # Examples /// /// ``` /// # use std::sync::Arc; /// # use anyhow::Result; /// use opendal::BlockingOperator; /// /// # fn test(op: BlockingOperator) -> Result<()> { /// let info = op.info(); /// # Ok(()) /// # } /// ``` pub fn info(&self) -> OperatorInfo { OperatorInfo::new(self.accessor.info()) } } /// # Operator blocking API. impl BlockingOperator { /// Get given path's metadata. /// /// # Behavior /// /// ## Services that support `create_dir` /// /// `test` and `test/` may vary in some services such as S3. However, on a local file system, /// they're identical. Therefore, the behavior of `stat("test")` and `stat("test/")` might differ /// in certain edge cases. Always use `stat("test/")` when you need to access a directory if possible. /// /// Here are the behavior list: /// /// | Case | Path | Result | /// |------------------------|-----------------|--------------------------------------------| /// | stat existing dir | `abc/` | Metadata with dir mode | /// | stat existing file | `abc/def_file` | Metadata with file mode | /// | stat dir without `/` | `abc/def_dir` | Error `NotFound` or metadata with dir mode | /// | stat file with `/` | `abc/def_file/` | Error `NotFound` | /// | stat not existing path | `xyz` | Error `NotFound` | /// /// Refer to [RFC: List Prefix][crate::docs::rfcs::rfc_3243_list_prefix] for more details. /// /// ## Services that not support `create_dir` /// /// For services that not support `create_dir`, `stat("test/")` will return `NotFound` even /// when `test/abc` exists since the service won't have the concept of dir. There is nothing /// we can do about this. /// /// # Examples /// /// ## Check if file exists /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::BlockingOperator; /// use opendal::ErrorKind; /// # /// # fn test(op: BlockingOperator) -> Result<()> { /// if let Err(e) = op.stat("test") { /// if e.kind() == ErrorKind::NotFound { /// println!("file not exist") /// } /// } /// # Ok(()) /// # } /// ``` pub fn stat(&self, path: &str) -> Result { self.stat_with(path).call() } /// Get given path's metadata with extra options. /// /// # Behavior /// /// ## Services that support `create_dir` /// /// `test` and `test/` may vary in some services such as S3. However, on a local file system, /// they're identical. Therefore, the behavior of `stat("test")` and `stat("test/")` might differ /// in certain edge cases. Always use `stat("test/")` when you need to access a directory if possible. /// /// Here are the behavior list: /// /// | Case | Path | Result | /// |------------------------|-----------------|--------------------------------------------| /// | stat existing dir | `abc/` | Metadata with dir mode | /// | stat existing file | `abc/def_file` | Metadata with file mode | /// | stat dir without `/` | `abc/def_dir` | Error `NotFound` or metadata with dir mode | /// | stat file with `/` | `abc/def_file/` | Error `NotFound` | /// | stat not existing path | `xyz` | Error `NotFound` | /// /// Refer to [RFC: List Prefix][crate::docs::rfcs::rfc_3243_list_prefix] for more details. /// /// ## Services that not support `create_dir` /// /// For services that not support `create_dir`, `stat("test/")` will return `NotFound` even /// when `test/abc` exists since the service won't have the concept of dir. There is nothing /// we can do about this. /// /// # Examples /// /// ## Get metadata while `ETag` matches /// /// `stat_with` will /// /// - return `Ok(metadata)` if `ETag` matches /// - return `Err(error)` and `error.kind() == ErrorKind::ConditionNotMatch` if file exists but /// `ETag` mismatch /// - return `Err(err)` if other errors occur, for example, `NotFound`. /// /// ``` /// # use anyhow::Result; /// # use opendal::BlockingOperator; /// use opendal::ErrorKind; /// # /// # fn test(op: BlockingOperator) -> Result<()> { /// if let Err(e) = op.stat_with("test").if_match("").call() { /// if e.kind() == ErrorKind::ConditionNotMatch { /// println!("file exists, but etag mismatch") /// } /// if e.kind() == ErrorKind::NotFound { /// println!("file not exist") /// } /// } /// # Ok(()) /// # } /// ``` pub fn stat_with(&self, path: &str) -> FunctionStat { let path = normalize_path(path); FunctionStat(OperatorFunction::new( self.inner().clone(), path, OpStat::default(), |inner, path, args| { let rp = inner.blocking_stat(&path, args)?; let meta = rp.into_metadata(); Ok(meta) }, )) } /// Check if this path exists or not. /// /// # Example /// /// ```no_run /// use anyhow::Result; /// use opendal::BlockingOperator; /// fn test(op: BlockingOperator) -> Result<()> { /// let _ = op.exists("test")?; /// /// Ok(()) /// } /// ``` pub fn exists(&self, path: &str) -> Result { let r = self.stat(path); match r { Ok(_) => Ok(true), Err(err) => match err.kind() { ErrorKind::NotFound => Ok(false), _ => Err(err), }, } } /// Check if this path exists or not. /// /// # Example /// /// ```no_run /// use anyhow::Result; /// use opendal::BlockingOperator; /// fn test(op: BlockingOperator) -> Result<()> { /// let _ = op.is_exist("test")?; /// /// Ok(()) /// } /// ``` #[deprecated(note = "rename to `exists` for consistence with `std::fs::exists`")] pub fn is_exist(&self, path: &str) -> Result { self.exists(path) } /// Create a dir at given path. /// /// # Notes /// /// To indicate that a path is a directory, it is compulsory to include /// a trailing / in the path. Failure to do so may result in /// `NotADirectory` error being returned by OpenDAL. /// /// # Behavior /// /// - Create on existing dir will succeed. /// - Create dir is always recursive, works like `mkdir -p` /// /// # Examples /// /// ```no_run /// # use opendal::Result; /// # use opendal::BlockingOperator; /// # use futures::TryStreamExt; /// # fn test(op: BlockingOperator) -> Result<()> { /// op.create_dir("path/to/dir/")?; /// # Ok(()) /// # } /// ``` pub fn create_dir(&self, path: &str) -> Result<()> { let path = normalize_path(path); if !validate_path(&path, EntryMode::DIR) { return Err(Error::new( ErrorKind::NotADirectory, "the path trying to create should end with `/`", ) .with_operation("create_dir") .with_context("service", self.inner().info().scheme()) .with_context("path", &path)); } self.inner() .blocking_create_dir(&path, OpCreateDir::new())?; Ok(()) } /// Read the whole path into a bytes. /// /// This function will allocate a new bytes internally. For more precise memory control or /// reading data lazily, please use [`BlockingOperator::reader`] /// /// # Examples /// /// ```no_run /// # use opendal::Result; /// # use opendal::BlockingOperator; /// # /// # fn test(op: BlockingOperator) -> Result<()> { /// let bs = op.read("path/to/file")?; /// # Ok(()) /// # } /// ``` pub fn read(&self, path: &str) -> Result { self.read_with(path).call() } /// Read the whole path into a bytes with extra options. /// /// This function will allocate a new bytes internally. For more precise memory control or /// reading data lazily, please use [`BlockingOperator::reader`] /// /// # Examples /// /// ```no_run /// # use anyhow::Result; /// use opendal::BlockingOperator; /// use opendal::EntryMode; /// # fn test(op: BlockingOperator) -> Result<()> { /// let bs = op.read_with("path/to/file").range(0..10).call()?; /// # Ok(()) /// # } /// ``` pub fn read_with(&self, path: &str) -> FunctionRead { let path = normalize_path(path); FunctionRead(OperatorFunction::new( self.inner().clone(), path, (OpRead::default(), BytesRange::default()), |inner, path, (args, range)| { if !validate_path(&path, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "read path is a directory") .with_operation("BlockingOperator::read_with") .with_context("service", inner.info().scheme().into_static()) .with_context("path", &path), ); } let context = ReadContext::new(inner, path, args, OpReader::default()); let r = BlockingReader::new(context); let buf = r.read(range.to_range())?; Ok(buf) }, )) } /// Create a new reader which can read the whole path. /// /// # Examples /// /// ```no_run /// # use opendal::Result; /// # use opendal::BlockingOperator; /// # use futures::TryStreamExt; /// # fn test(op: BlockingOperator) -> Result<()> { /// let r = op.reader("path/to/file")?; /// # Ok(()) /// # } /// ``` pub fn reader(&self, path: &str) -> Result { self.reader_with(path).call() } /// Create a new reader with extra options /// /// # Examples /// /// ```no_run /// # use anyhow::Result; /// use opendal::BlockingOperator; /// use opendal::EntryMode; /// # fn test(op: BlockingOperator) -> Result<()> { /// let r = op /// .reader_with("path/to/file") /// .version("version_id") /// .call()?; /// # Ok(()) /// # } /// ``` pub fn reader_with(&self, path: &str) -> FunctionReader { let path = normalize_path(path); FunctionReader(OperatorFunction::new( self.inner().clone(), path, OpRead::default(), |inner, path, args| { if !validate_path(&path, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "reader path is a directory") .with_operation("BlockingOperator::reader_with") .with_context("service", inner.info().scheme().into_static()) .with_context("path", &path), ); } let context = ReadContext::new(inner, path, args, OpReader::default()); Ok(BlockingReader::new(context)) }, )) } /// Write bytes into given path. /// /// # Notes /// /// - Write will make sure all bytes has been written, or an error will be returned. /// /// # Examples /// /// ```no_run /// # use opendal::Result; /// # use opendal::BlockingOperator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # fn test(op: BlockingOperator) -> Result<()> { /// op.write("path/to/file", vec![0; 4096])?; /// # Ok(()) /// # } /// ``` pub fn write(&self, path: &str, bs: impl Into) -> Result { self.write_with(path, bs).call() } /// Copy a file from `from` to `to`. /// /// # Notes /// /// - `from` and `to` must be a file. /// - `to` will be overwritten if it exists. /// - If `from` and `to` are the same, nothing will happen. /// - `copy` is idempotent. For same `from` and `to` input, the result will be the same. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::BlockingOperator; /// /// # fn test(op: BlockingOperator) -> Result<()> { /// op.copy("path/to/file", "path/to/file2")?; /// # Ok(()) /// # } /// ``` pub fn copy(&self, from: &str, to: &str) -> Result<()> { let from = normalize_path(from); if !validate_path(&from, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "from path is a directory") .with_operation("BlockingOperator::copy") .with_context("service", self.info().scheme()) .with_context("from", from), ); } let to = normalize_path(to); if !validate_path(&to, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "to path is a directory") .with_operation("BlockingOperator::copy") .with_context("service", self.info().scheme()) .with_context("to", to), ); } if from == to { return Err( Error::new(ErrorKind::IsSameFile, "from and to paths are same") .with_operation("BlockingOperator::copy") .with_context("service", self.info().scheme()) .with_context("from", from) .with_context("to", to), ); } self.inner().blocking_copy(&from, &to, OpCopy::new())?; Ok(()) } /// Rename a file from `from` to `to`. /// /// # Notes /// /// - `from` and `to` must be a file. /// - `to` will be overwritten if it exists. /// - If `from` and `to` are the same, a `IsSameFile` error will occur. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::BlockingOperator; /// /// # fn test(op: BlockingOperator) -> Result<()> { /// op.rename("path/to/file", "path/to/file2")?; /// # Ok(()) /// # } /// ``` pub fn rename(&self, from: &str, to: &str) -> Result<()> { let from = normalize_path(from); if !validate_path(&from, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "from path is a directory") .with_operation("BlockingOperator::move") .with_context("service", self.info().scheme()) .with_context("from", from), ); } let to = normalize_path(to); if !validate_path(&to, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "to path is a directory") .with_operation("BlockingOperator::move") .with_context("service", self.info().scheme()) .with_context("to", to), ); } if from == to { return Err( Error::new(ErrorKind::IsSameFile, "from and to paths are same") .with_operation("BlockingOperator::move") .with_context("service", self.info().scheme()) .with_context("from", from) .with_context("to", to), ); } self.inner().blocking_rename(&from, &to, OpRename::new())?; Ok(()) } /// Write data with options. /// /// # Notes /// /// - Write will make sure all bytes has been written, or an error will be returned. /// /// # Examples /// /// ```no_run /// # use opendal::Result; /// # use opendal::BlockingOperator; /// use bytes::Bytes; /// /// # fn test(op: BlockingOperator) -> Result<()> { /// let bs = b"hello, world!".to_vec(); /// let _ = op /// .write_with("hello.txt", bs) /// .content_type("text/plain") /// .call()?; /// # Ok(()) /// # } /// ``` pub fn write_with(&self, path: &str, bs: impl Into) -> FunctionWrite { let path = normalize_path(path); let bs = bs.into(); FunctionWrite(OperatorFunction::new( self.inner().clone(), path, (OpWrite::default(), OpWriter::default(), bs), |inner, path, (args, options, bs)| { if !validate_path(&path, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "write path is a directory") .with_operation("BlockingOperator::write_with") .with_context("service", inner.info().scheme().into_static()) .with_context("path", &path), ); } let context = WriteContext::new(inner, path, args, options); let mut w = BlockingWriter::new(context)?; w.write(bs)?; w.close() }, )) } /// Write multiple bytes into given path. /// /// # Notes /// /// - Write will make sure all bytes has been written, or an error will be returned. /// /// # Examples /// /// ```no_run /// # use opendal::Result; /// # use opendal::BlockingOperator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # fn test(op: BlockingOperator) -> Result<()> { /// let mut w = op.writer("path/to/file")?; /// w.write(vec![0; 4096])?; /// w.write(vec![1; 4096])?; /// w.close()?; /// # Ok(()) /// # } /// ``` pub fn writer(&self, path: &str) -> Result { self.writer_with(path).call() } /// Create a new reader with extra options /// /// # Examples /// /// ```no_run /// # use anyhow::Result; /// use opendal::BlockingOperator; /// use opendal::EntryMode; /// # fn test(op: BlockingOperator) -> Result<()> { /// let mut w = op.writer_with("path/to/file").call()?; /// w.write(vec![0; 4096])?; /// w.write(vec![1; 4096])?; /// w.close()?; /// # Ok(()) /// # } /// ``` pub fn writer_with(&self, path: &str) -> FunctionWriter { let path = normalize_path(path); FunctionWriter(OperatorFunction::new( self.inner().clone(), path, (OpWrite::default(), OpWriter::default()), |inner, path, (args, options)| { let path = normalize_path(&path); if !validate_path(&path, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "write path is a directory") .with_operation("BlockingOperator::writer_with") .with_context("service", inner.info().scheme().into_static()) .with_context("path", &path), ); } let context = WriteContext::new(inner, path, args, options); let w = BlockingWriter::new(context)?; Ok(w) }, )) } /// Delete given path. /// /// # Notes /// /// - Delete not existing error won't return errors. /// /// # Examples /// /// ```no_run /// # use anyhow::Result; /// # use futures::io; /// # use opendal::BlockingOperator; /// # fn test(op: BlockingOperator) -> Result<()> { /// op.delete("path/to/file")?; /// # Ok(()) /// # } /// ``` pub fn delete(&self, path: &str) -> Result<()> { self.delete_with(path).call()?; Ok(()) } /// Delete given path with options. /// /// # Notes /// /// - Delete not existing error won't return errors. /// /// # Examples /// /// ```no_run /// # use anyhow::Result; /// # use futures::io; /// # use opendal::BlockingOperator; /// # fn test(op: BlockingOperator) -> Result<()> { /// let _ = op /// .delete_with("path/to/file") /// .version("example_version") /// .call()?; /// # Ok(()) /// # } /// ``` pub fn delete_with(&self, path: &str) -> FunctionDelete { let path = normalize_path(path); FunctionDelete(OperatorFunction::new( self.inner().clone(), path, OpDelete::new(), |inner, path, args| { let (_, mut deleter) = inner.blocking_delete()?; deleter.delete(&path, args)?; deleter.flush()?; Ok(()) }, )) } /// Delete an infallible iterator of paths. /// /// Also see: /// /// - [`BlockingOperator::delete_try_iter`]: delete an fallible iterator of paths. pub fn delete_iter(&self, iter: I) -> Result<()> where I: IntoIterator, D: IntoDeleteInput, { let mut deleter = self.deleter()?; deleter.delete_iter(iter)?; deleter.close()?; Ok(()) } /// Delete a fallible iterator of paths. /// /// Also see: /// /// - [`BlockingOperator::delete_iter`]: delete an infallible iterator of paths. pub fn delete_try_iter(&self, try_iter: I) -> Result<()> where I: IntoIterator>, D: IntoDeleteInput, { let mut deleter = self.deleter()?; deleter.delete_try_iter(try_iter)?; deleter.close()?; Ok(()) } /// Create a [`BlockingDeleter`] to continuously remove content from storage. /// /// It leverages batch deletion capabilities provided by storage services for efficient removal. /// /// Users can have more control over the deletion process by using [`BlockingDeleter`] directly. pub fn deleter(&self) -> Result { BlockingDeleter::create(self.inner().clone()) } /// remove will remove files via the given paths. /// /// remove_via will remove files via the given vector iterators. /// /// # Notes /// /// We don't support batch delete now. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::BlockingOperator; /// # fn test(op: BlockingOperator) -> Result<()> { /// let stream = vec!["abc".to_string(), "def".to_string()].into_iter(); /// op.remove_via(stream)?; /// # Ok(()) /// # } /// ``` #[deprecated(note = "use `BlockingOperator::delete_iter` instead", since = "0.52.0")] pub fn remove_via(&self, input: impl Iterator) -> Result<()> { for path in input { self.delete(&path)?; } Ok(()) } /// # Notes /// /// We don't support batch delete now. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::BlockingOperator; /// # fn test(op: BlockingOperator) -> Result<()> { /// op.remove(vec!["abc".to_string(), "def".to_string()])?; /// # Ok(()) /// # } /// ``` #[deprecated(note = "use `BlockingOperator::delete_iter` instead", since = "0.52.0")] pub fn remove(&self, paths: Vec) -> Result<()> { self.delete_iter(paths) } /// Remove the path and all nested dirs and files recursively. /// /// # Notes /// /// We don't support batch delete now. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::BlockingOperator; /// # fn test(op: BlockingOperator) -> Result<()> { /// op.remove_all("path/to/dir")?; /// # Ok(()) /// # } /// ``` pub fn remove_all(&self, path: &str) -> Result<()> { match self.stat(path) { Ok(metadata) => { if metadata.mode() != EntryMode::DIR { self.delete(path)?; // There may still be objects prefixed with the path in some backend, so we can't return here. } } // If dir not found, it may be a prefix in object store like S3, // and we still need to delete objects under the prefix. Err(e) if e.kind() == ErrorKind::NotFound => {} Err(e) => return Err(e), }; let lister = self.lister_with(path).recursive(true).call()?; self.delete_try_iter(lister)?; Ok(()) } /// List entries that starts with given `path` in parent dir. /// /// # Notes /// /// ## Recursively List /// /// This function only read the children of the given directory. To read /// all entries recursively, use `BlockingOperator::list_with("path").recursive(true)` /// instead. /// /// ## Streaming List /// /// This function will read all entries in the given directory. It could /// take very long time and consume a lot of memory if the directory /// contains a lot of entries. /// /// In order to avoid this, you can use [`BlockingOperator::lister`] to list entries in /// a streaming way. /// /// # Examples /// /// ```no_run /// # use anyhow::Result; /// use opendal::BlockingOperator; /// use opendal::EntryMode; /// # fn test(op: BlockingOperator) -> Result<()> { /// let mut entries = op.list("path/to/dir/")?; /// for entry in entries { /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file") /// } /// EntryMode::DIR => { /// println!("Handling dir {}", entry.path()) /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` pub fn list(&self, path: &str) -> Result> { self.list_with(path).call() } /// List entries that starts with given `path` in parent dir. with options. /// /// # Notes /// /// ## Streaming List /// /// This function will read all entries in the given directory. It could /// take very long time and consume a lot of memory if the directory /// contains a lot of entries. /// /// In order to avoid this, you can use [`BlockingOperator::lister`] to list entries in /// a streaming way. /// /// # Examples /// /// ## List entries with prefix /// /// This function can also be used to list entries in recursive way. /// /// ```no_run /// # use anyhow::Result; /// use opendal::BlockingOperator; /// use opendal::EntryMode; /// # fn test(op: BlockingOperator) -> Result<()> { /// let mut entries = op.list_with("prefix/").recursive(true).call()?; /// for entry in entries { /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file") /// } /// EntryMode::DIR => { /// println!("Handling dir like start a new list via meta.path()") /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` pub fn list_with(&self, path: &str) -> FunctionList { let path = normalize_path(path); FunctionList(OperatorFunction::new( self.inner().clone(), path, OpList::default(), |inner, path, args| { let lister = BlockingLister::create(inner, &path, args)?; lister.collect() }, )) } /// List entries that starts with given `path` in parent dir. /// /// This function will create a new [`BlockingLister`] to list entries. Users can stop listing /// via dropping this [`Lister`]. /// /// # Notes /// /// ## Recursively List /// /// This function only read the children of the given directory. To read /// all entries recursively, use [`BlockingOperator::lister_with`] and `delimiter("")` /// instead. /// /// # Examples /// /// ```no_run /// # use anyhow::Result; /// # use futures::io; /// use futures::TryStreamExt; /// use opendal::BlockingOperator; /// use opendal::EntryMode; /// # fn test(op: BlockingOperator) -> Result<()> { /// let mut ds = op.lister("path/to/dir/")?; /// for de in ds { /// let de = de?; /// match de.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file") /// } /// EntryMode::DIR => { /// println!("Handling dir like start a new list via meta.path()") /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` pub fn lister(&self, path: &str) -> Result { self.lister_with(path).call() } /// List entries within a given directory as an iterator with options. /// /// This function will create a new handle to list entries. /// /// An error will be returned if given path doesn't end with `/`. /// /// # Examples /// /// ## List current dir /// /// ```no_run /// # use anyhow::Result; /// # use futures::io; /// use futures::TryStreamExt; /// use opendal::BlockingOperator; /// use opendal::EntryMode; /// # fn test(op: BlockingOperator) -> Result<()> { /// let mut ds = op /// .lister_with("path/to/dir/") /// .limit(10) /// .start_after("start") /// .call()?; /// for entry in ds { /// let entry = entry?; /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file {}", entry.path()) /// } /// EntryMode::DIR => { /// println!("Handling dir {}", entry.path()) /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` /// /// ## List all files recursively /// /// ```no_run /// # use anyhow::Result; /// # use futures::io; /// use futures::TryStreamExt; /// use opendal::BlockingOperator; /// use opendal::EntryMode; /// # fn test(op: BlockingOperator) -> Result<()> { /// let mut ds = op.lister_with("path/to/dir/").recursive(true).call()?; /// for entry in ds { /// let entry = entry?; /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file {}", entry.path()) /// } /// EntryMode::DIR => { /// println!("Handling dir {}", entry.path()) /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` pub fn lister_with(&self, path: &str) -> FunctionLister { let path = normalize_path(path); FunctionLister(OperatorFunction::new( self.inner().clone(), path, OpList::default(), |inner, path, args| BlockingLister::create(inner, &path, args), )) } } impl From for Operator { fn from(v: BlockingOperator) -> Self { Operator::from_inner(v.accessor) } } opendal-0.52.0/src/types/operator/builder.rs000064400000000000000000000504641046102023000171250ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::sync::Arc; use crate::layers::*; use crate::raw::*; use crate::*; /// # Operator build API /// /// Operator should be built via [`OperatorBuilder`]. We recommend to use [`Operator::new`] to get started: /// /// ``` /// # use anyhow::Result; /// use opendal::services::Fs; /// use opendal::Operator; /// async fn test() -> Result<()> { /// // Create fs backend builder. /// let builder = Fs::default().root("/tmp"); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::new(builder)?.finish(); /// /// Ok(()) /// } /// ``` impl Operator { /// Create a new operator with input builder. /// /// OpenDAL will call `builder.build()` internally, so we don't need /// to import `opendal::Builder` trait. /// /// # Examples /// /// Read more backend init examples in [examples](https://github.com/apache/opendal/tree/main/examples). /// /// ``` /// # use anyhow::Result; /// use opendal::services::Fs; /// use opendal::Operator; /// async fn test() -> Result<()> { /// // Create fs backend builder. /// let builder = Fs::default().root("/tmp"); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::new(builder)?.finish(); /// /// Ok(()) /// } /// ``` #[allow(clippy::new_ret_no_self)] pub fn new(ab: B) -> Result> { let acc = ab.build()?; Ok(OperatorBuilder::new(acc)) } /// Create a new operator from given config. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// use std::collections::HashMap; /// /// use opendal::services::MemoryConfig; /// use opendal::Operator; /// async fn test() -> Result<()> { /// let cfg = MemoryConfig::default(); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::from_config(cfg)?.finish(); /// /// Ok(()) /// } /// ``` pub fn from_config(cfg: C) -> Result> { let builder = cfg.into_builder(); let acc = builder.build()?; Ok(OperatorBuilder::new(acc)) } /// Create a new operator from given iterator in static dispatch. /// /// # Notes /// /// `from_iter` generates a `OperatorBuilder` which allows adding layer in zero-cost way. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// use std::collections::HashMap; /// /// use opendal::services::Fs; /// use opendal::Operator; /// async fn test() -> Result<()> { /// let map = HashMap::from([ /// // Set the root for fs, all operations will happen under this root. /// // /// // NOTE: the root must be absolute path. /// ("root".to_string(), "/tmp".to_string()), /// ]); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::from_iter::(map)?.finish(); /// /// Ok(()) /// } /// ``` #[allow(clippy::should_implement_trait)] pub fn from_iter( iter: impl IntoIterator, ) -> Result> { let builder = B::Config::from_iter(iter)?.into_builder(); let acc = builder.build()?; Ok(OperatorBuilder::new(acc)) } /// Create a new operator via given scheme and iterator of config value in dynamic dispatch. /// /// # Notes /// /// `via_iter` generates a `Operator` which allows building operator without generic type. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// use std::collections::HashMap; /// /// use opendal::Operator; /// use opendal::Scheme; /// async fn test() -> Result<()> { /// let map = [ /// // Set the root for fs, all operations will happen under this root. /// // /// // NOTE: the root must be absolute path. /// ("root".to_string(), "/tmp".to_string()), /// ]; /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::via_iter(Scheme::Fs, map)?; /// /// Ok(()) /// } /// ``` #[allow(unused_variables, unreachable_code)] pub fn via_iter( scheme: Scheme, iter: impl IntoIterator, ) -> Result { let op = match scheme { #[cfg(feature = "services-aliyun-drive")] Scheme::AliyunDrive => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-atomicserver")] Scheme::Atomicserver => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-alluxio")] Scheme::Alluxio => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-compfs")] Scheme::Compfs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-upyun")] Scheme::Upyun => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-koofr")] Scheme::Koofr => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-yandex-disk")] Scheme::YandexDisk => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-pcloud")] Scheme::Pcloud => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-chainsafe")] Scheme::Chainsafe => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-azblob")] Scheme::Azblob => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-azdls")] Scheme::Azdls => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-azfile")] Scheme::Azfile => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-b2")] Scheme::B2 => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-cacache")] Scheme::Cacache => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-cos")] Scheme::Cos => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-d1")] Scheme::D1 => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-dashmap")] Scheme::Dashmap => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-dropbox")] Scheme::Dropbox => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-etcd")] Scheme::Etcd => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-foundationdb")] Scheme::Foundationdb => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-fs")] Scheme::Fs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-ftp")] Scheme::Ftp => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-gcs")] Scheme::Gcs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-ghac")] Scheme::Ghac => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-gridfs")] Scheme::Gridfs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-github")] Scheme::Github => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-hdfs")] Scheme::Hdfs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-http")] Scheme::Http => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-huggingface")] Scheme::Huggingface => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-ipfs")] Scheme::Ipfs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-ipmfs")] Scheme::Ipmfs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-icloud")] Scheme::Icloud => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-memcached")] Scheme::Memcached => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-memory")] Scheme::Memory => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-mini-moka")] Scheme::MiniMoka => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-moka")] Scheme::Moka => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-monoiofs")] Scheme::Monoiofs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-mysql")] Scheme::Mysql => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-obs")] Scheme::Obs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-onedrive")] Scheme::Onedrive => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-postgresql")] Scheme::Postgresql => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-gdrive")] Scheme::Gdrive => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-oss")] Scheme::Oss => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-persy")] Scheme::Persy => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-redis")] Scheme::Redis => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-rocksdb")] Scheme::Rocksdb => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-s3")] Scheme::S3 => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-seafile")] Scheme::Seafile => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-sftp")] Scheme::Sftp => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-sled")] Scheme::Sled => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-sqlite")] Scheme::Sqlite => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-supabase")] Scheme::Supabase => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-swift")] Scheme::Swift => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-tikv")] Scheme::Tikv => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-vercel-artifacts")] Scheme::VercelArtifacts => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-vercel-blob")] Scheme::VercelBlob => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-webdav")] Scheme::Webdav => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-webhdfs")] Scheme::Webhdfs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-redb")] Scheme::Redb => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-mongodb")] Scheme::Mongodb => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-hdfs-native")] Scheme::HdfsNative => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-lakefs")] Scheme::Lakefs => Self::from_iter::(iter)?.finish(), #[cfg(feature = "services-nebula-graph")] Scheme::NebulaGraph => Self::from_iter::(iter)?.finish(), v => { return Err(Error::new( ErrorKind::Unsupported, "scheme is not enabled or supported", ) .with_context("scheme", v)) } }; Ok(op) } /// Create a new operator from given map. /// /// # Notes /// /// from_map is using static dispatch layers which is zero cost. via_map is /// using dynamic dispatch layers which has a bit runtime overhead with an /// extra vtable lookup and unable to inline. But from_map requires generic /// type parameter which is not always easy to be used. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// use std::collections::HashMap; /// /// use opendal::services::Fs; /// use opendal::Operator; /// async fn test() -> Result<()> { /// let map = HashMap::from([ /// // Set the root for fs, all operations will happen under this root. /// // /// // NOTE: the root must be absolute path. /// ("root".to_string(), "/tmp".to_string()), /// ]); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::from_map::(map)?.finish(); /// /// Ok(()) /// } /// ``` #[deprecated = "use from_iter instead"] pub fn from_map( map: HashMap, ) -> Result> { Self::from_iter::(map) } /// Create a new operator from given scheme and map. /// /// # Notes /// /// from_map is using static dispatch layers which is zero cost. via_map is /// using dynamic dispatch layers which has a bit runtime overhead with an /// extra vtable lookup and unable to inline. But from_map requires generic /// type parameter which is not always easy to be used. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// use std::collections::HashMap; /// /// use opendal::Operator; /// use opendal::Scheme; /// async fn test() -> Result<()> { /// let map = HashMap::from([ /// // Set the root for fs, all operations will happen under this root. /// // /// // NOTE: the root must be absolute path. /// ("root".to_string(), "/tmp".to_string()), /// ]); /// /// // Build an `Operator` to start operating the storage. /// let op: Operator = Operator::via_map(Scheme::Fs, map)?; /// /// Ok(()) /// } /// ``` #[deprecated = "use via_iter instead"] pub fn via_map(scheme: Scheme, map: HashMap) -> Result { Self::via_iter(scheme, map) } /// Create a new layer with dynamic dispatch. /// /// # Notes /// /// `OperatorBuilder::layer()` is using static dispatch which is zero /// cost. `Operator::layer()` is using dynamic dispatch which has a /// bit runtime overhead with an extra vtable lookup and unable to /// inline. /// /// It's always recommended to use `OperatorBuilder::layer()` instead. /// /// # Examples /// /// ```no_run /// # use std::sync::Arc; /// # use anyhow::Result; /// use opendal::layers::LoggingLayer; /// use opendal::services::Fs; /// use opendal::Operator; /// /// # async fn test() -> Result<()> { /// let op = Operator::new(Fs::default())?.finish(); /// let op = op.layer(LoggingLayer::default()); /// // All operations will go through the new_layer /// let _ = op.read("test_file").await?; /// # Ok(()) /// # } /// ``` #[must_use] pub fn layer>(self, layer: L) -> Self { Self::from_inner(Arc::new( TypeEraseLayer.layer(layer.layer(self.into_inner())), )) } } /// OperatorBuilder is a typed builder to build an Operator. /// /// # Notes /// /// OpenDAL uses static dispatch internally and only performs dynamic /// dispatch at the outmost type erase layer. OperatorBuilder is the only /// public API provided by OpenDAL come with generic parameters. /// /// It's required to call `finish` after the operator built. /// /// # Examples /// /// For users who want to support many services, we can build a helper function like the following: /// /// ``` /// use std::collections::HashMap; /// /// use opendal::layers::LoggingLayer; /// use opendal::layers::RetryLayer; /// use opendal::services; /// use opendal::Builder; /// use opendal::Operator; /// use opendal::Result; /// use opendal::Scheme; /// /// fn init_service(cfg: HashMap) -> Result { /// let op = Operator::from_map::(cfg)? /// .layer(LoggingLayer::default()) /// .layer(RetryLayer::new()) /// .finish(); /// /// Ok(op) /// } /// /// async fn init(scheme: Scheme, cfg: HashMap) -> Result<()> { /// let _ = match scheme { /// Scheme::S3 => init_service::(cfg)?, /// Scheme::Fs => init_service::(cfg)?, /// _ => todo!(), /// }; /// /// Ok(()) /// } /// ``` pub struct OperatorBuilder { accessor: A, } impl OperatorBuilder { /// Create a new operator builder. #[allow(clippy::new_ret_no_self)] pub fn new(accessor: A) -> OperatorBuilder { // Make sure error context layer has been attached. OperatorBuilder { accessor } .layer(ErrorContextLayer) .layer(CompleteLayer) .layer(CorrectnessCheckLayer) } /// Create a new layer with static dispatch. /// /// # Notes /// /// `OperatorBuilder::layer()` is using static dispatch which is zero /// cost. `Operator::layer()` is using dynamic dispatch which has a /// bit runtime overhead with an extra vtable lookup and unable to /// inline. /// /// It's always recommended to use `OperatorBuilder::layer()` instead. /// /// # Examples /// /// ```no_run /// # use std::sync::Arc; /// # use anyhow::Result; /// use opendal::layers::LoggingLayer; /// use opendal::services::Fs; /// use opendal::Operator; /// /// # async fn test() -> Result<()> { /// let op = Operator::new(Fs::default())? /// .layer(LoggingLayer::default()) /// .finish(); /// // All operations will go through the new_layer /// let _ = op.read("test_file").await?; /// # Ok(()) /// # } /// ``` #[must_use] pub fn layer>(self, layer: L) -> OperatorBuilder { OperatorBuilder { accessor: layer.layer(self.accessor), } } /// Finish the building to construct an Operator. pub fn finish(self) -> Operator { let ob = self.layer(TypeEraseLayer); Operator::from_inner(Arc::new(ob.accessor) as Accessor) } } opendal-0.52.0/src/types/operator/metadata.rs000064400000000000000000000035711046102023000172540ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use crate::raw::*; use crate::*; /// Metadata for operator, users can use this metadata to get information of operator. #[derive(Clone, Debug, Default)] pub struct OperatorInfo(Arc); impl OperatorInfo { pub(super) fn new(acc: Arc) -> Self { OperatorInfo(acc) } /// [`Scheme`] of operator. pub fn scheme(&self) -> Scheme { self.0.scheme() } /// Root of operator, will be in format like `/path/to/dir/` pub fn root(&self) -> &str { self.0.root() } /// Name of backend, could be empty if underlying backend doesn't have namespace concept. /// /// For example: /// /// - name for `s3` => bucket name /// - name for `azblob` => container name pub fn name(&self) -> &str { self.0.name() } /// Get [`Full Capability`] of operator. pub fn full_capability(&self) -> Capability { self.0.full_capability() } /// Get [`Native Capability`] of operator. pub fn native_capability(&self) -> Capability { self.0.native_capability() } } opendal-0.52.0/src/types/operator/mod.rs000064400000000000000000000022011046102023000162400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Operator's API will be split into different mods. #[allow(clippy::module_inception)] mod operator; pub use operator::Operator; mod blocking_operator; pub use blocking_operator::BlockingOperator; mod builder; pub use builder::OperatorBuilder; mod metadata; pub use metadata::OperatorInfo; pub mod operator_functions; pub mod operator_futures; opendal-0.52.0/src/types/operator/operator.rs000064400000000000000000002034371046102023000173320ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use std::time::Duration; use futures::Stream; use futures::StreamExt; use futures::TryStreamExt; use super::BlockingOperator; use crate::operator_futures::*; use crate::raw::oio::DeleteDyn; use crate::raw::*; use crate::types::delete::Deleter; use crate::*; /// The `Operator` serves as the entry point for all public asynchronous APIs. /// /// For more details about the `Operator`, refer to the [`concepts`][crate::docs::concepts] section. /// /// ## Build /// /// Users can initialize an `Operator` through the following methods: /// /// - [`Operator::new`]: Creates an operator using a [`services`] builder, such as [`services::S3`]. /// - [`Operator::from_config`]: Creates an operator using a [`services`] configuration, such as [`services::S3Config`]. /// - [`Operator::from_iter`]: Creates an operator from an iterator of configuration key-value pairs. /// /// ``` /// # use anyhow::Result; /// use opendal::services::Memory; /// use opendal::Operator; /// async fn test() -> Result<()> { /// // Build an `Operator` to start operating the storage. /// let _: Operator = Operator::new(Memory::default())?.finish(); /// /// Ok(()) /// } /// ``` /// /// ## Layer /// /// After the operator is built, users can add the layers they need on top of it. /// /// OpenDAL offers various layers for users to choose from, such as `RetryLayer`, `LoggingLayer`, and more. Visit [`layers`] for further details. /// /// ``` /// # use anyhow::Result; /// use opendal::layers::RetryLayer; /// use opendal::services::Memory; /// use opendal::Operator; /// async fn test() -> Result<()> { /// let op: Operator = Operator::new(Memory::default())?.finish(); /// /// // OpenDAL will retry failed operations now. /// let op = op.layer(RetryLayer::default()); /// /// Ok(()) /// } /// ``` /// /// ## Operate /// /// After the operator is built and the layers are added, users can start operating the storage. /// /// The operator is `Send`, `Sync`, and `Clone`. It has no internal state, and all APIs only take /// a `&self` reference, making it safe to share the operator across threads. /// /// Operator provides a consistent API pattern for data operations. For reading operations, it exposes: /// /// - [`Operator::read`]: Basic operation that reads entire content into memory /// - [`Operator::read_with`]: Enhanced read operation with additional options (range, if_match, if_none_match) /// - [`Operator::reader`]: Creates a lazy reader for on-demand data streaming /// - [`Operator::reader_with`]: Creates a configurable reader with conditional options (if_match, if_none_match) /// /// The [`Reader`] created by [`Operator`] supports custom read control methods and can be converted /// into `futures::AsyncRead` for broader ecosystem compatibility. /// /// ``` /// # use anyhow::Result; /// use opendal::layers::RetryLayer; /// use opendal::services::Memory; /// use opendal::Operator; /// async fn test() -> Result<()> { /// let op: Operator = Operator::new(Memory::default())?.finish(); /// /// // OpenDAL will retry failed operations now. /// let op = op.layer(RetryLayer::default()); /// /// // Read all data into memory. /// let data = op.read("path/to/file").await?; /// /// Ok(()) /// } /// ``` #[derive(Clone, Debug)] pub struct Operator { // accessor is what Operator delegates for accessor: Accessor, /// The default executor that used to run futures in background. default_executor: Option, } /// # Operator basic API. impl Operator { /// Fetch the internal accessor. pub fn inner(&self) -> &Accessor { &self.accessor } /// Convert inner accessor into operator. pub fn from_inner(accessor: Accessor) -> Self { Self { accessor, default_executor: None, } } /// Convert operator into inner accessor. pub fn into_inner(self) -> Accessor { self.accessor } /// Get current operator's limit. /// Limit is usually the maximum size of data that operator will handle in one operation. #[deprecated(note = "limit is no-op for now", since = "0.52.0")] pub fn limit(&self) -> usize { 0 } /// Specify the batch limit. /// /// Default: 1000 #[deprecated(note = "limit is no-op for now", since = "0.52.0")] pub fn with_limit(&self, _: usize) -> Self { self.clone() } /// Get the default executor. pub fn default_executor(&self) -> Option { self.default_executor.clone() } /// Specify the default executor. pub fn with_default_executor(&self, executor: Executor) -> Self { let mut op = self.clone(); op.default_executor = Some(executor); op } /// Get information of underlying accessor. /// /// # Examples /// /// ``` /// # use std::sync::Arc; /// # use anyhow::Result; /// use opendal::Operator; /// /// # async fn test(op: Operator) -> Result<()> { /// let info = op.info(); /// # Ok(()) /// # } /// ``` pub fn info(&self) -> OperatorInfo { OperatorInfo::new(self.accessor.info()) } /// Create a new blocking operator. /// /// This operation is nearly no cost. pub fn blocking(&self) -> BlockingOperator { BlockingOperator::from_inner(self.accessor.clone()) } } /// # Operator async API. impl Operator { /// Check if this operator can work correctly. /// /// We will send a `list` request to path and return any errors we met. /// /// ``` /// # use std::sync::Arc; /// # use anyhow::Result; /// use opendal::Operator; /// /// # async fn test(op: Operator) -> Result<()> { /// op.check().await?; /// # Ok(()) /// # } /// ``` pub async fn check(&self) -> Result<()> { let mut ds = self.lister("/").await?; match ds.next().await { Some(Err(e)) if e.kind() != ErrorKind::NotFound => Err(e), _ => Ok(()), } } /// Get given path's metadata. /// /// # Notes /// /// ## Extra Options /// /// [`Operator::stat`] is a wrapper of [`Operator::stat_with`] without any options. To use extra /// options like `if_match` and `if_none_match`, please use [`Operator::stat_with`] instead. /// /// # Examples /// /// ## Check if file exists /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::Operator; /// use opendal::ErrorKind; /// # /// # async fn test(op: Operator) -> Result<()> { /// if let Err(e) = op.stat("test").await { /// if e.kind() == ErrorKind::NotFound { /// println!("file not exist") /// } /// } /// # Ok(()) /// # } /// ``` pub async fn stat(&self, path: &str) -> Result { self.stat_with(path).await } /// Get given path's metadata with extra options. /// /// # Options /// /// ## `if_match` /// /// Set `if_match` for this `stat` request. /// /// This feature can be used to check if the file's `ETag` matches the given `ETag`. /// /// If file exists, and it's etag doesn't match, an error with kind [`ErrorKind::ConditionNotMatch`] /// will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// /// # async fn test(op: Operator, etag: &str) -> Result<()> { /// let mut metadata = op.stat_with("path/to/file").if_match(etag).await?; /// # Ok(()) /// # } /// ``` /// /// ## `if_none_match` /// /// Set `if_none_match` for this `stat` request. /// /// This feature can be used to check if the file's `ETag` doesn't match the given `ETag`. /// /// If file exists, and it's etag match, an error with kind [`ErrorKind::ConditionNotMatch`] /// will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// /// # async fn test(op: Operator, etag: &str) -> Result<()> { /// let mut metadata = op.stat_with("path/to/file").if_none_match(etag).await?; /// # Ok(()) /// # } /// ``` /// /// ## `if_modified_since` /// /// set `if_modified_since` for this `stat` request. /// /// This feature can be used to check if the file has been modified since the given time. /// /// If file exists, and it's not modified after the given time, an error with kind [`ErrorKind::ConditionNotMatch`] /// will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// use chrono::Utc; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut metadata = op.stat_with("path/to/file").if_modified_since(Utc::now()).await?; /// # Ok(()) /// # } /// ``` /// /// ## `if_unmodified_since` /// /// set `if_unmodified_since` for this `stat` request. /// /// This feature can be used to check if the file has NOT been modified since the given time. /// /// If file exists, and it's modified after the given time, an error with kind [`ErrorKind::ConditionNotMatch`] /// will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// use chrono::Utc; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut metadata = op.stat_with("path/to/file").if_unmodified_since(Utc::now()).await?; /// # Ok(()) /// # } /// ``` /// /// ## `version` /// /// Set `version` for this `stat` request. /// /// This feature can be used to retrieve the metadata of a specific version of the given path /// /// If the version doesn't exist, an error with kind [`ErrorKind::NotFound`] will be returned. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// /// # async fn test(op: Operator, version: &str) -> Result<()> { /// let mut metadata = op.stat_with("path/to/file").version(version).await?; /// # Ok(()) /// # } /// ``` /// /// # Examples /// /// ## Get metadata while `ETag` matches /// /// `stat_with` will /// /// - return `Ok(metadata)` if `ETag` matches /// - return `Err(error)` and `error.kind() == ErrorKind::ConditionNotMatch` if file exists but /// `ETag` mismatch /// - return `Err(err)` if other errors occur, for example, `NotFound`. /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::Operator; /// use opendal::ErrorKind; /// # /// # async fn test(op: Operator) -> Result<()> { /// if let Err(e) = op.stat_with("test").if_match("").await { /// if e.kind() == ErrorKind::ConditionNotMatch { /// println!("file exists, but etag mismatch") /// } /// if e.kind() == ErrorKind::NotFound { /// println!("file not exist") /// } /// } /// # Ok(()) /// # } /// ``` /// /// --- /// /// # Behavior /// /// ## Services that support `create_dir` /// /// `test` and `test/` may vary in some services such as S3. However, on a local file system, /// they're identical. Therefore, the behavior of `stat("test")` and `stat("test/")` might differ /// in certain edge cases. Always use `stat("test/")` when you need to access a directory if possible. /// /// Here are the behavior list: /// /// | Case | Path | Result | /// |------------------------|-----------------|--------------------------------------------| /// | stat existing dir | `abc/` | Metadata with dir mode | /// | stat existing file | `abc/def_file` | Metadata with file mode | /// | stat dir without `/` | `abc/def_dir` | Error `NotFound` or metadata with dir mode | /// | stat file with `/` | `abc/def_file/` | Error `NotFound` | /// | stat not existing path | `xyz` | Error `NotFound` | /// /// Refer to [RFC: List Prefix][crate::docs::rfcs::rfc_3243_list_prefix] for more details. /// /// ## Services that not support `create_dir` /// /// For services that not support `create_dir`, `stat("test/")` will return `NotFound` even /// when `test/abc` exists since the service won't have the concept of dir. There is nothing /// we can do about this. pub fn stat_with(&self, path: &str) -> FutureStat>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, OpStat::default(), |inner, path, args| async move { let rp = inner.stat(&path, args).await?; Ok(rp.into_metadata()) }, ) } /// Check if this path exists or not. /// /// # Example /// /// ``` /// use anyhow::Result; /// use futures::io; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let _ = op.exists("test").await?; /// /// Ok(()) /// } /// ``` pub async fn exists(&self, path: &str) -> Result { let r = self.stat(path).await; match r { Ok(_) => Ok(true), Err(err) => match err.kind() { ErrorKind::NotFound => Ok(false), _ => Err(err), }, } } /// Check if this path exists or not. /// /// # Example /// /// ``` /// use anyhow::Result; /// use futures::io; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let _ = op.is_exist("test").await?; /// /// Ok(()) /// } /// ``` #[deprecated(note = "rename to `exists` for consistence with `std::fs::exists`")] pub async fn is_exist(&self, path: &str) -> Result { let r = self.stat(path).await; match r { Ok(_) => Ok(true), Err(err) => match err.kind() { ErrorKind::NotFound => Ok(false), _ => Err(err), }, } } /// Create a dir at given path. /// /// # Notes /// /// To indicate that a path is a directory, it is compulsory to include /// a trailing / in the path. Failure to do so may result in /// `NotADirectory` error being returned by OpenDAL. /// /// # Behavior /// /// - Create on existing dir will succeed. /// - Create dir is always recursive, works like `mkdir -p` /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// op.create_dir("path/to/dir/").await?; /// # Ok(()) /// # } /// ``` pub async fn create_dir(&self, path: &str) -> Result<()> { let path = normalize_path(path); if !validate_path(&path, EntryMode::DIR) { return Err(Error::new( ErrorKind::NotADirectory, "the path trying to create should end with `/`", ) .with_operation("create_dir") .with_context("service", self.inner().info().scheme()) .with_context("path", &path)); } self.inner().create_dir(&path, OpCreateDir::new()).await?; Ok(()) } /// Read the whole path into a bytes. /// /// # Notes /// /// ## Extra Options /// /// [`Operator::read`] is a wrapper of [`Operator::read_with`] without any options. To use /// extra options like `range` and `if_match`, please use [`Operator::read_with`] instead. /// /// ## Streaming Read /// /// This function will allocate a new bytes internally. For more precise memory control or /// reading data lazily, please use [`Operator::reader`] /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::TryStreamExt; /// # async fn test(op: Operator) -> Result<()> { /// let bs = op.read("path/to/file").await?; /// # Ok(()) /// # } /// ``` pub async fn read(&self, path: &str) -> Result { self.read_with(path).await } /// Read the whole path into a bytes with extra options. /// /// This function will allocate a new bytes internally. For more precise memory control or /// reading data lazily, please use [`Operator::reader`] /// /// # Notes /// /// ## Streaming Read /// /// This function will allocate a new bytes internally. For more precise memory control or /// reading data lazily, please use [`Operator::reader`] /// /// # Options /// /// Visit [`FutureRead`] for all available options. /// /// - [`range`](./operator_futures/type.FutureRead.html#method.version): Set `range` for the read. /// - [`concurrent`](./operator_futures/type.FutureRead.html#method.concurrent): Set `concurrent` for the read. /// - [`chunk`](./operator_futures/type.FutureRead.html#method.chunk): Set `chunk` for the read. /// - [`version`](./operator_futures/type.FutureRead.html#method.version): Set `version` for the read. /// - [`if_match`](./operator_futures/type.FutureRead.html#method.if_match): Set `if-match` for the read. /// - [`if_none_match`](./operator_futures/type.FutureRead.html#method.if_none_match): Set `if-none-match` for the read. /// - [`if_modified_since`](./operator_futures/type.FutureRead.html#method.if_modified_since): Set `if-modified-since` for the read. /// - [`if_unmodified_since`](./operator_futures/type.FutureRead.html#method.if_unmodified_since): Set `if-unmodified-since` for the read. /// /// # Examples /// /// Read the whole path into a bytes. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::TryStreamExt; /// # async fn test(op: Operator) -> Result<()> { /// let bs = op.read_with("path/to/file").await?; /// let bs = op.read_with("path/to/file").range(0..10).await?; /// # Ok(()) /// # } /// ``` pub fn read_with(&self, path: &str) -> FutureRead>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, ( OpRead::default().merge_executor(self.default_executor.clone()), OpReader::default(), ), |inner, path, (args, options)| async move { if !validate_path(&path, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "read path is a directory") .with_operation("read") .with_context("service", inner.info().scheme()) .with_context("path", &path), ); } let range = args.range(); let context = ReadContext::new(inner, path, args, options); let r = Reader::new(context); let buf = r.read(range.to_range()).await?; Ok(buf) }, ) } /// Create a new reader which can read the whole path. /// /// # Notes /// /// ## Extra Options /// /// [`Operator::reader`] is a wrapper of [`Operator::reader_with`] without any options. To use /// extra options like `concurrent`, please use [`Operator::reader_with`] instead. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::TryStreamExt; /// # use opendal::Scheme; /// # async fn test(op: Operator) -> Result<()> { /// let r = op.reader("path/to/file").await?; /// # Ok(()) /// # } /// ``` pub async fn reader(&self, path: &str) -> Result { self.reader_with(path).await } /// Create a new reader with extra options /// /// # Notes /// /// ## Extra Options /// /// [`Operator::reader`] is a wrapper of [`Operator::reader_with`] without any options. To use /// extra options like `version`, please use [`Operator::reader_with`] instead. /// /// # Options /// /// Visit [`FutureReader`] for all available options. /// /// - [`version`](./operator_futures/type.FutureReader.html#method.version): Set `version` for the reader. /// - [`concurrent`](./operator_futures/type.FutureReader.html#method.concurrent): Set `concurrent` for the reader. /// - [`chunk`](./operator_futures/type.FutureReader.html#method.chunk): Set `chunk` for the reader. /// - [`gap`](./operator_futures/type.FutureReader.html#method.gap): Set `gap` for the reader. /// - [`if_match`](./operator_futures/type.FutureReader.html#method.if_match): Set `if-match` for the reader. /// - [`if_none_match`](./operator_futures/type.FutureReader.html#method.if_none_match): Set `if-none-match` for the reader. /// - [`if_modified_since`](./operator_futures/type.FutureReader.html#method.if_modified_since): Set `if-modified-since` for the reader. /// - [`if_unmodified_since`](./operator_futures/type.FutureReader.html#method.if_unmodified_since): Set `if-unmodified-since` for the reader. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use opendal::Scheme; /// # async fn test(op: Operator) -> Result<()> { /// let r = op.reader_with("path/to/file").version("version_id").await?; /// # Ok(()) /// # } /// ``` pub fn reader_with(&self, path: &str) -> FutureReader>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, ( OpRead::default().merge_executor(self.default_executor.clone()), OpReader::default(), ), |inner, path, (args, options)| async move { if !validate_path(&path, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "read path is a directory") .with_operation("Operator::reader") .with_context("service", inner.info().scheme()) .with_context("path", path), ); } let context = ReadContext::new(inner, path, args, options); Ok(Reader::new(context)) }, ) } /// Write bytes into path. /// /// # Notes /// /// ## Extra Options /// /// [`Operator::write`] is a simplified version of [`Operator::write_with`] without additional options. /// For advanced features like `content_type` and `cache_control`, use [`Operator::write_with`] instead. /// /// ## Streaming Write /// /// This method performs a single bulk write operation. For finer-grained memory control /// or streaming data writes, use [`Operator::writer`] instead. /// /// ## Multipart Uploads /// /// OpenDAL provides multipart upload functionality through the [`Writer`] abstraction, /// handling all upload details automatically. You can customize the upload behavior by /// configuring `chunk` size and `concurrent` operations via [`Operator::writer_with`]. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// op.write("path/to/file", vec![0; 4096]).await?; /// # Ok(()) /// # } /// ``` pub async fn write(&self, path: &str, bs: impl Into) -> Result { let bs = bs.into(); self.write_with(path, bs).await } /// Copy a file from `from` to `to`. /// /// # Notes /// /// - `from` and `to` must be a file. /// - `to` will be overwritten if it exists. /// - If `from` and `to` are the same, an `IsSameFile` error will occur. /// - `copy` is idempotent. For same `from` and `to` input, the result will be the same. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// /// # async fn test(op: Operator) -> Result<()> { /// op.copy("path/to/file", "path/to/file2").await?; /// # Ok(()) /// # } /// ``` pub async fn copy(&self, from: &str, to: &str) -> Result<()> { let from = normalize_path(from); if !validate_path(&from, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "from path is a directory") .with_operation("Operator::copy") .with_context("service", self.info().scheme()) .with_context("from", from), ); } let to = normalize_path(to); if !validate_path(&to, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "to path is a directory") .with_operation("Operator::copy") .with_context("service", self.info().scheme()) .with_context("to", to), ); } if from == to { return Err( Error::new(ErrorKind::IsSameFile, "from and to paths are same") .with_operation("Operator::copy") .with_context("service", self.info().scheme()) .with_context("from", from) .with_context("to", to), ); } self.inner().copy(&from, &to, OpCopy::new()).await?; Ok(()) } /// Rename a file from `from` to `to`. /// /// # Notes /// /// - `from` and `to` must be a file. /// - `to` will be overwritten if it exists. /// - If `from` and `to` are the same, an `IsSameFile` error will occur. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// /// # async fn test(op: Operator) -> Result<()> { /// op.rename("path/to/file", "path/to/file2").await?; /// # Ok(()) /// # } /// ``` pub async fn rename(&self, from: &str, to: &str) -> Result<()> { let from = normalize_path(from); if !validate_path(&from, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "from path is a directory") .with_operation("Operator::move_") .with_context("service", self.info().scheme()) .with_context("from", from), ); } let to = normalize_path(to); if !validate_path(&to, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "to path is a directory") .with_operation("Operator::move_") .with_context("service", self.info().scheme()) .with_context("to", to), ); } if from == to { return Err( Error::new(ErrorKind::IsSameFile, "from and to paths are same") .with_operation("Operator::move_") .with_context("service", self.info().scheme()) .with_context("from", from) .with_context("to", to), ); } self.inner().rename(&from, &to, OpRename::new()).await?; Ok(()) } /// Create a writer for streaming data to the given path. /// /// # Notes /// /// ## Writer Features /// /// The writer provides several powerful capabilities: /// - Streaming writes for continuous data transfer /// - Automatic multipart upload handling /// - Memory-efficient chunk-based writing /// /// ## Extra Options /// /// [`Operator::writer`] is a simplified version of [`Operator::writer_with`] without additional options. /// For advanced features like `content_type` and `cache_control`, use [`Operator::writer_with`] instead. /// /// ## Chunk Size Handling /// /// Storage services often have specific requirements for chunk sizes: /// - Services like `s3` may return `EntityTooSmall` errors for undersized chunks /// - Using small chunks in cloud storage services can lead to increased costs /// /// OpenDAL automatically determines optimal chunk sizes based on the service's /// [Capability](crate::types::Capability). However, you can override this by explicitly /// setting the `chunk` parameter. /// /// For improved performance, consider setting an appropriate chunk size using /// [`Operator::writer_with`]. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op.writer("path/to/file").await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub async fn writer(&self, path: &str) -> Result { self.writer_with(path).await } /// Create a writer for streaming data to the given path with more options. /// /// ## Options /// /// Visit [`FutureWriter`] for all available options. /// /// - [`append`](./operator_futures/type.FutureWriter.html#method.append): Sets append mode for this write request. /// - [`chunk`](./operator_futures/type.FutureWriter.html#method.chunk): Sets chunk size for buffered writes. /// - [`concurrent`](./operator_futures/type.FutureWriter.html#method.concurrent): Sets concurrent write operations for this writer. /// - [`cache_control`](./operator_futures/type.FutureWriter.html#method.cache_control): Sets cache control for this write request. /// - [`content_type`](./operator_futures/type.FutureWriter.html#method.content_type): Sets content type for this write request. /// - [`content_disposition`](./operator_futures/type.FutureWriter.html#method.content_disposition): Sets content disposition for this write request. /// - [`content_encoding`](./operator_futures/type.FutureWriter.html#method.content_encoding): Sets content encoding for this write request. /// - [`if_match`](./operator_futures/type.FutureWriter.html#method.if_match): Sets if-match for this write request. /// - [`if_none_match`](./operator_futures/type.FutureWriter.html#method.if_none_match): Sets if-none-match for this write request. /// - [`if_not_exist`](./operator_futures/type.FutureWriter.html#method.if_not_exist): Sets if-not-exist for this write request. /// - [`user_metadata`](./operator_futures/type.FutureWriter.html#method.user_metadata): Sets user metadata for this write request. /// /// ## Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op.writer_with("path/to/file") /// .chunk(4*1024*1024) /// .concurrent(8) /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn writer_with(&self, path: &str) -> FutureWriter>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, ( OpWrite::default().merge_executor(self.default_executor.clone()), OpWriter::default(), ), |inner, path, (args, options)| async move { if !validate_path(&path, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "write path is a directory") .with_operation("Operator::writer") .with_context("service", inner.info().scheme().into_static()) .with_context("path", &path), ); } let context = WriteContext::new(inner, path, args, options); let w = Writer::new(context).await?; Ok(w) }, ) } /// Write data with extra options. /// /// # Notes /// /// ## Streaming Write /// /// This method performs a single bulk write operation for all bytes. For finer-grained /// memory control or lazy writing, consider using [`Operator::writer_with`] instead. /// /// ## Multipart Uploads /// /// OpenDAL handles multipart uploads through the [`Writer`] abstraction, managing all /// the upload details automatically. You can customize the upload behavior by configuring /// `chunk` size and `concurrent` operations via [`Operator::writer_with`]. /// /// # Options /// /// Visit [`FutureWrite`] for all available options. /// /// - [`append`](./operator_futures/type.FutureWrite.html#method.append): Sets append mode for this write request. /// - [`chunk`](./operator_futures/type.FutureWrite.html#method.chunk): Sets chunk size for buffered writes. /// - [`concurrent`](./operator_futures/type.FutureWrite.html#method.concurrent): Sets concurrent write operations for this writer. /// - [`cache_control`](./operator_futures/type.FutureWrite.html#method.cache_control): Sets cache control for this write request. /// - [`content_type`](./operator_futures/type.FutureWrite.html#method.content_type): Sets content type for this write request. /// - [`content_disposition`](./operator_futures/type.FutureWrite.html#method.content_disposition): Sets content disposition for this write request. /// - [`content_encoding`](./operator_futures/type.FutureWrite.html#method.content_encoding): Sets content encoding for this write request. /// - [`if_match`](./operator_futures/type.FutureWrite.html#method.if_match): Sets if-match for this write request. /// - [`if_none_match`](./operator_futures/type.FutureWrite.html#method.if_none_match): Sets if-none-match for this write request. /// - [`if_not_exist`](./operator_futures/type.FutureWrite.html#method.if_not_exist): Sets if-not-exist for this write request. /// - [`user_metadata`](./operator_futures/type.FutureWrite.html#method.user_metadata): Sets user metadata for this write request. /// /// # Examples /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let _ = op.write_with("path/to/file", vec![0; 4096]) /// .if_not_exists(true) /// .await?; /// # Ok(()) /// # } /// ``` pub fn write_with( &self, path: &str, bs: impl Into, ) -> FutureWrite>> { let path = normalize_path(path); let bs = bs.into(); OperatorFuture::new( self.inner().clone(), path, ( OpWrite::default().merge_executor(self.default_executor.clone()), OpWriter::default(), bs, ), |inner, path, (args, options, bs)| async move { if !validate_path(&path, EntryMode::FILE) { return Err( Error::new(ErrorKind::IsADirectory, "write path is a directory") .with_operation("Operator::write_with") .with_context("service", inner.info().scheme().into_static()) .with_context("path", &path), ); } let context = WriteContext::new(inner, path, args, options); let mut w = Writer::new(context).await?; w.write(bs).await?; w.close().await }, ) } /// Delete the given path. /// /// # Notes /// /// - Deleting a file that does not exist won't return errors. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// op.delete("test").await?; /// # Ok(()) /// # } /// ``` pub async fn delete(&self, path: &str) -> Result<()> { self.delete_with(path).await } /// Delete the given path with extra options. /// /// # Notes /// /// - Deleting a file that does not exist won't return errors. /// /// # Options /// /// ## `version` /// /// Set `version` for this `delete` request. /// /// remove a specific version of the given path. /// /// If the version doesn't exist, OpenDAL will not return errors. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// /// # async fn test(op: Operator, version: &str) -> Result<()> { /// op.delete_with("path/to/file").version(version).await?; /// # Ok(()) /// # } ///``` /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::Operator; /// /// # async fn test(op: Operator) -> Result<()> { /// op.delete_with("test").await?; /// # Ok(()) /// # } /// ``` pub fn delete_with(&self, path: &str) -> FutureDelete>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, OpDelete::default(), |inner, path, args| async move { let (_, mut deleter) = inner.delete_dyn().await?; deleter.delete_dyn(&path, args)?; deleter.flush_dyn().await?; Ok(()) }, ) } /// Delete an infallible iterator of paths. /// /// Also see: /// /// - [`Operator::delete_try_iter`]: delete an fallible iterator of paths. /// - [`Operator::delete_stream`]: delete an infallible stream of paths. /// - [`Operator::delete_try_stream`]: delete an fallible stream of paths. pub async fn delete_iter(&self, iter: I) -> Result<()> where I: IntoIterator, D: IntoDeleteInput, { let mut deleter = self.deleter().await?; deleter.delete_iter(iter).await?; deleter.close().await?; Ok(()) } /// Delete a fallible iterator of paths. /// /// Also see: /// /// - [`Operator::delete_iter`]: delete an infallible iterator of paths. /// - [`Operator::delete_stream`]: delete an infallible stream of paths. /// - [`Operator::delete_try_stream`]: delete an fallible stream of paths. pub async fn delete_try_iter(&self, try_iter: I) -> Result<()> where I: IntoIterator>, D: IntoDeleteInput, { let mut deleter = self.deleter().await?; deleter.delete_try_iter(try_iter).await?; deleter.close().await?; Ok(()) } /// Delete an infallible stream of paths. /// /// Also see: /// /// - [`Operator::delete_iter`]: delete an infallible iterator of paths. /// - [`Operator::delete_try_iter`]: delete an fallible iterator of paths. /// - [`Operator::delete_try_stream`]: delete an fallible stream of paths. pub async fn delete_stream(&self, stream: S) -> Result<()> where S: Stream, D: IntoDeleteInput, { let mut deleter = self.deleter().await?; deleter.delete_stream(stream).await?; deleter.close().await?; Ok(()) } /// Delete an fallible stream of paths. /// /// Also see: /// /// - [`Operator::delete_iter`]: delete an infallible iterator of paths. /// - [`Operator::delete_try_iter`]: delete an fallible iterator of paths. /// - [`Operator::delete_stream`]: delete an infallible stream of paths. pub async fn delete_try_stream(&self, try_stream: S) -> Result<()> where S: Stream>, D: IntoDeleteInput, { let mut deleter = self.deleter().await?; deleter.delete_try_stream(try_stream).await?; deleter.close().await?; Ok(()) } /// Create a [`Deleter`] to continuously remove content from storage. /// /// It leverages batch deletion capabilities provided by storage services for efficient removal. /// /// Users can have more control over the deletion process by using [`Deleter`] directly. pub async fn deleter(&self) -> Result { Deleter::create(self.inner().clone()).await } /// # Notes /// /// If underlying services support delete in batch, we will use batch /// delete instead. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::Operator; /// # /// # async fn test(op: Operator) -> Result<()> { /// op.remove(vec!["abc".to_string(), "def".to_string()]) /// .await?; /// # Ok(()) /// # } /// ``` #[deprecated(note = "use `Operator::delete_iter` instead", since = "0.52.0")] pub async fn remove(&self, paths: Vec) -> Result<()> { let mut deleter = self.deleter().await?; deleter.delete_iter(paths).await?; deleter.close().await?; Ok(()) } /// remove will remove files via the given paths. /// /// remove_via will remove files via the given stream. /// /// We will delete by chunks with given batch limit on the stream. /// /// # Notes /// /// If underlying services support delete in batch, we will use batch /// delete instead. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::Operator; /// use futures::stream; /// # /// # async fn test(op: Operator) -> Result<()> { /// let stream = stream::iter(vec!["abc".to_string(), "def".to_string()]); /// op.remove_via(stream).await?; /// # Ok(()) /// # } /// ``` #[deprecated(note = "use `Operator::delete_stream` instead", since = "0.52.0")] pub async fn remove_via(&self, input: impl Stream + Unpin) -> Result<()> { let mut deleter = self.deleter().await?; deleter .delete_stream(input.map(|v| normalize_path(&v))) .await?; deleter.close().await?; Ok(()) } /// Remove the path and all nested dirs and files recursively. /// /// # Notes /// /// If underlying services support delete in batch, we will use batch /// delete instead. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// # use opendal::Operator; /// # /// # async fn test(op: Operator) -> Result<()> { /// op.remove_all("path/to/dir").await?; /// # Ok(()) /// # } /// ``` pub async fn remove_all(&self, path: &str) -> Result<()> { match self.stat(path).await { // If object exists. Ok(metadata) => { // If the object is a file, we can delete it. if metadata.mode() != EntryMode::DIR { self.delete(path).await?; // There may still be objects prefixed with the path in some backend, so we can't return here. } } // If dir not found, it may be a prefix in object store like S3, // and we still need to delete objects under the prefix. Err(e) if e.kind() == ErrorKind::NotFound => {} // Pass on any other error. Err(e) => return Err(e), }; let lister = self.lister_with(path).recursive(true).await?; self.delete_try_stream(lister).await?; Ok(()) } /// List entries that starts with given `path` in parent dir. /// /// # Notes /// /// ## Recursively List /// /// This function only read the children of the given directory. To read /// all entries recursively, use `Operator::list_with("path").recursive(true)` /// instead. /// /// ## Streaming List /// /// This function will read all entries in the given directory. It could /// take very long time and consume a lot of memory if the directory /// contains a lot of entries. /// /// In order to avoid this, you can use [`Operator::lister`] to list entries in /// a streaming way. /// /// # Examples /// /// ## List entries under a dir /// /// This example will list all entries under the dir `path/to/dir/`. /// /// ``` /// # use anyhow::Result; /// use opendal::EntryMode; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut entries = op.list("path/to/dir/").await?; /// for entry in entries { /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file") /// } /// EntryMode::DIR => { /// println!("Handling dir {}", entry.path()) /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` /// /// ## List entries with prefix /// /// This example will list all entries under the prefix `path/to/prefix`. /// /// NOTE: it's possible that the prefix itself is also a dir. In this case, you could get /// `path/to/prefix/`, `path/to/prefix_1` and so on. If you do want to list a dir, please /// make sure the path is end with `/`. /// /// ``` /// # use anyhow::Result; /// use opendal::EntryMode; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut entries = op.list("path/to/prefix").await?; /// for entry in entries { /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file") /// } /// EntryMode::DIR => { /// println!("Handling dir {}", entry.path()) /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` pub async fn list(&self, path: &str) -> Result> { self.list_with(path).await } /// List entries that starts with given `path` in parent dir with more options. /// /// # Notes /// /// ## Streaming list /// /// This function will read all entries in the given directory. It could /// take very long time and consume a lot of memory if the directory /// contains a lot of entries. /// /// In order to avoid this, you can use [`Operator::lister`] to list entries in /// a streaming way. /// /// # Options /// /// ## `start_after` /// /// Specify the specified key to start listing from. /// /// This feature can be used to resume a listing from a previous point. /// /// The following example will resume the list operation from the `breakpoint`. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut entries = op /// .list_with("path/to/dir/") /// .start_after("breakpoint") /// .await?; /// # Ok(()) /// # } /// ``` /// /// ## `recursive` /// /// Specify whether to list recursively or not. /// /// If `recursive` is set to `true`, we will list all entries recursively. If not, we'll only /// list the entries in the specified dir. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut entries = op.list_with("path/to/dir/").recursive(true).await?; /// # Ok(()) /// # } /// ``` /// /// ## `version` /// /// Specify whether to list files along with all their versions /// /// if `version` is enabled, all file versions will be returned; otherwise, /// only the current files will be returned. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut entries = op.list_with("path/to/dir/").version(true).await?; /// # Ok(()) /// # } /// ``` /// /// # Examples /// /// ## List all entries recursively /// /// This example will list all entries under the dir `path/to/dir/` /// /// ``` /// # use anyhow::Result; /// use opendal::EntryMode; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut entries = op.list_with("path/to/dir/").recursive(true).await?; /// for entry in entries { /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file") /// } /// EntryMode::DIR => { /// println!("Handling dir like start a new list via meta.path()") /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` /// /// ## List all entries start with prefix /// /// This example will list all entries starts with prefix `path/to/prefix` /// /// ``` /// # use anyhow::Result; /// use opendal::EntryMode; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut entries = op.list_with("path/to/prefix").recursive(true).await?; /// for entry in entries { /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file") /// } /// EntryMode::DIR => { /// println!("Handling dir like start a new list via meta.path()") /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` pub fn list_with(&self, path: &str) -> FutureList>>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, OpList::default(), |inner, path, args| async move { let lister = Lister::create(inner, &path, args).await?; lister.try_collect().await }, ) } /// List entries that starts with given `path` in parent dir. /// /// This function will create a new [`Lister`] to list entries. Users can stop /// listing via dropping this [`Lister`]. /// /// # Notes /// /// ## Recursively list /// /// This function only read the children of the given directory. To read /// all entries recursively, use [`Operator::lister_with`] and `recursive(true)` /// instead. /// /// # Examples /// /// ``` /// # use anyhow::Result; /// # use futures::io; /// use futures::TryStreamExt; /// use opendal::EntryMode; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut ds = op.lister("path/to/dir/").await?; /// while let Some(mut de) = ds.try_next().await? { /// match de.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file") /// } /// EntryMode::DIR => { /// println!("Handling dir like start a new list via meta.path()") /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` pub async fn lister(&self, path: &str) -> Result { self.lister_with(path).await } /// List entries that starts with given `path` in parent dir with options. /// /// This function will create a new [`Lister`] to list entries. Users can stop listing via /// dropping this [`Lister`]. /// /// # Options /// /// ## `start_after` /// /// Specify the specified key to start listing from. /// /// This feature can be used to resume a listing from a previous point. /// /// The following example will resume the list operation from the `breakpoint`. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut lister = op /// .lister_with("path/to/dir/") /// .start_after("breakpoint") /// .await?; /// # Ok(()) /// # } /// ``` /// /// ## `recursive` /// /// Specify whether to list recursively or not. /// /// If `recursive` is set to `true`, we will list all entries recursively. If not, we'll only /// list the entries in the specified dir. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut lister = op.lister_with("path/to/dir/").recursive(true).await?; /// # Ok(()) /// # } /// ``` /// /// ## `version` /// /// Specify whether to list files along with all their versions /// /// if `version` is enabled, all file versions will be returned; otherwise, /// only the current files will be returned. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut entries = op.lister_with("path/to/dir/").version(true).await?; /// # Ok(()) /// # } /// ``` /// /// # Examples /// /// ## List all files recursively /// /// ``` /// # use anyhow::Result; /// use futures::TryStreamExt; /// use opendal::EntryMode; /// use opendal::Operator; /// # async fn test(op: Operator) -> Result<()> { /// let mut lister = op.lister_with("path/to/dir/").recursive(true).await?; /// while let Some(mut entry) = lister.try_next().await? { /// match entry.metadata().mode() { /// EntryMode::FILE => { /// println!("Handling file {}", entry.path()) /// } /// EntryMode::DIR => { /// println!("Handling dir {}", entry.path()) /// } /// EntryMode::Unknown => continue, /// } /// } /// # Ok(()) /// # } /// ``` pub fn lister_with(&self, path: &str) -> FutureLister>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, OpList::default(), |inner, path, args| async move { Lister::create(inner, &path, args).await }, ) } } /// Operator presign API. impl Operator { /// Presign an operation for stat(head). /// /// # Example /// /// ``` /// use anyhow::Result; /// use futures::io; /// use opendal::Operator; /// use std::time::Duration; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op.presign_stat("test",Duration::from_secs(3600)).await?; /// let req = http::Request::builder() /// .method(signed_req.method()) /// .uri(signed_req.uri()) /// .body(())?; /// /// # Ok(()) /// # } /// ``` pub async fn presign_stat(&self, path: &str, expire: Duration) -> Result { let path = normalize_path(path); let op = OpPresign::new(OpStat::new(), expire); let rp = self.inner().presign(&path, op).await?; Ok(rp.into_presigned_request()) } /// Presign an operation for stat(head). /// /// # Example /// /// ``` /// use anyhow::Result; /// use futures::io; /// use opendal::Operator; /// use std::time::Duration; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op.presign_stat_with("test",Duration::from_secs(3600)).override_content_disposition("attachment; filename=\"othertext.txt\"").await?; /// # Ok(()) /// # } /// ``` pub fn presign_stat_with( &self, path: &str, expire: Duration, ) -> FuturePresignStat>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, (OpStat::default(), expire), |inner, path, (args, dur)| async move { let op = OpPresign::new(args, dur); let rp = inner.presign(&path, op).await?; Ok(rp.into_presigned_request()) }, ) } /// Presign an operation for read. /// /// # Notes /// /// ## Extra Options /// /// `presign_read` is a wrapper of [`Self::presign_read_with`] without any options. To use /// extra options like `override_content_disposition`, please use [`Self::presign_read_with`] /// instead. /// /// # Example /// /// ``` /// use anyhow::Result; /// use futures::io; /// use opendal::Operator; /// use std::time::Duration; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op.presign_read("test.txt", Duration::from_secs(3600)).await?; /// # Ok(()) /// # } /// ``` /// /// - `signed_req.method()`: `GET` /// - `signed_req.uri()`: `https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=` /// - `signed_req.headers()`: `{ "host": "s3.amazonaws.com" }` /// /// We can download this file via `curl` or other tools without credentials: /// /// ```shell /// curl "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=" -O /tmp/test.txt /// ``` pub async fn presign_read(&self, path: &str, expire: Duration) -> Result { let path = normalize_path(path); let op = OpPresign::new(OpRead::new(), expire); let rp = self.inner().presign(&path, op).await?; Ok(rp.into_presigned_request()) } /// Presign an operation for read with extra options. /// /// # Options /// /// ## `override_content_disposition` /// /// Override the [`content-disposition`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition) header returned by storage services. /// /// ``` /// use std::time::Duration; /// /// use anyhow::Result; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op /// .presign_read_with("test.txt", Duration::from_secs(3600)) /// .override_content_disposition("attachment; filename=\"othertext.txt\"") /// .await?; /// Ok(()) /// } /// ``` /// /// ## `override_cache_control` /// /// Override the [`cache-control`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) header returned by storage services. /// /// ``` /// use std::time::Duration; /// /// use anyhow::Result; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op /// .presign_read_with("test.txt", Duration::from_secs(3600)) /// .override_cache_control("no-store") /// .await?; /// Ok(()) /// } /// ``` /// /// ## `override_content_type` /// /// Override the [`content-type`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type) header returned by storage services. /// /// ``` /// use std::time::Duration; /// /// use anyhow::Result; /// use futures::io; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op /// .presign_read_with("test.txt", Duration::from_secs(3600)) /// .override_content_type("text/plain") /// .await?; /// Ok(()) /// } /// ``` pub fn presign_read_with( &self, path: &str, expire: Duration, ) -> FuturePresignRead>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, (OpRead::default(), expire), |inner, path, (args, dur)| async move { let op = OpPresign::new(args, dur); let rp = inner.presign(&path, op).await?; Ok(rp.into_presigned_request()) }, ) } /// Presign an operation for write. /// /// # Notes /// /// ## Extra Options /// /// `presign_write` is a wrapper of [`Self::presign_write_with`] without any options. To use /// extra options like `content_type`, please use [`Self::presign_write_with`] instead. /// /// # Example /// /// ``` /// use std::time::Duration; /// /// use anyhow::Result; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op /// .presign_write("test.txt", Duration::from_secs(3600)) /// .await?; /// Ok(()) /// } /// ``` /// /// - `signed_req.method()`: `PUT` /// - `signed_req.uri()`: `https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=` /// - `signed_req.headers()`: `{ "host": "s3.amazonaws.com" }` /// /// We can upload file as this file via `curl` or other tools without credential: /// /// ```shell /// curl -X PUT "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=" -d "Hello, World!" /// ``` pub async fn presign_write(&self, path: &str, expire: Duration) -> Result { self.presign_write_with(path, expire).await } /// Presign an operation for write with extra options. /// /// # Options /// /// ## `content_type` /// /// Set the [`content-type`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type) header returned by storage services. /// /// ``` /// use std::time::Duration; /// /// use anyhow::Result; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op /// .presign_write_with("test", Duration::from_secs(3600)) /// .content_type("text/csv") /// .await?; /// let req = http::Request::builder() /// .method(signed_req.method()) /// .uri(signed_req.uri()) /// .body(())?; /// /// Ok(()) /// } /// ``` /// /// ## `content_disposition` /// /// Set the [`content-disposition`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition) header returned by storage services. /// /// ``` /// use std::time::Duration; /// /// use anyhow::Result; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op /// .presign_write_with("test", Duration::from_secs(3600)) /// .content_disposition("attachment; filename=\"cool.html\"") /// .await?; /// let req = http::Request::builder() /// .method(signed_req.method()) /// .uri(signed_req.uri()) /// .body(())?; /// /// Ok(()) /// } /// ``` /// /// ## `cache_control` /// /// Set the [`cache-control`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) header returned by storage services. /// /// ``` /// use std::time::Duration; /// /// use anyhow::Result; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let signed_req = op /// .presign_write_with("test", Duration::from_secs(3600)) /// .cache_control("no-store") /// .await?; /// let req = http::Request::builder() /// .method(signed_req.method()) /// .uri(signed_req.uri()) /// .body(())?; /// /// Ok(()) /// } /// ``` pub fn presign_write_with( &self, path: &str, expire: Duration, ) -> FuturePresignWrite>> { let path = normalize_path(path); OperatorFuture::new( self.inner().clone(), path, (OpWrite::default(), expire), |inner, path, (args, dur)| async move { let op = OpPresign::new(args, dur); let rp = inner.presign(&path, op).await?; Ok(rp.into_presigned_request()) }, ) } } opendal-0.52.0/src/types/operator/operator_functions.rs000064400000000000000000000326241046102023000214200ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Functions provides the functions generated by [`BlockingOperator`] //! //! By using functions, users can add more options for operation. use std::ops::RangeBounds; use crate::raw::*; use crate::*; /// OperatorFunction is the function generated by [`BlockingOperator`]. /// /// The function will consume all the input to generate a result. pub(crate) struct OperatorFunction { inner: Accessor, path: String, args: T, f: fn(Accessor, String, T) -> Result, } impl OperatorFunction { pub fn new( inner: Accessor, path: String, args: T, f: fn(Accessor, String, T) -> Result, ) -> Self { Self { inner, path, args, f, } } fn map_args(self, f: impl FnOnce(T) -> T) -> Self { Self { inner: self.inner, path: self.path, args: f(self.args), f: self.f, } } fn call(self) -> Result { (self.f)(self.inner, self.path, self.args) } } /// Function that generated by [`BlockingOperator::write_with`]. /// /// Users can add more options by public functions provided by this struct. pub struct FunctionWrite( /// The args for FunctionWrite is a bit special because we also /// need to move the bytes input this function. pub(crate) OperatorFunction<(OpWrite, OpWriter, Buffer), Metadata>, ); impl FunctionWrite { /// Set the append mode of op. /// /// If the append mode is set, the data will be appended to the end of the file. /// /// # Notes /// /// Service could return `Unsupported` if the underlying storage does not support append. pub fn append(mut self, v: bool) -> Self { self.0 = self .0 .map_args(|(args, options, bs)| (args.with_append(v), options, bs)); self } /// Set the chunk size of op. /// /// If chunk size is set, the data will be chunked by the underlying writer. /// /// ## NOTE /// /// Service could have their own limitation for chunk size. It's possible that chunk size /// is not equal to the given chunk size. /// /// For example: /// /// - AWS S3 requires the part size to be in [5MiB, 5GiB]. /// - GCS requires the part size to be aligned with 256 KiB. /// /// The services will alter the chunk size to meet their requirements. pub fn chunk(mut self, v: usize) -> Self { self.0 = self .0 .map_args(|(args, options, bs)| (args, options.with_chunk(v), bs)); self } /// Set the content type of option pub fn content_type(mut self, v: &str) -> Self { self.0 = self .0 .map_args(|(args, options, bs)| (args.with_content_type(v), options, bs)); self } /// Set the content disposition of option pub fn content_disposition(mut self, v: &str) -> Self { self.0 = self .0 .map_args(|(args, options, bs)| (args.with_content_disposition(v), options, bs)); self } /// Set the content type of option pub fn cache_control(mut self, v: &str) -> Self { self.0 = self .0 .map_args(|(args, options, bs)| (args.with_cache_control(v), options, bs)); self } /// Call the function to consume all the input and generate a /// result. pub fn call(self) -> Result { self.0.call() } } /// Function that generated by [`BlockingOperator::writer_with`]. /// /// Users can add more options by public functions provided by this struct. pub struct FunctionWriter( /// The args for FunctionWriter is a bit special because we also /// need to move the bytes input this function. pub(crate) OperatorFunction<(OpWrite, OpWriter), BlockingWriter>, ); impl FunctionWriter { /// Set the append mode of op. /// /// If the append mode is set, the data will be appended to the end of the file. /// /// # Notes /// /// Service could return `Unsupported` if the underlying storage does not support append. pub fn append(mut self, v: bool) -> Self { self.0 = self .0 .map_args(|(args, options)| (args.with_append(v), options)); self } /// Set the chunk size of op. /// /// If chunk size is set, the data will be chunked by the underlying writer. /// /// ## NOTE /// /// Service could have their own limitation for chunk size. It's possible that chunk size /// is not equal to the given chunk size. /// /// For example: /// /// - AWS S3 requires the part size to be in [5MiB, 5GiB]. /// - GCS requires the part size to be aligned with 256 KiB. /// /// The services will alter the chunk size to meet their requirements. pub fn chunk(mut self, v: usize) -> Self { self.0 = self .0 .map_args(|(args, options)| (args, options.with_chunk(v))); self } /// Set the chunk size of op. #[deprecated(note = "Please use `chunk` instead")] pub fn buffer(self, v: usize) -> Self { self.chunk(v) } /// Set the content type of option pub fn content_type(mut self, v: &str) -> Self { self.0 = self .0 .map_args(|(args, options)| (args.with_content_type(v), options)); self } /// Set the content disposition of option pub fn content_disposition(mut self, v: &str) -> Self { self.0 = self .0 .map_args(|(args, options)| (args.with_content_disposition(v), options)); self } /// Set the content type of option pub fn cache_control(mut self, v: &str) -> Self { self.0 = self .0 .map_args(|(args, options)| (args.with_cache_control(v), options)); self } /// Call the function to consume all the input and generate a /// result. pub fn call(self) -> Result { self.0.call() } } /// Function that generated by [`BlockingOperator::delete_with`]. /// /// Users can add more options by public functions provided by this struct. pub struct FunctionDelete(pub(crate) OperatorFunction); impl FunctionDelete { /// Set the version for this operation. pub fn version(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_version(v)); self } /// Call the function to consume all the input and generate a /// result. pub fn call(self) -> Result<()> { self.0.call() } } /// Function that generated by [`BlockingOperator::list_with`]. /// /// Users can add more options by public functions provided by this struct. pub struct FunctionList(pub(crate) OperatorFunction>); impl FunctionList { /// The limit passed to underlying service to specify the max results /// that could return per-request. /// /// Users could use this to control the memory usage of list operation. pub fn limit(mut self, v: usize) -> Self { self.0 = self.0.map_args(|args| args.with_limit(v)); self } /// The start_after passes to underlying service to specify the specified key /// to start listing from. pub fn start_after(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_start_after(v)); self } /// The recursive is used to control whether the list operation is recursive. /// /// - If `false`, list operation will only list the entries under the given path. /// - If `true`, list operation will list all entries that starts with given path. /// /// Default to `false`. pub fn recursive(mut self, v: bool) -> Self { self.0 = self.0.map_args(|args| args.with_recursive(v)); self } /// Call the function to consume all the input and generate a /// result. pub fn call(self) -> Result> { self.0.call() } } /// Function that generated by [`BlockingOperator::lister_with`]. /// /// Users can add more options by public functions provided by this struct. pub struct FunctionLister(pub(crate) OperatorFunction); impl FunctionLister { /// The limit passed to underlying service to specify the max results /// that could return per-request. /// /// Users could use this to control the memory usage of list operation. pub fn limit(mut self, v: usize) -> Self { self.0 = self.0.map_args(|args| args.with_limit(v)); self } /// The start_after passes to underlying service to specify the specified key /// to start listing from. pub fn start_after(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_start_after(v)); self } /// The recursive is used to control whether the list operation is recursive. /// /// - If `false`, list operation will only list the entries under the given path. /// - If `true`, list operation will list all entries that starts with given path. /// /// Default to `false`. pub fn recursive(mut self, v: bool) -> Self { self.0 = self.0.map_args(|args| args.with_recursive(v)); self } /// Call the function to consume all the input and generate a /// result. pub fn call(self) -> Result { self.0.call() } } /// Function that generated by [`BlockingOperator::read_with`]. /// /// Users can add more options by public functions provided by this struct. pub struct FunctionRead(pub(crate) OperatorFunction<(OpRead, BytesRange), Buffer>); impl FunctionRead { /// Set the range for this operation. pub fn range(mut self, range: impl RangeBounds) -> Self { self.0 = self.0.map_args(|(args, _)| (args, range.into())); self } /// Call the function to consume all the input and generate a /// result. pub fn call(self) -> Result { self.0.call() } } /// Function that generated by [`BlockingOperator::reader_with`]. /// /// Users can add more options by public functions provided by this struct. pub struct FunctionReader(pub(crate) OperatorFunction); impl FunctionReader { /// Sets the content-disposition header that should be send back by the remote read operation. pub fn override_content_disposition(mut self, content_disposition: &str) -> Self { self.0 = self .0 .map_args(|args| args.with_override_content_disposition(content_disposition)); self } /// Sets the cache-control header that should be send back by the remote read operation. pub fn override_cache_control(mut self, cache_control: &str) -> Self { self.0 = self .0 .map_args(|args| args.with_override_cache_control(cache_control)); self } /// Sets the content-type header that should be send back by the remote read operation. pub fn override_content_type(mut self, content_type: &str) -> Self { self.0 = self .0 .map_args(|args| args.with_override_content_type(content_type)); self } /// Set the If-Match for this operation. pub fn if_match(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_if_match(v)); self } /// Set the If-None-Match for this operation. pub fn if_none_match(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_if_none_match(v)); self } /// Set the version for this operation. pub fn version(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_version(v)); self } /// Call the function to consume all the input and generate a /// result. pub fn call(self) -> Result { self.0.call() } } /// Function that generated by [`BlockingOperator::stat_with`]. /// /// Users can add more options by public functions provided by this struct. pub struct FunctionStat(pub(crate) OperatorFunction); impl FunctionStat { /// Set the If-Match for this operation. pub fn if_match(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_if_match(v)); self } /// Set the If-None-Match for this operation. pub fn if_none_match(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_if_none_match(v)); self } /// Set the version for this operation. pub fn version(mut self, v: &str) -> Self { self.0 = self.0.map_args(|args| args.with_version(v)); self } /// Call the function to consume all the input and generate a /// result. pub fn call(self) -> Result { self.0.call() } } opendal-0.52.0/src/types/operator/operator_futures.rs000064400000000000000000001662011046102023000211040ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Futures provides the futures generated by [`Operator`] //! //! By using futures, users can add more options for operation. use chrono::{DateTime, Utc}; use futures::Future; use std::collections::HashMap; use std::future::IntoFuture; use std::ops::RangeBounds; use std::time::Duration; use crate::raw::*; use crate::*; /// OperatorFuture is the future generated by [`Operator`]. /// /// The future will consume all the input to generate a future. /// /// # NOTES /// /// This struct is by design to keep in crate. We don't want /// users to use this struct directly. pub struct OperatorFuture>> { /// The accessor to the underlying object storage acc: Accessor, /// The path of string path: String, /// The input args args: I, /// The function which will move all the args and return a static future f: fn(Accessor, String, I) -> F, } impl>> OperatorFuture { /// # NOTES /// /// This struct is by design to keep in crate. We don't want /// users to use this struct directly. pub(crate) fn new( inner: Accessor, path: String, args: I, f: fn(Accessor, String, I) -> F, ) -> Self { OperatorFuture { acc: inner, path, args, f, } } } impl>> OperatorFuture { /// Change the operation's args. fn map(mut self, f: impl FnOnce(I) -> I) -> Self { self.args = f(self.args); self } } impl IntoFuture for OperatorFuture where F: Future>, { type Output = Result; type IntoFuture = F; fn into_future(self) -> Self::IntoFuture { (self.f)(self.acc, self.path, self.args) } } /// Future that generated by [`Operator::stat_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FutureStat = OperatorFuture; impl>> FutureStat { /// Set the If-Match for this operation. pub fn if_match(self, v: &str) -> Self { self.map(|args| args.with_if_match(v)) } /// Set the If-None-Match for this operation. pub fn if_none_match(self, v: &str) -> Self { self.map(|args| args.with_if_none_match(v)) } /// Set the If-Modified-Since for this operation. pub fn if_modified_since(self, v: DateTime) -> Self { self.map(|args| args.with_if_modified_since(v)) } /// Set the If-Unmodified-Since for this operation. pub fn if_unmodified_since(self, v: DateTime) -> Self { self.map(|args| args.with_if_unmodified_since(v)) } /// Set the version for this operation. pub fn version(self, v: &str) -> Self { self.map(|args| args.with_version(v)) } } /// Future that generated by [`Operator::presign_stat_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FuturePresignStat = OperatorFuture<(OpStat, Duration), PresignedRequest, F>; impl>> FuturePresignStat { /// Sets the content-disposition header that should be sent back by the remote read operation. pub fn override_content_disposition(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_override_content_disposition(v), dur)) } /// Sets the cache-control header that should be sent back by the remote read operation. pub fn override_cache_control(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_override_cache_control(v), dur)) } /// Sets the content-type header that should be sent back by the remote read operation. pub fn override_content_type(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_override_content_type(v), dur)) } /// Set the If-Match of the option pub fn if_match(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_if_match(v), dur)) } /// Set the If-None-Match of the option pub fn if_none_match(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_if_none_match(v), dur)) } } /// Future that generated by [`Operator::presign_read_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FuturePresignRead = OperatorFuture<(OpRead, Duration), PresignedRequest, F>; impl>> FuturePresignRead { /// Sets the content-disposition header that should be sent back by the remote read operation. pub fn override_content_disposition(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_override_content_disposition(v), dur)) } /// Sets the cache-control header that should be sent back by the remote read operation. pub fn override_cache_control(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_override_cache_control(v), dur)) } /// Sets the content-type header that should be sent back by the remote read operation. pub fn override_content_type(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_override_content_type(v), dur)) } /// Set the If-Match of the option pub fn if_match(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_if_match(v), dur)) } /// Set the If-None-Match of the option pub fn if_none_match(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_if_none_match(v), dur)) } } /// Future that generated by [`Operator::presign_write_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FuturePresignWrite = OperatorFuture<(OpWrite, Duration), PresignedRequest, F>; impl>> FuturePresignWrite { /// Set the content type of option pub fn content_type(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_content_type(v), dur)) } /// Set the content disposition of option pub fn content_disposition(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_content_disposition(v), dur)) } /// Set the content encoding of the operation pub fn content_encoding(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_content_encoding(v), dur)) } /// Set the content type of option pub fn cache_control(self, v: &str) -> Self { self.map(|(args, dur)| (args.with_cache_control(v), dur)) } } /// Future that generated by [`Operator::read_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FutureRead = OperatorFuture<(OpRead, OpReader), Buffer, F>; impl>> FutureRead { /// Set the executor for this operation. pub fn executor(self, executor: Executor) -> Self { self.map(|(args, op_reader)| (args.with_executor(executor), op_reader)) } /// Set `range` for this `read` request. /// /// If we have a file with size `n`. /// /// - `..` means read bytes in range `[0, n)` of file. /// - `0..1024` and `..1024` means read bytes in range `[0, 1024)` of file /// - `1024..` means read bytes in range `[1024, n)` of file /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::TryStreamExt; /// # async fn test(op: Operator) -> Result<()> { /// let bs = op.read_with("path/to/file").range(0..1024).await?; /// # Ok(()) /// # } /// ``` pub fn range(self, range: impl RangeBounds) -> Self { self.map(|(args, op_reader)| (args.with_range(range.into()), op_reader)) } /// Set `concurrent` for the reader. /// /// OpenDAL by default to write file without concurrent. This is not efficient for cases when users /// read large chunks of data. By setting `concurrent`, opendal will read files concurrently /// on support storage services. /// /// By setting `concurrent`, opendal will fetch chunks concurrently with /// the given chunk size. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use opendal::Scheme; /// # async fn test(op: Operator) -> Result<()> { /// let r = op.read_with("path/to/file").concurrent(8).await?; /// # Ok(()) /// # } /// ``` pub fn concurrent(self, concurrent: usize) -> Self { self.map(|(args, op_reader)| (args, op_reader.with_concurrent(concurrent))) } /// OpenDAL will use services' preferred chunk size by default. Users can set chunk based on their own needs. /// /// This following example will make opendal read data in 4MiB chunks: /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use opendal::Scheme; /// # async fn test(op: Operator) -> Result<()> { /// let r = op.read_with("path/to/file").chunk(4 * 1024 * 1024).await?; /// # Ok(()) /// # } /// ``` pub fn chunk(self, chunk_size: usize) -> Self { self.map(|(args, op_reader)| (args, op_reader.with_chunk(chunk_size))) } /// Set `version` for this `read` request. /// /// This feature can be used to retrieve the data of a specified version of the given path. /// /// If the version doesn't exist, an error with kind [`ErrorKind::NotFound`] will be returned. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// /// # async fn test(op: Operator, version: &str) -> Result<()> { /// let mut bs = op.read_with("path/to/file").version(version).await?; /// # Ok(()) /// # } /// ``` pub fn version(self, v: &str) -> Self { self.map(|(args, op_reader)| (args.with_version(v), op_reader)) } /// Set `if_match` for this `read` request. /// /// This feature can be used to check if the file's `ETag` matches the given `ETag`. /// /// If file exists and it's etag doesn't match, an error with kind [`ErrorKind::ConditionNotMatch`] /// will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// # async fn test(op: Operator, etag: &str) -> Result<()> { /// let mut metadata = op.read_with("path/to/file").if_match(etag).await?; /// # Ok(()) /// # } /// ``` pub fn if_match(self, v: &str) -> Self { self.map(|(args, op_reader)| (args.with_if_match(v), op_reader)) } /// Set `if_none_match` for this `read` request. /// /// This feature can be used to check if the file's `ETag` doesn't match the given `ETag`. /// /// If file exists and it's etag match, an error with kind [`ErrorKind::ConditionNotMatch`] /// will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// # async fn test(op: Operator, etag: &str) -> Result<()> { /// let mut metadata = op.read_with("path/to/file").if_none_match(etag).await?; /// # Ok(()) /// # } /// ``` pub fn if_none_match(self, v: &str) -> Self { self.map(|(args, op_reader)| (args.with_if_none_match(v), op_reader)) } /// ## `if_modified_since` /// /// Set `if_modified_since` for this `read` request. /// /// This feature can be used to check if the file has been modified since the given timestamp. /// /// If file exists and it hasn't been modified since the specified time, an error with kind /// [`ErrorKind::ConditionNotMatch`] will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// use chrono::DateTime; /// use chrono::Utc; /// # async fn test(op: Operator, time: DateTime) -> Result<()> { /// let mut metadata = op.read_with("path/to/file").if_modified_since(time).await?; /// # Ok(()) /// # } /// ``` pub fn if_modified_since(self, v: DateTime) -> Self { self.map(|(args, op_reader)| (args.with_if_modified_since(v), op_reader)) } /// Set `if_unmodified_since` for this `read` request. /// /// This feature can be used to check if the file hasn't been modified since the given timestamp. /// /// If file exists and it has been modified since the specified time, an error with kind /// [`ErrorKind::ConditionNotMatch`] will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// use chrono::DateTime; /// use chrono::Utc; /// # async fn test(op: Operator, time: DateTime) -> Result<()> { /// let mut metadata = op.read_with("path/to/file").if_unmodified_since(time).await?; /// # Ok(()) /// # } /// ``` pub fn if_unmodified_since(self, v: DateTime) -> Self { self.map(|(args, op_reader)| (args.with_if_unmodified_since(v), op_reader)) } } /// Future that generated by [`Operator::read_with`] or [`Operator::reader_with`]. /// /// Users can add more options by public functions provided by this struct. /// /// # Notes /// /// `(OpRead, ())` is a trick to make sure `FutureReader` is different from `FutureRead` pub type FutureReader = OperatorFuture<(OpRead, OpReader), Reader, F>; impl>> FutureReader { /// Set `version` for this `reader`. /// /// This feature can be used to retrieve the data of a specified version of the given path. /// /// If the version doesn't exist, an error with kind [`ErrorKind::NotFound`] will be returned. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// /// # async fn test(op: Operator, version: &str) -> Result<()> { /// let mut r = op.reader_with("path/to/file").version(version).await?; /// # Ok(()) /// # } /// ``` pub fn version(self, v: &str) -> Self { self.map(|(op_read, op_reader)| (op_read.with_version(v), op_reader)) } /// Set `concurrent` for the reader. /// /// OpenDAL by default to write file without concurrent. This is not efficient for cases when users /// read large chunks of data. By setting `concurrent`, opendal will reading files concurrently /// on support storage services. /// /// By setting `concurrent`, opendal will fetch chunks concurrently with /// the give chunk size. /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use opendal::Scheme; /// # async fn test(op: Operator) -> Result<()> { /// let r = op.reader_with("path/to/file").concurrent(8).await?; /// # Ok(()) /// # } /// ``` pub fn concurrent(self, concurrent: usize) -> Self { self.map(|(op_read, op_reader)| (op_read, op_reader.with_concurrent(concurrent))) } /// OpenDAL will use services' preferred chunk size by default. Users can set chunk based on their own needs. /// /// This following example will make opendal read data in 4MiB chunks: /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use opendal::Scheme; /// # async fn test(op: Operator) -> Result<()> { /// let r = op /// .reader_with("path/to/file") /// .chunk(4 * 1024 * 1024) /// .await?; /// # Ok(()) /// # } /// ``` pub fn chunk(self, chunk_size: usize) -> Self { self.map(|(op_read, op_reader)| (op_read, op_reader.with_chunk(chunk_size))) } /// Controls the optimization strategy for range reads in [`Reader::fetch`]. /// /// When performing range reads, if the gap between two requested ranges is smaller than /// the configured `gap` size, OpenDAL will merge these ranges into a single read request /// and discard the unrequested data in between. This helps reduce the number of API calls /// to remote storage services. /// /// This optimization is particularly useful when performing multiple small range reads /// that are close to each other, as it reduces the overhead of multiple network requests /// at the cost of transferring some additional data. /// /// In this example, if two requested ranges are separated by less than 1MiB, /// they will be merged into a single read request: /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use opendal::Scheme; /// # async fn test(op: Operator) -> Result<()> { /// let r = op /// .reader_with("path/to/file") /// .chunk(4 * 1024 * 1024) /// .gap(1024 * 1024) // 1MiB gap /// .await?; /// # Ok(()) /// # } /// ``` pub fn gap(self, gap_size: usize) -> Self { self.map(|(op_read, op_reader)| (op_read, op_reader.with_gap(gap_size))) } /// Set `if-match` for this `read` request. /// /// This feature can be used to check if the file's `ETag` matches the given `ETag`. /// /// If file exists and it's etag doesn't match, an error with kind [`ErrorKind::ConditionNotMatch`] /// will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// # async fn test(op: Operator, etag: &str) -> Result<()> { /// let mut r = op.reader_with("path/to/file").if_match(etag).await?; /// # Ok(()) /// # } /// ``` pub fn if_match(self, etag: &str) -> Self { self.map(|(op_read, op_reader)| (op_read.with_if_match(etag), op_reader)) } /// Set `if-none-match` for this `read` request. /// /// This feature can be used to check if the file's `ETag` doesn't match the given `ETag`. /// /// If file exists and it's etag match, an error with kind [`ErrorKind::ConditionNotMatch`] /// will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// # async fn test(op: Operator, etag: &str) -> Result<()> { /// let mut r = op.reader_with("path/to/file").if_none_match(etag).await?; /// # Ok(()) /// # } /// ``` pub fn if_none_match(self, etag: &str) -> Self { self.map(|(op_read, op_reader)| (op_read.with_if_none_match(etag), op_reader)) } /// Set `if-modified-since` for this `read` request. /// /// This feature can be used to check if the file has been modified since the given timestamp. /// /// If file exists and it hasn't been modified since the specified time, an error with kind /// [`ErrorKind::ConditionNotMatch`] will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// use chrono::DateTime; /// use chrono::Utc; /// # async fn test(op: Operator, time: DateTime) -> Result<()> { /// let mut r = op.reader_with("path/to/file").if_modified_since(time).await?; /// # Ok(()) /// # } /// ``` pub fn if_modified_since(self, v: DateTime) -> Self { self.map(|(op_read, op_reader)| (op_read.with_if_modified_since(v), op_reader)) } /// Set `if-unmodified-since` for this `read` request. /// /// This feature can be used to check if the file hasn't been modified since the given timestamp. /// /// If file exists and it has been modified since the specified time, an error with kind /// [`ErrorKind::ConditionNotMatch`] will be returned. /// /// ``` /// # use opendal::Result; /// use opendal::Operator; /// use chrono::DateTime; /// use chrono::Utc; /// # async fn test(op: Operator, time: DateTime) -> Result<()> { /// let mut r = op.reader_with("path/to/file").if_unmodified_since(time).await?; /// # Ok(()) /// # } /// ``` pub fn if_unmodified_since(self, v: DateTime) -> Self { self.map(|(op_read, op_reader)| (op_read.with_if_unmodified_since(v), op_reader)) } } /// Future that generated by [`Operator::write_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FutureWrite = OperatorFuture<(OpWrite, OpWriter, Buffer), Metadata, F>; impl>> FutureWrite { /// Set the executor for this operation. pub fn executor(self, executor: Executor) -> Self { self.map(|(args, options, bs)| (args.with_executor(executor), options, bs)) } /// Sets append mode for this write request. /// /// ### Capability /// /// Check [`Capability::write_can_append`] before using this feature. /// /// ### Behavior /// /// - By default, write operations overwrite existing files /// - When append is set to true: /// - New data will be appended to the end of existing file /// - If file doesn't exist, it will be created /// - If not supported, will return an error /// /// This operation allows adding data to existing files instead of overwriting them. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let _ = op.write_with("path/to/file", vec![0; 4096]).append(true).await?; /// # Ok(()) /// # } /// ``` pub fn append(self, v: bool) -> Self { self.map(|(args, options, bs)| (args.with_append(v), options, bs)) } /// Sets chunk size for buffered writes. /// /// ### Capability /// /// Check [`Capability::write_multi_min_size`] and [`Capability::write_multi_max_size`] for size limits. /// /// ### Behavior /// /// - By default, OpenDAL sets optimal chunk size based on service capabilities /// - When chunk size is set: /// - Data will be buffered until reaching chunk size /// - One API call will be made per chunk /// - Last chunk may be smaller than chunk size /// - Important considerations: /// - Some services require minimum chunk sizes (e.g. S3's EntityTooSmall error) /// - Smaller chunks increase API calls and costs /// - Larger chunks increase memory usage, but improve performance and reduce costs /// /// ### Performance Impact /// /// Setting appropriate chunk size can: /// - Reduce number of API calls /// - Improve overall throughput /// - Lower operation costs /// - Better utilize network bandwidth /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// // Set 8MiB chunk size - data will be sent in one API call at close /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .chunk(8 * 1024 * 1024) /// .await?; /// # Ok(()) /// # } /// ``` pub fn chunk(self, v: usize) -> Self { self.map(|(args, options, bs)| (args, options.with_chunk(v), bs)) } /// Sets concurrent write operations for this writer. /// /// ## Behavior /// /// - By default, OpenDAL writes files sequentially /// - When concurrent is set: /// - Multiple write operations can execute in parallel /// - Write operations return immediately without waiting if tasks space are available /// - Close operation ensures all writes complete in order /// - Memory usage increases with concurrency level /// - If not supported, falls back to sequential writes /// /// This feature significantly improves performance when: /// - Writing large files /// - Network latency is high /// - Storage service supports concurrent uploads like multipart uploads /// /// ## Performance Impact /// /// Setting appropriate concurrency can: /// - Increase write throughput /// - Reduce total write time /// - Better utilize available bandwidth /// - Trade memory for performance /// /// ## Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// // Enable concurrent writes with 8 parallel operations at 128B chunk. /// let _ = op.write_with("path/to/file", vec![0; 4096]).chunk(128).concurrent(8).await?; /// # Ok(()) /// # } /// ``` pub fn concurrent(self, v: usize) -> Self { self.map(|(args, options, bs)| (args.with_concurrent(v), options, bs)) } /// Sets Cache-Control header for this write operation. /// /// ### Capability /// /// Check [`Capability::write_with_cache_control`] before using this feature. /// /// ### Behavior /// /// - If supported, sets Cache-Control as system metadata on the target file /// - The value should follow HTTP Cache-Control header format /// - If not supported, the value will be ignored /// /// This operation allows controlling caching behavior for the written content. /// /// ### Use Cases /// /// - Setting browser cache duration /// - Configuring CDN behavior /// - Optimizing content delivery /// - Managing cache invalidation /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// // Cache content for 7 days (604800 seconds) /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .cache_control("max-age=604800") /// .await?; /// # Ok(()) /// # } /// ``` /// /// ### References /// /// - [MDN Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) /// - [RFC 7234 Section 5.2](https://tools.ietf.org/html/rfc7234#section-5.2) pub fn cache_control(self, v: &str) -> Self { self.map(|(args, options, bs)| (args.with_cache_control(v), options, bs)) } /// Sets `Content-Type` header for this write operation. /// /// ## Capability /// /// Check [`Capability::write_with_content_type`] before using this feature. /// /// ### Behavior /// /// - If supported, sets Content-Type as system metadata on the target file /// - The value should follow MIME type format (e.g. "text/plain", "image/jpeg") /// - If not supported, the value will be ignored /// /// This operation allows specifying the media type of the content being written. /// /// ## Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// // Set content type for plain text file /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .content_type("text/plain") /// .await?; /// # Ok(()) /// # } /// ``` pub fn content_type(self, v: &str) -> Self { self.map(|(args, options, bs)| (args.with_content_type(v), options, bs)) } /// ## `content_disposition` /// /// Sets Content-Disposition header for this write request. /// /// ### Capability /// /// Check [`Capability::write_with_content_disposition`] before using this feature. /// /// ### Behavior /// /// - If supported, sets Content-Disposition as system metadata on the target file /// - The value should follow HTTP Content-Disposition header format /// - Common values include: /// - `inline` - Content displayed within browser /// - `attachment` - Content downloaded as file /// - `attachment; filename="example.jpg"` - Downloaded with specified filename /// - If not supported, the value will be ignored /// /// This operation allows controlling how the content should be displayed or downloaded. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .content_disposition("attachment; filename=\"filename.jpg\"") /// .await?; /// # Ok(()) /// # } /// ``` pub fn content_disposition(self, v: &str) -> Self { self.map(|(args, options, bs)| (args.with_content_disposition(v), options, bs)) } /// Sets Content-Encoding header for this write request. /// /// ### Capability /// /// Check [`Capability::write_with_content_encoding`] before using this feature. /// /// ### Behavior /// /// - If supported, sets Content-Encoding as system metadata on the target file /// - The value should follow HTTP Content-Encoding header format /// - Common values include: /// - `gzip` - Content encoded using gzip compression /// - `deflate` - Content encoded using deflate compression /// - `br` - Content encoded using Brotli compression /// - `identity` - No encoding applied (default value) /// - If not supported, the value will be ignored /// /// This operation allows specifying the encoding applied to the content being written. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .content_encoding("gzip") /// .await?; /// # Ok(()) /// # } /// ``` pub fn content_encoding(self, v: &str) -> Self { self.map(|(args, options, bs)| (args.with_content_encoding(v), options, bs)) } /// Sets If-Match header for this write request. /// /// ### Capability /// /// Check [`Capability::write_with_if_match`] before using this feature. /// /// ### Behavior /// /// - If supported, the write operation will only succeed if the target's ETag matches the specified value /// - The value should be a valid ETag string /// - Common values include: /// - A specific ETag value like `"686897696a7c876b7e"` /// - `*` - Matches any existing resource /// - If not supported, the value will be ignored /// /// This operation provides conditional write functionality based on ETag matching, /// helping prevent unintended overwrites in concurrent scenarios. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .if_match("\"686897696a7c876b7e\"") /// .await?; /// # Ok(()) /// # } /// ``` pub fn if_match(self, s: &str) -> Self { self.map(|(args, options, bs)| (args.with_if_match(s), options, bs)) } /// Sets If-None-Match header for this write request. /// /// Note: Certain services, like `s3`, support `if_not_exists` but not `if_none_match`. /// Use `if_not_exists` if you only want to check whether a file exists. /// /// ### Capability /// /// Check [`Capability::write_with_if_none_match`] before using this feature. /// /// ### Behavior /// /// - If supported, the write operation will only succeed if the target's ETag does not match the specified value /// - The value should be a valid ETag string /// - Common values include: /// - A specific ETag value like `"686897696a7c876b7e"` /// - `*` - Matches if the resource does not exist /// - If not supported, the value will be ignored /// /// This operation provides conditional write functionality based on ETag non-matching, /// useful for preventing overwriting existing resources or ensuring unique writes. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .if_none_match("\"686897696a7c876b7e\"") /// .await?; /// # Ok(()) /// # } /// ``` pub fn if_none_match(self, s: &str) -> Self { self.map(|(args, options, bs)| (args.with_if_none_match(s), options, bs)) } /// Sets the condition that write operation will succeed only if target does not exist. /// /// ### Capability /// /// Check [`Capability::write_with_if_not_exists`] before using this feature. /// /// ### Behavior /// /// - If supported, the write operation will only succeed if the target path does not exist /// - Will return error if target already exists /// - If not supported, the value will be ignored /// /// This operation provides a way to ensure write operations only create new resources /// without overwriting existing ones, useful for implementing "create if not exists" logic. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .if_not_exists(true) /// .await?; /// # Ok(()) /// # } /// ``` pub fn if_not_exists(self, b: bool) -> Self { self.map(|(args, options, bs)| (args.with_if_not_exists(b), options, bs)) } /// Sets user metadata for this write request. /// /// ### Capability /// /// Check [`Capability::write_with_user_metadata`] before using this feature. /// /// ### Behavior /// /// - If supported, the user metadata will be attached to the object during write /// - Accepts key-value pairs where both key and value are strings /// - Keys are case-insensitive in most services /// - Services may have limitations for user metadata, for example: /// - Key length is typically limited (e.g., 1024 bytes) /// - Value length is typically limited (e.g., 4096 bytes) /// - Total metadata size might be limited /// - Some characters might be forbidden in keys /// - If not supported, the metadata will be ignored /// /// User metadata provides a way to attach custom metadata to objects during write operations. /// This metadata can be retrieved later when reading the object. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let _ = op /// .write_with("path/to/file", vec![0; 4096]) /// .user_metadata([ /// ("language".to_string(), "rust".to_string()), /// ("author".to_string(), "OpenDAL".to_string()), /// ]) /// .await?; /// # Ok(()) /// # } /// ``` pub fn user_metadata(self, data: impl IntoIterator) -> Self { self.map(|(args, options, bs)| { ( args.with_user_metadata(HashMap::from_iter(data)), options, bs, ) }) } } /// Future that generated by [`Operator::writer_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FutureWriter = OperatorFuture<(OpWrite, OpWriter), Writer, F>; impl>> FutureWriter { /// Set the executor for this operation. pub fn executor(self, executor: Executor) -> Self { self.map(|(args, options)| (args.with_executor(executor), options)) } /// Sets append mode for this write request. /// /// ### Capability /// /// Check [`Capability::write_can_append`] before using this feature. /// /// ### Behavior /// /// - By default, write operations overwrite existing files /// - When append is set to true: /// - New data will be appended to the end of existing file /// - If file doesn't exist, it will be created /// - If not supported, will return an error /// /// This operation allows adding data to existing files instead of overwriting them. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op.writer_with("path/to/file").append(true).await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn append(self, v: bool) -> Self { self.map(|(args, options)| (args.with_append(v), options)) } /// Sets chunk size for buffered writes. /// /// ### Capability /// /// Check [`Capability::write_multi_min_size`] and [`Capability::write_multi_max_size`] for size limits. /// /// ### Behavior /// /// - By default, OpenDAL sets optimal chunk size based on service capabilities /// - When chunk size is set: /// - Data will be buffered until reaching chunk size /// - One API call will be made per chunk /// - Last chunk may be smaller than chunk size /// - Important considerations: /// - Some services require minimum chunk sizes (e.g. S3's EntityTooSmall error) /// - Smaller chunks increase API calls and costs /// - Larger chunks increase memory usage, but improve performance and reduce costs /// /// ### Performance Impact /// /// Setting appropriate chunk size can: /// - Reduce number of API calls /// - Improve overall throughput /// - Lower operation costs /// - Better utilize network bandwidth /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// // Set 8MiB chunk size - data will be sent in one API call at close /// let mut w = op /// .writer_with("path/to/file") /// .chunk(8 * 1024 * 1024) /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn chunk(self, v: usize) -> Self { self.map(|(args, options)| (args, options.with_chunk(v))) } /// Sets concurrent write operations for this writer. /// /// ## Behavior /// /// - By default, OpenDAL writes files sequentially /// - When concurrent is set: /// - Multiple write operations can execute in parallel /// - Write operations return immediately without waiting if tasks space are available /// - Close operation ensures all writes complete in order /// - Memory usage increases with concurrency level /// - If not supported, falls back to sequential writes /// /// This feature significantly improves performance when: /// - Writing large files /// - Network latency is high /// - Storage service supports concurrent uploads like multipart uploads /// /// ## Performance Impact /// /// Setting appropriate concurrency can: /// - Increase write throughput /// - Reduce total write time /// - Better utilize available bandwidth /// - Trade memory for performance /// /// ## Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// // Enable concurrent writes with 8 parallel operations /// let mut w = op.writer_with("path/to/file").concurrent(8).await?; /// /// // First write starts immediately /// w.write(vec![0; 4096]).await?; /// /// // Second write runs concurrently with first /// w.write(vec![1; 4096]).await?; /// /// // Ensures all writes complete successfully and in order /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn concurrent(self, v: usize) -> Self { self.map(|(args, options)| (args.with_concurrent(v), options)) } /// Sets Cache-Control header for this write operation. /// /// ### Capability /// /// Check [`Capability::write_with_cache_control`] before using this feature. /// /// ### Behavior /// /// - If supported, sets Cache-Control as system metadata on the target file /// - The value should follow HTTP Cache-Control header format /// - If not supported, the value will be ignored /// /// This operation allows controlling caching behavior for the written content. /// /// ### Use Cases /// /// - Setting browser cache duration /// - Configuring CDN behavior /// - Optimizing content delivery /// - Managing cache invalidation /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// // Cache content for 7 days (604800 seconds) /// let mut w = op /// .writer_with("path/to/file") /// .cache_control("max-age=604800") /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` /// /// ### References /// /// - [MDN Cache-Control](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control) /// - [RFC 7234 Section 5.2](https://tools.ietf.org/html/rfc7234#section-5.2) pub fn cache_control(self, v: &str) -> Self { self.map(|(args, options)| (args.with_cache_control(v), options)) } /// Sets `Content-Type` header for this write operation. /// /// ## Capability /// /// Check [`Capability::write_with_content_type`] before using this feature. /// /// ### Behavior /// /// - If supported, sets Content-Type as system metadata on the target file /// - The value should follow MIME type format (e.g. "text/plain", "image/jpeg") /// - If not supported, the value will be ignored /// /// This operation allows specifying the media type of the content being written. /// /// ## Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// // Set content type for plain text file /// let mut w = op /// .writer_with("path/to/file") /// .content_type("text/plain") /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn content_type(self, v: &str) -> Self { self.map(|(args, options)| (args.with_content_type(v), options)) } /// ## `content_disposition` /// /// Sets Content-Disposition header for this write request. /// /// ### Capability /// /// Check [`Capability::write_with_content_disposition`] before using this feature. /// /// ### Behavior /// /// - If supported, sets Content-Disposition as system metadata on the target file /// - The value should follow HTTP Content-Disposition header format /// - Common values include: /// - `inline` - Content displayed within browser /// - `attachment` - Content downloaded as file /// - `attachment; filename="example.jpg"` - Downloaded with specified filename /// - If not supported, the value will be ignored /// /// This operation allows controlling how the content should be displayed or downloaded. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op /// .writer_with("path/to/file") /// .content_disposition("attachment; filename=\"filename.jpg\"") /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn content_disposition(self, v: &str) -> Self { self.map(|(args, options)| (args.with_content_disposition(v), options)) } /// Sets Content-Encoding header for this write request. /// /// ### Capability /// /// Check [`Capability::write_with_content_encoding`] before using this feature. /// /// ### Behavior /// /// - If supported, sets Content-Encoding as system metadata on the target file /// - The value should follow HTTP Content-Encoding header format /// - Common values include: /// - `gzip` - Content encoded using gzip compression /// - `deflate` - Content encoded using deflate compression /// - `br` - Content encoded using Brotli compression /// - `identity` - No encoding applied (default value) /// - If not supported, the value will be ignored /// /// This operation allows specifying the encoding applied to the content being written. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op /// .writer_with("path/to/file") /// .content_encoding("gzip") /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn content_encoding(self, v: &str) -> Self { self.map(|(args, options)| (args.with_content_encoding(v), options)) } /// Sets If-Match header for this write request. /// /// ### Capability /// /// Check [`Capability::write_with_if_match`] before using this feature. /// /// ### Behavior /// /// - If supported, the write operation will only succeed if the target's ETag matches the specified value /// - The value should be a valid ETag string /// - Common values include: /// - A specific ETag value like `"686897696a7c876b7e"` /// - `*` - Matches any existing resource /// - If not supported, the value will be ignored /// /// This operation provides conditional write functionality based on ETag matching, /// helping prevent unintended overwrites in concurrent scenarios. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op /// .writer_with("path/to/file") /// .if_match("\"686897696a7c876b7e\"") /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn if_match(self, s: &str) -> Self { self.map(|(args, options)| (args.with_if_match(s), options)) } /// Sets If-None-Match header for this write request. /// /// Note: Certain services, like `s3`, support `if_not_exists` but not `if_none_match`. /// Use `if_not_exists` if you only want to check whether a file exists. /// /// ### Capability /// /// Check [`Capability::write_with_if_none_match`] before using this feature. /// /// ### Behavior /// /// - If supported, the write operation will only succeed if the target's ETag does not match the specified value /// - The value should be a valid ETag string /// - Common values include: /// - A specific ETag value like `"686897696a7c876b7e"` /// - `*` - Matches if the resource does not exist /// - If not supported, the value will be ignored /// /// This operation provides conditional write functionality based on ETag non-matching, /// useful for preventing overwriting existing resources or ensuring unique writes. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op /// .writer_with("path/to/file") /// .if_none_match("\"686897696a7c876b7e\"") /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn if_none_match(self, s: &str) -> Self { self.map(|(args, options)| (args.with_if_none_match(s), options)) } /// Sets the condition that write operation will succeed only if target does not exist. /// /// ### Capability /// /// Check [`Capability::write_with_if_not_exists`] before using this feature. /// /// ### Behavior /// /// - If supported, the write operation will only succeed if the target path does not exist /// - Will return error if target already exists /// - If not supported, the value will be ignored /// /// This operation provides a way to ensure write operations only create new resources /// without overwriting existing ones, useful for implementing "create if not exists" logic. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op /// .writer_with("path/to/file") /// .if_not_exists(true) /// .await?; /// w.write(vec![0; 4096]).await?; /// w.write(vec![1; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn if_not_exists(self, b: bool) -> Self { self.map(|(args, options)| (args.with_if_not_exists(b), options)) } /// Sets user metadata for this write request. /// /// ### Capability /// /// Check [`Capability::write_with_user_metadata`] before using this feature. /// /// ### Behavior /// /// - If supported, the user metadata will be attached to the object during write /// - Accepts key-value pairs where both key and value are strings /// - Keys are case-insensitive in most services /// - Services may have limitations for user metadata, for example: /// - Key length is typically limited (e.g., 1024 bytes) /// - Value length is typically limited (e.g., 4096 bytes) /// - Total metadata size might be limited /// - Some characters might be forbidden in keys /// - If not supported, the metadata will be ignored /// /// User metadata provides a way to attach custom metadata to objects during write operations. /// This metadata can be retrieved later when reading the object. /// /// ### Example /// /// ``` /// # use opendal::Result; /// # use opendal::Operator; /// # use futures::StreamExt; /// # use futures::SinkExt; /// use bytes::Bytes; /// /// # async fn test(op: Operator) -> Result<()> { /// let mut w = op /// .writer_with("path/to/file") /// .user_metadata([ /// ("content-type".to_string(), "text/plain".to_string()), /// ("author".to_string(), "OpenDAL".to_string()), /// ]) /// .await?; /// w.write(vec![0; 4096]).await?; /// w.close().await?; /// # Ok(()) /// # } /// ``` pub fn user_metadata(self, data: impl IntoIterator) -> Self { self.map(|(args, options)| (args.with_user_metadata(HashMap::from_iter(data)), options)) } } /// Future that generated by [`Operator::delete_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FutureDelete = OperatorFuture; impl>> FutureDelete { /// Change the version of this delete operation. pub fn version(self, v: &str) -> Self { self.map(|args| args.with_version(v)) } } /// Future that generated by [`Operator::deleter_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FutureDeleter = OperatorFuture; /// Future that generated by [`Operator::list_with`] or [`Operator::lister_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FutureList = OperatorFuture, F>; impl>>> FutureList { /// The limit passed to underlying service to specify the max results /// that could return per-request. /// /// Users could use this to control the memory usage of list operation. pub fn limit(self, v: usize) -> Self { self.map(|args| args.with_limit(v)) } /// The start_after passes to underlying service to specify the specified key /// to start listing from. pub fn start_after(self, v: &str) -> Self { self.map(|args| args.with_start_after(v)) } /// The recursive is used to control whether the list operation is recursive. /// /// - If `false`, list operation will only list the entries under the given path. /// - If `true`, list operation will list all entries that starts with given path. /// /// Default to `false`. pub fn recursive(self, v: bool) -> Self { self.map(|args| args.with_recursive(v)) } /// The version is used to control whether the object versions should be returned. /// /// - If `false`, list operation will not return with object versions /// - If `true`, list operation will return with object versions if object versioning is supported /// by the underlying service /// /// Default to `false` #[deprecated(since = "0.51.1", note = "use versions instead")] pub fn version(self, v: bool) -> Self { self.map(|args| args.with_versions(v)) } /// Controls whether the `list` operation should return file versions. /// /// This function allows you to specify if the `list` operation, when executed, should include /// information about different versions of files, if versioning is supported and enabled. /// /// If `true`, subsequent `list` operations will include version information for each file. /// If `false`, version information will be omitted from the `list` results. /// /// Default to `false` pub fn versions(self, v: bool) -> Self { self.map(|args| args.with_versions(v)) } /// Controls whether the `list` operation should include deleted files (or versions). /// /// This function allows you to specify if the `list` operation, when executed, should include /// entries for files or versions that have been marked as deleted. This is particularly relevant /// in object storage systems that support soft deletion or versioning. /// /// If `true`, subsequent `list` operations will include deleted files or versions. /// If `false`, deleted files or versions will be excluded from the `list` results. pub fn deleted(self, v: bool) -> Self { self.map(|args| args.with_deleted(v)) } } /// Future that generated by [`Operator::list_with`] or [`Operator::lister_with`]. /// /// Users can add more options by public functions provided by this struct. pub type FutureLister = OperatorFuture; impl>> FutureLister { /// The limit passed to underlying service to specify the max results /// that could return per-request. /// /// Users could use this to control the memory usage of list operation. pub fn limit(self, v: usize) -> Self { self.map(|args| args.with_limit(v)) } /// The start_after passes to underlying service to specify the specified key /// to start listing from. pub fn start_after(self, v: &str) -> Self { self.map(|args| args.with_start_after(v)) } /// The recursive is used to control whether the list operation is recursive. /// /// - If `false`, list operation will only list the entries under the given path. /// - If `true`, list operation will list all entries that starts with given path. /// /// Default to `false`. pub fn recursive(self, v: bool) -> Self { self.map(|args| args.with_recursive(v)) } /// The version is used to control whether the object versions should be returned. /// /// - If `false`, list operation will not return with object versions /// - If `true`, list operation will return with object versions if object versioning is supported /// by the underlying service /// /// Default to `false` #[deprecated(since = "0.51.1", note = "use versions instead")] pub fn version(self, v: bool) -> Self { self.map(|args| args.with_versions(v)) } /// Controls whether the `list` operation should return file versions. /// /// This function allows you to specify if the `list` operation, when executed, should include /// information about different versions of files, if versioning is supported and enabled. /// /// If `true`, subsequent `list` operations will include version information for each file. /// If `false`, version information will be omitted from the `list` results. /// /// Default to `false` pub fn versions(self, v: bool) -> Self { self.map(|args| args.with_versions(v)) } /// Controls whether the `list` operation should include deleted files (or versions). /// /// This function allows you to specify if the `list` operation, when executed, should include /// entries for files or versions that have been marked as deleted. This is particularly relevant /// in object storage systems that support soft deletion or versioning. /// /// If `true`, subsequent `list` operations will include deleted files or versions. /// If `false`, deleted files or versions will be excluded from the `list` results. pub fn deleted(self, v: bool) -> Self { self.map(|args| args.with_deleted(v)) } } opendal-0.52.0/src/types/read/buffer_stream.rs000064400000000000000000000200551046102023000173740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ops::RangeBounds; use std::pin::Pin; use std::sync::Arc; use std::task::Context; use std::task::Poll; use futures::ready; use futures::Stream; use crate::raw::oio::Read; use crate::raw::*; use crate::*; /// StreamingReader will stream the content of the file without reading into /// memory first. /// /// StreamingReader is good for small memory footprint and optimized for latency. pub struct StreamingReader { generator: ReadGenerator, reader: Option, } impl StreamingReader { /// Create a new streaming reader. #[inline] fn new(ctx: Arc, range: BytesRange) -> Self { let generator = ReadGenerator::new(ctx.clone(), range.offset(), range.size()); Self { generator, reader: None, } } } impl oio::Read for StreamingReader { async fn read(&mut self) -> Result { loop { if self.reader.is_none() { self.reader = self.generator.next_reader().await?; } let Some(r) = self.reader.as_mut() else { return Ok(Buffer::new()); }; let buf = r.read().await?; // Reset reader to None if this reader returns empty buffer. if buf.is_empty() { self.reader = None; continue; } else { return Ok(buf); } } } } /// ChunkedReader will read the file in chunks. /// /// ChunkedReader is good for concurrent read and optimized for throughput. pub struct ChunkedReader { generator: ReadGenerator, tasks: ConcurrentTasks, done: bool, } impl ChunkedReader { /// Create a new chunked reader. /// /// # Notes /// /// We don't need to handle `Executor::timeout` since we are outside the layer. fn new(ctx: Arc, range: BytesRange) -> Self { let tasks = ConcurrentTasks::new( ctx.args().executor().cloned().unwrap_or_default(), ctx.options().concurrent(), |mut r: oio::Reader| { Box::pin(async { match r.read_all().await { Ok(buf) => (r, Ok(buf)), Err(err) => (r, Err(err)), } }) }, ); let generator = ReadGenerator::new(ctx, range.offset(), range.size()); Self { generator, tasks, done: false, } } } impl oio::Read for ChunkedReader { async fn read(&mut self) -> Result { while self.tasks.has_remaining() && !self.done { if let Some(r) = self.generator.next_reader().await? { self.tasks.execute(r).await?; } else { self.done = true; break; } if self.tasks.has_result() { break; } } Ok(self.tasks.next().await.transpose()?.unwrap_or_default()) } } /// BufferStream is a stream of buffers. /// /// # Notes /// /// The underlying reader is either a StreamingReader or a ChunkedReader. /// /// - If chunk is None, BufferStream will use StreamingReader to iterate /// data in streaming way. /// - Otherwise, BufferStream will use ChunkedReader to read data in chunks. pub struct BufferStream { state: State, } enum State { Idle(Option>), Reading(BoxedStaticFuture<(TwoWays, Result)>), } impl BufferStream { /// Create a new buffer stream with already calculated offset and size. pub fn new(ctx: Arc, offset: u64, size: Option) -> Self { debug_assert!( size.is_some() || ctx.options().chunk().is_none(), "size must be known if chunk is set" ); let reader = if ctx.options().chunk().is_some() { TwoWays::Two(ChunkedReader::new(ctx, BytesRange::new(offset, size))) } else { TwoWays::One(StreamingReader::new(ctx, BytesRange::new(offset, size))) }; Self { state: State::Idle(Some(reader)), } } /// Create a new buffer stream with given range bound. /// /// If users is going to perform chunked read but the read size is unknown, we will parse /// into range first. pub async fn create(ctx: Arc, range: impl RangeBounds) -> Result { let reader = if ctx.options().chunk().is_some() { let range = ctx.parse_into_range(range).await?; TwoWays::Two(ChunkedReader::new(ctx, range.into())) } else { TwoWays::One(StreamingReader::new(ctx, range.into())) }; Ok(Self { state: State::Idle(Some(reader)), }) } } impl Stream for BufferStream { type Item = Result; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { let this = self.get_mut(); loop { match &mut this.state { State::Idle(reader) => { let mut reader = reader.take().unwrap(); let fut = async move { let ret = reader.read().await; (reader, ret) }; this.state = State::Reading(Box::pin(fut)); } State::Reading(fut) => { let fut = fut.as_mut(); let (reader, buf) = ready!(fut.poll(cx)); this.state = State::Idle(Some(reader)); return match buf { Ok(buf) if buf.is_empty() => Poll::Ready(None), Ok(buf) => Poll::Ready(Some(Ok(buf))), Err(err) => Poll::Ready(Some(Err(err))), }; } } } } } #[cfg(test)] mod tests { use std::sync::Arc; use bytes::Buf; use bytes::Bytes; use futures::TryStreamExt; use pretty_assertions::assert_eq; use super::*; #[tokio::test] async fn test_trait() -> Result<()> { let acc = Operator::via_iter(Scheme::Memory, [])?.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new(), )); let v = BufferStream::create(ctx, 4..8).await?; let _: Box = Box::new(v); Ok(()) } #[tokio::test] async fn test_buffer_stream() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new(), )); let s = BufferStream::create(ctx, 4..8).await?; let bufs: Vec<_> = s.try_collect().await.unwrap(); assert_eq!(bufs.len(), 1); assert_eq!(bufs[0].chunk(), "o".as_bytes()); let buf: Buffer = bufs.into_iter().flatten().collect(); assert_eq!(buf.len(), 4); assert_eq!(&buf.to_vec(), "oWor".as_bytes()); Ok(()) } } opendal-0.52.0/src/types/read/futures_async_reader.rs000064400000000000000000000206661046102023000207740ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io; use std::io::SeekFrom; use std::ops::Range; use std::pin::Pin; use std::sync::Arc; use std::task::ready; use std::task::Context; use std::task::Poll; use bytes::Buf; use futures::AsyncBufRead; use futures::AsyncRead; use futures::AsyncSeek; use futures::StreamExt; use crate::raw::*; use crate::*; /// FuturesAsyncReader is the adapter of [`AsyncRead`], [`AsyncBufRead`] /// and [`AsyncSeek`] generated by [`Reader::into_futures_async_read`]. /// /// Users can use this adapter in cases where they need to use [`AsyncRead`] /// related trait. FuturesAsyncReader reuses the same concurrent and chunk /// settings from [`Reader`]. /// /// FuturesAsyncReader also implements [`Unpin`], [`Send`] and [`Sync`] pub struct FuturesAsyncReader { ctx: Arc, stream: BufferStream, buf: Buffer, start: u64, end: u64, pos: u64, } /// Safety: FuturesAsyncReader only exposes `&mut self` to the outside world, unsafe impl Sync for FuturesAsyncReader {} impl FuturesAsyncReader { /// NOTE: don't allow users to create FuturesAsyncReader directly. /// /// # TODO /// /// Extend this API to accept `impl RangeBounds`. #[inline] pub(super) fn new(ctx: Arc, range: Range) -> Self { let (start, end) = (range.start, range.end); let stream = BufferStream::new(ctx.clone(), start, Some(end - start)); FuturesAsyncReader { ctx, stream, buf: Buffer::new(), start, end, pos: 0, } } } impl AsyncBufRead for FuturesAsyncReader { fn poll_fill_buf(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { let this = self.get_mut(); loop { if this.buf.has_remaining() { return Poll::Ready(Ok(this.buf.chunk())); } this.buf = match ready!(this.stream.poll_next_unpin(cx)) { Some(Ok(buf)) => buf, Some(Err(err)) => return Poll::Ready(Err(format_std_io_error(err))), None => return Poll::Ready(Ok(&[])), }; } } fn consume(mut self: Pin<&mut Self>, amt: usize) { self.buf.advance(amt); // Make sure buf has been dropped before starting new request. // Otherwise, we will hold those bytes in memory until next // buffer reaching. if self.buf.is_empty() { self.buf = Buffer::new(); } self.pos += amt as u64; } } /// TODO: implement vectored read. impl AsyncRead for FuturesAsyncReader { fn poll_read( self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut [u8], ) -> Poll> { let this = self.get_mut(); loop { if this.buf.remaining() > 0 { let size = this.buf.remaining().min(buf.len()); this.buf.copy_to_slice(&mut buf[..size]); this.pos += size as u64; return Poll::Ready(Ok(size)); } this.buf = match ready!(this.stream.poll_next_unpin(cx)) { Some(Ok(buf)) => buf, Some(Err(err)) => return Poll::Ready(Err(format_std_io_error(err))), None => return Poll::Ready(Ok(0)), }; } } } impl AsyncSeek for FuturesAsyncReader { fn poll_seek( mut self: Pin<&mut Self>, _: &mut Context<'_>, pos: SeekFrom, ) -> Poll> { let new_pos = match pos { SeekFrom::Start(pos) => pos as i64, SeekFrom::End(pos) => self.end as i64 - self.start as i64 + pos, SeekFrom::Current(pos) => self.pos as i64 + pos, }; // Check if new_pos is negative. if new_pos < 0 { return Poll::Ready(Err(io::Error::new( io::ErrorKind::InvalidInput, "invalid seek to a negative position", ))); } let new_pos = new_pos as u64; if (self.pos..self.pos + self.buf.remaining() as u64).contains(&new_pos) { let cnt = new_pos - self.pos; self.buf.advance(cnt as _); } else { self.buf = Buffer::new(); self.stream = BufferStream::new( self.ctx.clone(), new_pos + self.start, Some(self.end - self.start - new_pos), ); } self.pos = new_pos; Poll::Ready(Ok(self.pos)) } } #[cfg(test)] mod tests { use std::sync::Arc; use bytes::Bytes; use futures::AsyncBufReadExt; use futures::AsyncReadExt; use futures::AsyncSeekExt; use pretty_assertions::assert_eq; use super::*; #[tokio::test] async fn test_trait() -> Result<()> { let acc = Operator::via_iter(Scheme::Memory, [])?.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new(), )); let v = FuturesAsyncReader::new(ctx, 4..8); let _: Box = Box::new(v); Ok(()) } #[tokio::test] async fn test_futures_async_read() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new(), )); let mut fr = FuturesAsyncReader::new(ctx, 4..8); let mut bs = vec![]; fr.read_to_end(&mut bs).await.unwrap(); assert_eq!(&bs, "oWor".as_bytes()); let pos = fr.seek(SeekFrom::Current(-2)).await.unwrap(); assert_eq!(pos, 2); let mut bs = vec![]; fr.read_to_end(&mut bs).await.unwrap(); assert_eq!(&bs, "or".as_bytes()); Ok(()) } #[tokio::test] async fn test_futures_async_read_with_concurrent() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new().with_concurrent(3).with_chunk(1), )); let mut fr = FuturesAsyncReader::new(ctx, 4..8); let mut bs = vec![]; fr.read_to_end(&mut bs).await.unwrap(); assert_eq!(&bs, "oWor".as_bytes()); // let pos = fr.seek(SeekFrom::Current(-2)).await.unwrap(); // assert_eq!(pos, 2); // let mut bs = vec![]; // fr.read_to_end(&mut bs).await.unwrap(); // assert_eq!(&bs, "or".as_bytes()); Ok(()) } #[tokio::test] async fn test_futures_async_buf_read() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new().with_concurrent(3).with_chunk(1), )); let mut fr = FuturesAsyncReader::new(ctx, 4..8); let chunk = fr.fill_buf().await.unwrap(); assert_eq!(chunk, "o".as_bytes()); fr.consume_unpin(1); let chunk = fr.fill_buf().await.unwrap(); assert_eq!(chunk, "W".as_bytes()); Ok(()) } } opendal-0.52.0/src/types/read/futures_bytes_stream.rs000064400000000000000000000112671046102023000210330ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io; use std::ops::RangeBounds; use std::pin::Pin; use std::sync::Arc; use std::task::ready; use std::task::Context; use std::task::Poll; use bytes::Bytes; use futures::Stream; use futures::StreamExt; use crate::raw::*; use crate::*; /// FuturesBytesStream is the adapter of [`Stream`] generated by [`Reader::into_bytes_stream`]. /// /// Users can use this adapter in cases where they need to use [`Stream`] trait. FuturesBytesStream /// reuses the same concurrent adand chunk settings from [`Reader`]. ///ad /// FuturesStream also implements [`Unpin`], [`Send`] and [`Sync`]. pub struct FuturesBytesStream { stream: BufferStream, buf: Buffer, } /// Safety: FuturesBytesStream only exposes `&mut self` to the outside world, unsafe impl Sync for FuturesBytesStream {} impl FuturesBytesStream { /// NOTE: don't allow users to create FuturesStream directly. pub(crate) async fn new(ctx: Arc, range: impl RangeBounds) -> Result { let stream = BufferStream::create(ctx, range).await?; Ok(FuturesBytesStream { stream, buf: Buffer::new(), }) } } impl Stream for FuturesBytesStream { type Item = io::Result; fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { let this = self.get_mut(); loop { // Consume current buffer if let Some(bs) = Iterator::next(&mut this.buf) { return Poll::Ready(Some(Ok(bs))); } this.buf = match ready!(this.stream.poll_next_unpin(cx)) { Some(Ok(buf)) => buf, Some(Err(err)) => return Poll::Ready(Some(Err(format_std_io_error(err)))), None => return Poll::Ready(None), }; } } } #[cfg(test)] mod tests { use std::sync::Arc; use bytes::Bytes; use futures::TryStreamExt; use pretty_assertions::assert_eq; use super::*; #[tokio::test] async fn test_trait() -> Result<()> { let acc = Operator::via_iter(Scheme::Memory, [])?.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new(), )); let v = FuturesBytesStream::new(ctx, 4..8).await?; let _: Box = Box::new(v); Ok(()) } #[tokio::test] async fn test_futures_bytes_stream() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new(), )); let s = FuturesBytesStream::new(ctx, 4..8).await?; let bufs: Vec = s.try_collect().await.unwrap(); assert_eq!(&bufs[0], "o".as_bytes()); assert_eq!(&bufs[1], "Wor".as_bytes()); Ok(()) } #[tokio::test] async fn test_futures_bytes_stream_with_concurrent() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = Arc::new(ReadContext::new( acc, "test".to_string(), OpRead::new(), OpReader::new().with_concurrent(3).with_chunk(1), )); let s = FuturesBytesStream::new(ctx, 4..8).await?; let bufs: Vec = s.try_collect().await.unwrap(); assert_eq!(&bufs[0], "o".as_bytes()); assert_eq!(&bufs[1], "W".as_bytes()); assert_eq!(&bufs[2], "o".as_bytes()); assert_eq!(&bufs[3], "r".as_bytes()); Ok(()) } } opendal-0.52.0/src/types/read/mod.rs000064400000000000000000000021031046102023000153210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. #[allow(clippy::module_inception)] mod reader; pub use reader::Reader; mod buffer_stream; pub(crate) use buffer_stream::BufferStream; mod futures_async_reader; pub use futures_async_reader::FuturesAsyncReader; mod futures_bytes_stream; pub use futures_bytes_stream::FuturesBytesStream; opendal-0.52.0/src/types/read/reader.rs000064400000000000000000000352251046102023000160170ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ops::Range; use std::ops::RangeBounds; use std::sync::Arc; use bytes::BufMut; use futures::stream; use futures::StreamExt; use futures::TryStreamExt; use crate::*; /// Reader is designed to read data from given path in an asynchronous /// manner. /// /// # Usage /// /// [`Reader`] provides multiple ways to read data from given reader. /// /// `Reader` implements `Clone` so you can clone it and store in place where ever you want. /// /// ## Direct /// /// [`Reader`] provides public API including [`Reader::read`]. You can use those APIs directly without extra copy. /// /// ``` /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> Result<()> { /// let r = op.reader("path/to/file").await?; /// let bs = r.read(0..1024).await?; /// Ok(()) /// } /// ``` /// /// ## Read like `Stream` /// /// ``` /// use anyhow::Result; /// use bytes::Bytes; /// use futures::TryStreamExt; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let s = op /// .reader("path/to/file") /// .await? /// .into_bytes_stream(1024..2048) /// .await?; /// let bs: Vec = s.try_collect().await?; /// Ok(()) /// } /// ``` /// /// ## Read like `AsyncRead` and `AsyncBufRead` /// /// ``` /// use anyhow::Result; /// use bytes::Bytes; /// use futures::AsyncReadExt; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let mut r = op /// .reader("path/to/file") /// .await? /// .into_futures_async_read(1024..2048) /// .await?; /// let mut bs = vec![]; /// let n = r.read_to_end(&mut bs).await?; /// Ok(()) /// } /// ``` #[derive(Clone)] pub struct Reader { ctx: Arc, } impl Reader { /// Create a new reader. /// /// Create will use internal information to decide the most suitable /// implementation for users. /// /// We don't want to expose those details to users so keep this function /// in crate only. pub(crate) fn new(ctx: ReadContext) -> Self { Reader { ctx: Arc::new(ctx) } } /// Read give range from reader into [`Buffer`]. /// /// This operation is zero-copy, which means it keeps the [`bytes::Bytes`] returned by underlying /// storage services without any extra copy or intensive memory allocations. pub async fn read(&self, range: impl RangeBounds) -> Result { let bufs: Vec<_> = self.clone().into_stream(range).await?.try_collect().await?; Ok(bufs.into_iter().flatten().collect()) } /// Read all data from reader into given [`BufMut`]. /// /// This operation will copy and write bytes into given [`BufMut`]. Allocation happens while /// [`BufMut`] doesn't have enough space. pub async fn read_into( &self, buf: &mut impl BufMut, range: impl RangeBounds, ) -> Result { let mut stream = self.clone().into_stream(range).await?; let mut read = 0; loop { let Some(bs) = stream.try_next().await? else { return Ok(read); }; read += bs.len(); buf.put(bs); } } /// Fetch specific ranges from reader. /// /// This operation try to merge given ranges into a list of /// non-overlapping ranges. Users may also specify a `gap` to merge /// close ranges. /// /// The returning `Buffer` may share the same underlying memory without /// any extra copy. pub async fn fetch(&self, ranges: Vec>) -> Result> { let merged_ranges = self.merge_ranges(ranges.clone()); let merged_bufs: Vec<_> = stream::iter(merged_ranges.clone().into_iter().map(|v| self.read(v))) .buffered(self.ctx.options().concurrent()) .try_collect() .await?; let mut bufs = Vec::with_capacity(ranges.len()); for range in ranges { let idx = merged_ranges.partition_point(|v| v.start <= range.start) - 1; let start = range.start - merged_ranges[idx].start; let end = range.end - merged_ranges[idx].start; bufs.push(merged_bufs[idx].slice(start as usize..end as usize)); } Ok(bufs) } /// Merge given ranges into a list of non-overlapping ranges. fn merge_ranges(&self, mut ranges: Vec>) -> Vec> { let gap = self.ctx.options().gap().unwrap_or(1024 * 1024) as u64; // We don't care about the order of range with same start, they // will be merged in the next step. ranges.sort_unstable_by(|a, b| a.start.cmp(&b.start)); // We know that this vector will have at most element let mut merged = Vec::with_capacity(ranges.len()); let mut cur = ranges[0].clone(); for range in ranges.into_iter().skip(1) { if range.start <= cur.end + gap { // There is an overlap or the gap is small enough to merge cur.end = cur.end.max(range.end); } else { // No overlap and the gap is too large, push the current range to the list and start a new one merged.push(cur); cur = range; } } // Push the last range merged.push(cur); merged } /// Create a buffer stream to read specific range from given reader. /// /// # Notes /// /// This API can be public but we are not sure if it's useful for users. /// And the name `BufferStream` is not good enough to expose to users. /// Let's keep it inside for now. async fn into_stream(self, range: impl RangeBounds) -> Result { BufferStream::create(self.ctx, range).await } /// Convert reader into [`FuturesAsyncReader`] which implements [`futures::AsyncRead`], /// [`futures::AsyncSeek`] and [`futures::AsyncBufRead`]. /// /// # Notes /// /// FuturesAsyncReader is not a zero-cost abstraction. The underlying reader /// returns an owned [`Buffer`], which involves an extra copy operation. /// /// # Examples /// /// ## Basic Usage /// /// ``` /// use std::io; /// /// use futures::io::AsyncReadExt; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> io::Result<()> { /// let mut r = op /// .reader("hello.txt") /// .await? /// .into_futures_async_read(1024..2048) /// .await?; /// let mut bs = Vec::new(); /// r.read_to_end(&mut bs).await?; /// /// Ok(()) /// } /// ``` /// /// ## Concurrent Read /// /// The following example reads data in 256B chunks with 8 concurrent. /// /// ``` /// use std::io; /// /// use futures::io::AsyncReadExt; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> io::Result<()> { /// let mut r = op /// .reader_with("hello.txt") /// .concurrent(8) /// .chunk(256) /// .await? /// .into_futures_async_read(1024..2048) /// .await?; /// let mut bs = Vec::new(); /// r.read_to_end(&mut bs).await?; /// /// Ok(()) /// } /// ``` #[inline] pub async fn into_futures_async_read( self, range: impl RangeBounds, ) -> Result { let range = self.ctx.parse_into_range(range).await?; Ok(FuturesAsyncReader::new(self.ctx, range)) } /// Convert reader into [`FuturesBytesStream`] which implements [`futures::Stream`]. /// /// # Examples /// /// ## Basic Usage /// /// ``` /// use std::io; /// /// use bytes::Bytes; /// use futures::TryStreamExt; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> io::Result<()> { /// let mut s = op /// .reader("hello.txt") /// .await? /// .into_bytes_stream(1024..2048) /// .await?; /// let bs: Vec = s.try_collect().await?; /// /// Ok(()) /// } /// ``` /// /// ## Concurrent Read /// /// The following example reads data in 256B chunks with 8 concurrent. /// /// ``` /// use std::io; /// /// use bytes::Bytes; /// use futures::TryStreamExt; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> io::Result<()> { /// let mut s = op /// .reader_with("hello.txt") /// .concurrent(8) /// .chunk(256) /// .await? /// .into_bytes_stream(1024..2048) /// .await?; /// let bs: Vec = s.try_collect().await?; /// /// Ok(()) /// } /// ``` #[inline] pub async fn into_bytes_stream( self, range: impl RangeBounds, ) -> Result { FuturesBytesStream::new(self.ctx, range).await } } #[cfg(test)] mod tests { use bytes::Bytes; use rand::rngs::ThreadRng; use rand::Rng; use rand::RngCore; use super::*; use crate::raw::*; use crate::services; use crate::Operator; #[tokio::test] async fn test_trait() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; op.write( "test", Buffer::from(vec![Bytes::from("Hello"), Bytes::from("World")]), ) .await?; let acc = op.into_inner(); let ctx = ReadContext::new(acc, "test".to_string(), OpRead::new(), OpReader::new()); let _: Box = Box::new(Reader::new(ctx)); Ok(()) } fn gen_random_bytes() -> Vec { let mut rng = ThreadRng::default(); // Generate size between 1B..16MB. let size = rng.gen_range(1..16 * 1024 * 1024); let mut content = vec![0; size]; rng.fill_bytes(&mut content); content } fn gen_fixed_bytes(size: usize) -> Vec { let mut rng = ThreadRng::default(); let mut content = vec![0; size]; rng.fill_bytes(&mut content); content } #[tokio::test] async fn test_reader_read() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; let path = "test_file"; let content = gen_random_bytes(); op.write(path, content.clone()) .await .expect("write must succeed"); let reader = op.reader(path).await.unwrap(); let buf = reader.read(..).await.expect("read to end must succeed"); assert_eq!(buf.to_bytes(), content); Ok(()) } #[tokio::test] async fn test_reader_read_with_chunk() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; let path = "test_file"; let content = gen_random_bytes(); op.write(path, content.clone()) .await .expect("write must succeed"); let reader = op.reader_with(path).chunk(16).await.unwrap(); let buf = reader.read(..).await.expect("read to end must succeed"); assert_eq!(buf.to_bytes(), content); Ok(()) } #[tokio::test] async fn test_reader_read_with_concurrent() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; let path = "test_file"; let content = gen_random_bytes(); op.write(path, content.clone()) .await .expect("write must succeed"); let reader = op .reader_with(path) .chunk(128) .concurrent(16) .await .unwrap(); let buf = reader.read(..).await.expect("read to end must succeed"); assert_eq!(buf.to_bytes(), content); Ok(()) } #[tokio::test] async fn test_reader_read_into() -> Result<()> { let op = Operator::via_iter(Scheme::Memory, [])?; let path = "test_file"; let content = gen_random_bytes(); op.write(path, content.clone()) .await .expect("write must succeed"); let reader = op.reader(path).await.unwrap(); let mut buf = Vec::new(); reader .read_into(&mut buf, ..) .await .expect("read to end must succeed"); assert_eq!(buf, content); Ok(()) } #[tokio::test] async fn test_merge_ranges() -> Result<()> { let op = Operator::new(services::Memory::default()).unwrap().finish(); let path = "test_file"; let content = gen_random_bytes(); op.write(path, content.clone()) .await .expect("write must succeed"); let reader = op.reader_with(path).gap(1).await.unwrap(); let ranges = vec![0..10, 10..20, 21..30, 40..50, 40..60, 45..59]; let merged = reader.merge_ranges(ranges.clone()); assert_eq!(merged, vec![0..30, 40..60]); Ok(()) } #[tokio::test] async fn test_fetch() -> Result<()> { let op = Operator::new(services::Memory::default()).unwrap().finish(); let path = "test_file"; let content = gen_fixed_bytes(1024); op.write(path, content.clone()) .await .expect("write must succeed"); let reader = op.reader_with(path).gap(1).await.unwrap(); let ranges = vec![ 0..10, 40..50, 45..59, 10..20, 21..30, 40..50, 40..60, 45..59, ]; let merged = reader .fetch(ranges.clone()) .await .expect("fetch must succeed"); for (i, range) in ranges.iter().enumerate() { assert_eq!( merged[i].to_bytes(), content[range.start as usize..range.end as usize] ); } Ok(()) } } opendal-0.52.0/src/types/scheme.rs000064400000000000000000000441211046102023000151010ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashSet; use std::fmt::Display; use std::fmt::Formatter; use std::str::FromStr; use crate::Error; /// Services that OpenDAL supports /// /// # Notes /// /// - Scheme is `non_exhaustive`, new variant COULD be added at any time. /// - New variant SHOULD be added in alphabet orders, /// - Users MUST NOT relay on its order. #[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)] #[non_exhaustive] pub enum Scheme { /// [aliyun_drive][crate::services::AliyunDrive]: Aliyun Drive services. AliyunDrive, /// [atomicserver][crate::services::Atomicserver]: Atomicserver services. Atomicserver, /// [azblob][crate::services::Azblob]: Azure Storage Blob services. Azblob, /// [Azdls][crate::services::Azdls]: Azure Data Lake Storage Gen2. Azdls, /// [B2][crate::services::B2]: Backblaze B2 Services. B2, /// [Compfs][crate::services::Compfs]: Compio fs Services. Compfs, /// [Seafile][crate::services::Seafile]: Seafile Services. Seafile, /// [Upyun][crate::services::Upyun]: Upyun Services. Upyun, /// [VercelBlob][crate::services::VercelBlob]: VercelBlob Services. VercelBlob, /// [YandexDisk][crate::services::YandexDisk]: YandexDisk Services. YandexDisk, /// [Pcloud][crate::services::Pcloud]: Pcloud Services. Pcloud, /// [Koofr][crate::services::Koofr]: Koofr Services. Koofr, /// [Chainsafe][crate::services::Chainsafe]: Chainsafe Services. Chainsafe, /// [cacache][crate::services::Cacache]: cacache backend support. Cacache, /// [cloudflare-kv][crate::services::CloudflareKv]: Cloudflare KV services. CloudflareKv, /// [cos][crate::services::Cos]: Tencent Cloud Object Storage services. Cos, /// [d1][crate::services::D1]: D1 services D1, /// [dashmap][crate::services::Dashmap]: dashmap backend support. Dashmap, /// [etcd][crate::services::Etcd]: Etcd Services Etcd, /// [foundationdb][crate::services::Foundationdb]: Foundationdb services. Foundationdb, /// [dbfs][crate::services::Dbfs]: DBFS backend support. Dbfs, /// [fs][crate::services::Fs]: POSIX-like file system. Fs, /// [ftp][crate::services::Ftp]: FTP backend. Ftp, /// [gcs][crate::services::Gcs]: Google Cloud Storage backend. Gcs, /// [ghac][crate::services::Ghac]: GitHub Action Cache services. Ghac, /// [hdfs][crate::services::Hdfs]: Hadoop Distributed File System. Hdfs, /// [http][crate::services::Http]: HTTP backend. Http, /// [huggingface][crate::services::Huggingface]: Huggingface services. Huggingface, /// [alluxio][crate::services::Alluxio]: Alluxio services. Alluxio, /// [ipmfs][crate::services::Ipfs]: IPFS HTTP Gateway Ipfs, /// [ipmfs][crate::services::Ipmfs]: IPFS mutable file system Ipmfs, /// [icloud][crate::services::Icloud]: APPLE icloud services. Icloud, /// [memcached][crate::services::Memcached]: Memcached service support. Memcached, /// [memory][crate::services::Memory]: In memory backend support. Memory, /// [mini-moka][crate::services::MiniMoka]: Mini Moka backend support. MiniMoka, /// [moka][crate::services::Moka]: moka backend support. Moka, /// [monoiofs][crate::services::Monoiofs]: monoio fs services. Monoiofs, /// [obs][crate::services::Obs]: Huawei Cloud OBS services. Obs, /// [onedrive][crate::services::Onedrive]: Microsoft OneDrive services. Onedrive, /// [gdrive][crate::services::Gdrive]: GoogleDrive services. Gdrive, /// [dropbox][crate::services::Dropbox]: Dropbox services. Dropbox, /// [oss][crate::services::Oss]: Aliyun Object Storage Services Oss, /// [persy][crate::services::Persy]: persy backend support. Persy, /// [redis][crate::services::Redis]: Redis services Redis, /// [postgresql][crate::services::Postgresql]: Postgresql services Postgresql, /// [mysql][crate::services::Mysql]: Mysql services Mysql, /// [sqlite][crate::services::Sqlite]: Sqlite services Sqlite, /// [rocksdb][crate::services::Rocksdb]: RocksDB services Rocksdb, /// [s3][crate::services::S3]: AWS S3 alike services. S3, /// [sftp][crate::services::Sftp]: SFTP services Sftp, /// [sled][crate::services::Sled]: Sled services Sled, /// [Supabase][crate::services::Supabase]: Supabase storage service Supabase, /// [swift][crate::services::Swift]: Swift backend support. Swift, /// [Vercel Artifacts][crate::services::VercelArtifacts]: Vercel Artifacts service, as known as Vercel Remote Caching. VercelArtifacts, /// [webdav][crate::services::Webdav]: WebDAV support. Webdav, /// [webhdfs][crate::services::Webhdfs]: WebHDFS RESTful API Services Webhdfs, /// [redb][crate::services::Redb]: Redb Services Redb, /// [tikv][crate::services::Tikv]: Tikv Services Tikv, /// [azfile][crate::services::Azfile]: Azfile Services Azfile, /// [mongodb](crate::services::Mongodb): MongoDB Services Mongodb, /// [gridfs](crate::services::Gridfs): MongoDB Gridfs Services Gridfs, /// [Github Contents][crate::services::Github]: Github contents support. Github, /// [Native HDFS](crate::services::HdfsNative): Hdfs Native service, using rust hdfs-native client for hdfs HdfsNative, /// [surrealdb](crate::services::Surrealdb): Surrealdb Services Surrealdb, /// [lakefs](crate::services::Lakefs): LakeFS Services Lakefs, /// [NebulaGraph](crate::services::NebulaGraph): NebulaGraph Services NebulaGraph, /// Custom that allow users to implement services outside of OpenDAL. /// /// # NOTE /// /// - Custom must not overwrite any existing services name. /// - Custom must be in lower case. Custom(&'static str), } impl Scheme { /// Convert self into static str. pub fn into_static(self) -> &'static str { self.into() } /// Get all enabled schemes. /// /// OpenDAL could be compiled with different features, which will enable different schemes. /// This function returns all enabled schemes so users can make decisions based on it. /// /// # Examples /// /// ```rust,no_run /// use opendal::Scheme; /// /// let enabled_schemes = Scheme::enabled(); /// if !enabled_schemes.contains(&Scheme::Memory) { /// panic!("s3 support is not enabled") /// } /// ``` pub fn enabled() -> HashSet { HashSet::from([ #[cfg(feature = "services-aliyun-drive")] Scheme::AliyunDrive, #[cfg(feature = "services-atomicserver")] Scheme::Atomicserver, #[cfg(feature = "services-alluxio")] Scheme::Alluxio, #[cfg(feature = "services-azblob")] Scheme::Azblob, #[cfg(feature = "services-azdls")] Scheme::Azdls, #[cfg(feature = "services-azfile")] Scheme::Azfile, #[cfg(feature = "services-b2")] Scheme::B2, #[cfg(feature = "services-cacache")] Scheme::Cacache, #[cfg(feature = "services-cos")] Scheme::Cos, #[cfg(feature = "services-compfs")] Scheme::Compfs, #[cfg(feature = "services-dashmap")] Scheme::Dashmap, #[cfg(feature = "services-dropbox")] Scheme::Dropbox, #[cfg(feature = "services-etcd")] Scheme::Etcd, #[cfg(feature = "services-foundationdb")] Scheme::Foundationdb, #[cfg(feature = "services-fs")] Scheme::Fs, #[cfg(feature = "services-ftp")] Scheme::Ftp, #[cfg(feature = "services-gcs")] Scheme::Gcs, #[cfg(feature = "services-ghac")] Scheme::Ghac, #[cfg(feature = "services-hdfs")] Scheme::Hdfs, #[cfg(feature = "services-http")] Scheme::Http, #[cfg(feature = "services-huggingface")] Scheme::Huggingface, #[cfg(feature = "services-ipfs")] Scheme::Ipfs, #[cfg(feature = "services-ipmfs")] Scheme::Ipmfs, #[cfg(feature = "services-icloud")] Scheme::Icloud, #[cfg(feature = "services-memcached")] Scheme::Memcached, #[cfg(feature = "services-memory")] Scheme::Memory, #[cfg(feature = "services-mini-moka")] Scheme::MiniMoka, #[cfg(feature = "services-moka")] Scheme::Moka, #[cfg(feature = "services-monoiofs")] Scheme::Monoiofs, #[cfg(feature = "services-mysql")] Scheme::Mysql, #[cfg(feature = "services-obs")] Scheme::Obs, #[cfg(feature = "services-onedrive")] Scheme::Onedrive, #[cfg(feature = "services-postgresql")] Scheme::Postgresql, #[cfg(feature = "services-gdrive")] Scheme::Gdrive, #[cfg(feature = "services-oss")] Scheme::Oss, #[cfg(feature = "services-persy")] Scheme::Persy, #[cfg(feature = "services-redis")] Scheme::Redis, #[cfg(feature = "services-rocksdb")] Scheme::Rocksdb, #[cfg(feature = "services-s3")] Scheme::S3, #[cfg(feature = "services-seafile")] Scheme::Seafile, #[cfg(feature = "services-upyun")] Scheme::Upyun, #[cfg(feature = "services-yandex-disk")] Scheme::YandexDisk, #[cfg(feature = "services-pcloud")] Scheme::Pcloud, #[cfg(feature = "services-sftp")] Scheme::Sftp, #[cfg(feature = "services-sled")] Scheme::Sled, #[cfg(feature = "services-sqlite")] Scheme::Sqlite, #[cfg(feature = "services-supabase")] Scheme::Supabase, #[cfg(feature = "services-swift")] Scheme::Swift, #[cfg(feature = "services-tikv")] Scheme::Tikv, #[cfg(feature = "services-vercel-artifacts")] Scheme::VercelArtifacts, #[cfg(feature = "services-vercel-blob")] Scheme::VercelBlob, #[cfg(feature = "services-webdav")] Scheme::Webdav, #[cfg(feature = "services-webhdfs")] Scheme::Webhdfs, #[cfg(feature = "services-redb")] Scheme::Redb, #[cfg(feature = "services-mongodb")] Scheme::Mongodb, #[cfg(feature = "services-hdfs-native")] Scheme::HdfsNative, #[cfg(feature = "services-surrealdb")] Scheme::Surrealdb, #[cfg(feature = "services-lakefs")] Scheme::Lakefs, #[cfg(feature = "services-nebula-graph")] Scheme::NebulaGraph, ]) } } impl Default for Scheme { fn default() -> Self { Self::Memory } } impl Display for Scheme { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { write!(f, "{}", self.into_static()) } } impl FromStr for Scheme { type Err = Error; fn from_str(s: &str) -> Result { let s = s.to_lowercase(); match s.as_str() { "aliyun_drive" => Ok(Scheme::AliyunDrive), "atomicserver" => Ok(Scheme::Atomicserver), "azblob" => Ok(Scheme::Azblob), "alluxio" => Ok(Scheme::Alluxio), // Notes: // // OpenDAL used to call `azdls` as `azdfs`, we keep it for backward compatibility. // And abfs is widely used in hadoop ecosystem, keep it for easy to use. "azdls" | "azdfs" | "abfs" => Ok(Scheme::Azdls), "b2" => Ok(Scheme::B2), "chainsafe" => Ok(Scheme::Chainsafe), "cacache" => Ok(Scheme::Cacache), "compfs" => Ok(Scheme::Compfs), "cloudflare_kv" => Ok(Scheme::CloudflareKv), "cos" => Ok(Scheme::Cos), "d1" => Ok(Scheme::D1), "dashmap" => Ok(Scheme::Dashmap), "dropbox" => Ok(Scheme::Dropbox), "etcd" => Ok(Scheme::Etcd), "dbfs" => Ok(Scheme::Dbfs), "fs" => Ok(Scheme::Fs), "gcs" => Ok(Scheme::Gcs), "gdrive" => Ok(Scheme::Gdrive), "ghac" => Ok(Scheme::Ghac), "gridfs" => Ok(Scheme::Gridfs), "github" => Ok(Scheme::Github), "hdfs" => Ok(Scheme::Hdfs), "http" | "https" => Ok(Scheme::Http), "huggingface" | "hf" => Ok(Scheme::Huggingface), "ftp" | "ftps" => Ok(Scheme::Ftp), "ipfs" | "ipns" => Ok(Scheme::Ipfs), "ipmfs" => Ok(Scheme::Ipmfs), "icloud" => Ok(Scheme::Icloud), "koofr" => Ok(Scheme::Koofr), "memcached" => Ok(Scheme::Memcached), "memory" => Ok(Scheme::Memory), "mysql" => Ok(Scheme::Mysql), "sqlite" => Ok(Scheme::Sqlite), "mini_moka" => Ok(Scheme::MiniMoka), "moka" => Ok(Scheme::Moka), "monoiofs" => Ok(Scheme::Monoiofs), "obs" => Ok(Scheme::Obs), "onedrive" => Ok(Scheme::Onedrive), "persy" => Ok(Scheme::Persy), "postgresql" => Ok(Scheme::Postgresql), "redb" => Ok(Scheme::Redb), "redis" => Ok(Scheme::Redis), "rocksdb" => Ok(Scheme::Rocksdb), "s3" => Ok(Scheme::S3), "seafile" => Ok(Scheme::Seafile), "upyun" => Ok(Scheme::Upyun), "yandex_disk" => Ok(Scheme::YandexDisk), "pcloud" => Ok(Scheme::Pcloud), "sftp" => Ok(Scheme::Sftp), "sled" => Ok(Scheme::Sled), "supabase" => Ok(Scheme::Supabase), "swift" => Ok(Scheme::Swift), "oss" => Ok(Scheme::Oss), "vercel_artifacts" => Ok(Scheme::VercelArtifacts), "vercel_blob" => Ok(Scheme::VercelBlob), "webdav" => Ok(Scheme::Webdav), "webhdfs" => Ok(Scheme::Webhdfs), "tikv" => Ok(Scheme::Tikv), "azfile" => Ok(Scheme::Azfile), "mongodb" => Ok(Scheme::Mongodb), "hdfs_native" => Ok(Scheme::HdfsNative), "surrealdb" => Ok(Scheme::Surrealdb), "lakefs" => Ok(Scheme::Lakefs), "nebula_graph" => Ok(Scheme::NebulaGraph), _ => Ok(Scheme::Custom(Box::leak(s.into_boxed_str()))), } } } impl From for &'static str { fn from(v: Scheme) -> Self { match v { Scheme::AliyunDrive => "aliyun_drive", Scheme::Atomicserver => "atomicserver", Scheme::Azblob => "azblob", Scheme::Azdls => "azdls", Scheme::B2 => "b2", Scheme::Chainsafe => "chainsafe", Scheme::Cacache => "cacache", Scheme::CloudflareKv => "cloudflare_kv", Scheme::Cos => "cos", Scheme::Compfs => "compfs", Scheme::D1 => "d1", Scheme::Dashmap => "dashmap", Scheme::Etcd => "etcd", Scheme::Dbfs => "dbfs", Scheme::Fs => "fs", Scheme::Gcs => "gcs", Scheme::Ghac => "ghac", Scheme::Gridfs => "gridfs", Scheme::Hdfs => "hdfs", Scheme::Http => "http", Scheme::Huggingface => "huggingface", Scheme::Foundationdb => "foundationdb", Scheme::Ftp => "ftp", Scheme::Ipfs => "ipfs", Scheme::Ipmfs => "ipmfs", Scheme::Icloud => "icloud", Scheme::Koofr => "koofr", Scheme::Memcached => "memcached", Scheme::Memory => "memory", Scheme::MiniMoka => "mini_moka", Scheme::Moka => "moka", Scheme::Monoiofs => "monoiofs", Scheme::Obs => "obs", Scheme::Onedrive => "onedrive", Scheme::Persy => "persy", Scheme::Postgresql => "postgresql", Scheme::Mysql => "mysql", Scheme::Gdrive => "gdrive", Scheme::Github => "github", Scheme::Dropbox => "dropbox", Scheme::Redis => "redis", Scheme::Rocksdb => "rocksdb", Scheme::S3 => "s3", Scheme::Seafile => "seafile", Scheme::Sftp => "sftp", Scheme::Sled => "sled", Scheme::Supabase => "supabase", Scheme::Swift => "swift", Scheme::VercelArtifacts => "vercel_artifacts", Scheme::VercelBlob => "vercel_blob", Scheme::Oss => "oss", Scheme::Webdav => "webdav", Scheme::Webhdfs => "webhdfs", Scheme::Redb => "redb", Scheme::Tikv => "tikv", Scheme::Azfile => "azfile", Scheme::Sqlite => "sqlite", Scheme::Mongodb => "mongodb", Scheme::Alluxio => "alluxio", Scheme::Upyun => "upyun", Scheme::YandexDisk => "yandex_disk", Scheme::Pcloud => "pcloud", Scheme::HdfsNative => "hdfs_native", Scheme::Surrealdb => "surrealdb", Scheme::Lakefs => "lakefs", Scheme::NebulaGraph => "nebula_graph", Scheme::Custom(v) => v, } } } impl From for String { fn from(v: Scheme) -> Self { v.into_static().to_string() } } opendal-0.52.0/src/types/write/buffer_sink.rs000064400000000000000000000142701046102023000172660ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::pin::Pin; use std::task::ready; use std::task::Context; use std::task::Poll; use bytes::Buf; use crate::raw::*; use crate::*; /// BufferSink is the adapter of [`futures::Sink`] generated by [`Writer`]. /// /// Users can use this adapter in cases where they need to use [`futures::Sink`] pub struct BufferSink { state: State, buf: Buffer, } enum State { Idle(Option>), Writing(BoxedStaticFuture<(WriteGenerator, Result)>), Closing(BoxedStaticFuture<(WriteGenerator, Result)>), } /// # Safety /// /// FuturesReader only exposes `&mut self` to the outside world, so it's safe to be `Sync`. unsafe impl Sync for State {} impl BufferSink { /// Create a new sink from a [`oio::Writer`]. #[inline] pub(crate) fn new(w: WriteGenerator) -> Self { BufferSink { state: State::Idle(Some(w)), buf: Buffer::new(), } } } impl futures::Sink for BufferSink { type Error = Error; fn poll_ready(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { self.poll_flush(cx) } fn start_send(mut self: Pin<&mut Self>, item: Buffer) -> Result<()> { self.buf = item; Ok(()) } fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { let this = self.get_mut(); loop { match &mut this.state { State::Idle(w) => { if this.buf.is_empty() { return Poll::Ready(Ok(())); } let Some(mut w) = w.take() else { return Poll::Ready(Err(Error::new( ErrorKind::Unexpected, "state invalid: sink has been closed", ))); }; let buf = this.buf.clone(); let fut = async move { let res = w.write(buf).await; (w, res) }; this.state = State::Writing(Box::pin(fut)); } State::Writing(fut) => { let (w, res) = ready!(fut.as_mut().poll(cx)); this.state = State::Idle(Some(w)); match res { Ok(n) => { this.buf.advance(n); } Err(err) => return Poll::Ready(Err(err)), } } State::Closing(_) => { return Poll::Ready(Err(Error::new( ErrorKind::Unexpected, "state invalid: sink is closing", ))) } } } } fn poll_close(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { let this = self.get_mut(); loop { match &mut this.state { State::Idle(w) => { let Some(mut w) = w.take() else { return Poll::Ready(Err(Error::new( ErrorKind::Unexpected, "state invalid: sink has been closed", ))); }; if this.buf.is_empty() { let fut = async move { let res = w.close().await; (w, res) }; this.state = State::Closing(Box::pin(fut)); } else { let buf = this.buf.clone(); let fut = async move { let res = w.write(buf).await; (w, res) }; this.state = State::Writing(Box::pin(fut)); } } State::Writing(fut) => { let (w, res) = ready!(fut.as_mut().poll(cx)); this.state = State::Idle(Some(w)); match res { Ok(n) => { this.buf.advance(n); } Err(err) => return Poll::Ready(Err(err)), } } State::Closing(fut) => { let (w, res) = ready!(fut.as_mut().poll(cx)); this.state = State::Idle(Some(w)); match res { Ok(_) => { this.state = State::Idle(None); return Poll::Ready(Ok(())); } Err(err) => return Poll::Ready(Err(err)), } } } } } } #[cfg(test)] mod tests { use std::sync::Arc; use crate::raw::*; use crate::*; #[tokio::test] async fn test_trait() { let op = Operator::via_iter(Scheme::Memory, []).unwrap(); let acc = op.into_inner(); let ctx = Arc::new(WriteContext::new( acc, "test".to_string(), OpWrite::new(), OpWriter::new().with_chunk(1), )); let write_gen = WriteGenerator::create(ctx).await.unwrap(); let v = BufferSink::new(write_gen); let _: Box = Box::new(v); } } opendal-0.52.0/src/types/write/futures_async_writer.rs000064400000000000000000000100051046102023000212470ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::io; use std::pin::Pin; use std::task::ready; use std::task::Context; use std::task::Poll; use futures::AsyncWrite; use futures::SinkExt; use crate::raw::*; use crate::*; /// FuturesIoAsyncWriter is the adapter of [`AsyncWrite`] for [`Writer`]. /// /// Users can use this adapter in cases where they need to use [`AsyncWrite`] related trait. /// /// FuturesIoAsyncWriter also implements [`Unpin`], [`Send`] and [`Sync`] pub struct FuturesAsyncWriter { sink: BufferSink, buf: oio::FlexBuf, } impl FuturesAsyncWriter { /// NOTE: don't allow users to create directly. #[inline] pub(crate) fn new(w: WriteGenerator) -> Self { FuturesAsyncWriter { sink: BufferSink::new(w), buf: oio::FlexBuf::new(256 * 1024), } } } impl AsyncWrite for FuturesAsyncWriter { fn poll_write( self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8], ) -> Poll> { let this = self.get_mut(); loop { let n = this.buf.put(buf); if n > 0 { return Poll::Ready(Ok(n)); } ready!(this.sink.poll_ready_unpin(cx)).map_err(format_std_io_error)?; let bs = this.buf.get().expect("frozen buffer must be valid"); this.sink .start_send_unpin(Buffer::from(bs)) .map_err(format_std_io_error)?; this.buf.clean(); } } fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { let this = self.get_mut(); loop { // Make sure buf has been frozen. this.buf.freeze(); let Some(bs) = this.buf.get() else { return Poll::Ready(Ok(())); }; ready!(this.sink.poll_ready_unpin(cx)).map_err(format_std_io_error)?; this.sink .start_send_unpin(Buffer::from(bs)) .map_err(format_std_io_error)?; this.buf.clean(); } } fn poll_close(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { let this = self.get_mut(); loop { // Make sure buf has been frozen. this.buf.freeze(); let Some(bs) = this.buf.get() else { return this.sink.poll_close_unpin(cx).map_err(format_std_io_error); }; ready!(this.sink.poll_ready_unpin(cx)).map_err(format_std_io_error)?; this.sink .start_send_unpin(Buffer::from(bs)) .map_err(format_std_io_error)?; this.buf.clean(); } } } #[cfg(test)] mod tests { use std::sync::Arc; use super::*; use crate::raw::MaybeSend; #[tokio::test] async fn test_trait() { let op = Operator::via_iter(Scheme::Memory, []).unwrap(); let acc = op.into_inner(); let ctx = Arc::new(WriteContext::new( acc, "test".to_string(), OpWrite::new(), OpWriter::new().with_chunk(1), )); let write_gen = WriteGenerator::create(ctx).await.unwrap(); let v = FuturesAsyncWriter::new(write_gen); let _: Box = Box::new(v); } } opendal-0.52.0/src/types/write/futures_bytes_sink.rs000064400000000000000000000060451046102023000207210ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::pin::Pin; use std::task::Context; use std::task::Poll; use bytes::Bytes; use futures::SinkExt; use crate::raw::*; use crate::*; /// FuturesBytesSink is the adapter of [`futures::Sink`] generated by [`Writer::into_bytes_sink`]. /// /// Users can use this adapter in cases where they need to use [`futures::Sink`] trait. FuturesBytesSink /// reuses the same concurrent and chunk settings from [`Writer`]. /// /// FuturesBytesSink also implements [`Unpin`], [`Send`] and [`Sync`]. pub struct FuturesBytesSink { sink: BufferSink, } impl FuturesBytesSink { /// Create a new sink from a [`oio::Writer`]. #[inline] pub(crate) fn new(w: WriteGenerator) -> Self { FuturesBytesSink { sink: BufferSink::new(w), } } } impl futures::Sink for FuturesBytesSink { type Error = std::io::Error; fn poll_ready( mut self: Pin<&mut Self>, cx: &mut Context<'_>, ) -> Poll> { self.sink.poll_ready_unpin(cx).map_err(format_std_io_error) } fn start_send(mut self: Pin<&mut Self>, item: Bytes) -> Result<(), std::io::Error> { self.sink .start_send_unpin(Buffer::from(item)) .map_err(format_std_io_error) } fn poll_flush( mut self: Pin<&mut Self>, cx: &mut Context<'_>, ) -> Poll> { self.sink.poll_flush_unpin(cx).map_err(format_std_io_error) } fn poll_close( mut self: Pin<&mut Self>, cx: &mut Context<'_>, ) -> Poll> { self.sink.poll_close_unpin(cx).map_err(format_std_io_error) } } #[cfg(test)] mod tests { use std::sync::Arc; use super::*; use crate::raw::MaybeSend; #[tokio::test] async fn test_trait() { let op = Operator::via_iter(Scheme::Memory, []).unwrap(); let acc = op.into_inner(); let ctx = Arc::new(WriteContext::new( acc, "test".to_string(), OpWrite::new(), OpWriter::new().with_chunk(1), )); let write_gen = WriteGenerator::create(ctx).await.unwrap(); let v = FuturesBytesSink::new(write_gen); let _: Box = Box::new(v); } } opendal-0.52.0/src/types/write/mod.rs000064400000000000000000000020131046102023000155400ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod writer; pub use writer::Writer; mod buffer_sink; pub use buffer_sink::BufferSink; mod futures_async_writer; pub use futures_async_writer::FuturesAsyncWriter; mod futures_bytes_sink; pub use futures_bytes_sink::FuturesBytesSink; opendal-0.52.0/src/types/write/writer.rs000064400000000000000000000243071046102023000163070ustar 00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use bytes::Buf; use crate::raw::*; use crate::*; /// Writer is designed to write data into given path in an asynchronous /// manner. /// /// ## Notes /// /// Please make sure either `close` or `abort` has been called before /// dropping the writer otherwise the data could be lost. /// /// ## Usage /// /// ### Write Multiple Chunks /// /// Some services support to write multiple chunks of data into given path. Services that doesn't /// support write multiple chunks will return [`ErrorKind::Unsupported`] error when calling `write` /// at the second time. /// /// ``` /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> Result<()> { /// let mut w = op.writer("path/to/file").await?; /// w.write(vec![1; 1024]).await?; /// w.write(vec![2; 1024]).await?; /// w.close().await?; /// Ok(()) /// } /// ``` /// /// ### Write like `Sink` /// /// ``` /// use anyhow::Result; /// use futures::SinkExt; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let mut w = op.writer("path/to/file").await?.into_bytes_sink(); /// w.send(vec![1; 1024].into()).await?; /// w.send(vec![2; 1024].into()).await?; /// w.close().await?; /// Ok(()) /// } /// ``` /// /// ### Write like `AsyncWrite` /// /// ``` /// use anyhow::Result; /// use futures::AsyncWriteExt; /// use opendal::Operator; /// /// async fn test(op: Operator) -> Result<()> { /// let mut w = op.writer("path/to/file").await?.into_futures_async_write(); /// w.write(&vec![1; 1024]).await?; /// w.write(&vec![2; 1024]).await?; /// w.close().await?; /// Ok(()) /// } /// ``` /// /// ### Write with append enabled /// /// Writer also supports to write with append enabled. This is useful when users want to append /// some data to the end of the file. /// /// - If file doesn't exist, it will be created and just like calling `write`. /// - If file exists, data will be appended to the end of the file. /// /// Possible Errors: /// /// - Some services store normal file and appendable file in different way. Trying to append /// on non-appendable file could return [`ErrorKind::ConditionNotMatch`] error. /// - Services that doesn't support append will return [`ErrorKind::Unsupported`] error when /// creating writer with `append` enabled. pub struct Writer { /// Keep a reference to write context in writer. _ctx: Arc, inner: WriteGenerator, } impl Writer { /// Create a new writer from an `oio::Writer`. pub(crate) async fn new(ctx: WriteContext) -> Result { let ctx = Arc::new(ctx); let inner = WriteGenerator::create(ctx.clone()).await?; Ok(Self { _ctx: ctx, inner }) } /// Write [`Buffer`] into writer. /// /// This operation will write all data in given buffer into writer. /// /// ## Examples /// /// ``` /// use bytes::Bytes; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> Result<()> { /// let mut w = op.writer("hello.txt").await?; /// // Buffer can be created from continues bytes. /// w.write("hello, world").await?; /// // Buffer can also be created from non-continues bytes. /// w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]) /// .await?; /// /// // Make sure file has been written completely. /// w.close().await?; /// Ok(()) /// } /// ``` pub async fn write(&mut self, bs: impl Into) -> Result<()> { let mut bs = bs.into(); while !bs.is_empty() { let n = self.inner.write(bs.clone()).await?; bs.advance(n); } Ok(()) } /// Write [`bytes::Buf`] into inner writer. /// /// This operation will write all data in given buffer into writer. /// /// # TODO /// /// Optimize this function to avoid unnecessary copy. pub async fn write_from(&mut self, bs: impl Buf) -> Result<()> { let mut bs = bs; let bs = Buffer::from(bs.copy_to_bytes(bs.remaining())); self.write(bs).await } /// Abort the writer and clean up all written data. /// /// ## Notes /// /// Abort should only be called when the writer is not closed or /// aborted, otherwise an unexpected error could be returned. pub async fn abort(&mut self) -> Result<()> { self.inner.abort().await } /// Close the writer and make sure all data have been committed. /// /// ## Notes /// /// Close should only be called when the writer is not closed or /// aborted, otherwise an unexpected error could be returned. pub async fn close(&mut self) -> Result { self.inner.close().await } /// Convert writer into [`FuturesAsyncWriter`] which implements [`futures::AsyncWrite`], /// /// # Notes /// /// FuturesAsyncWriter is not a zero-cost abstraction. The underlying writer /// requires an owned [`Buffer`], which involves an extra copy operation. /// /// FuturesAsyncWriter is required to call `close()` to make sure all /// data have been written to the storage. /// /// # Examples /// /// ## Basic Usage /// /// ``` /// use std::io; /// /// use futures::io::AsyncWriteExt; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> io::Result<()> { /// let mut w = op.writer("hello.txt").await?.into_futures_async_write(); /// let bs = "Hello, World!".as_bytes(); /// w.write_all(bs).await?; /// w.close().await?; /// /// Ok(()) /// } /// ``` /// /// ## Concurrent Write /// /// ``` /// use std::io; /// /// use futures::io::AsyncWriteExt; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> io::Result<()> { /// let mut w = op /// .writer_with("hello.txt") /// .concurrent(8) /// .chunk(256) /// .await? /// .into_futures_async_write(); /// let bs = "Hello, World!".as_bytes(); /// w.write_all(bs).await?; /// w.close().await?; /// /// Ok(()) /// } /// ``` pub fn into_futures_async_write(self) -> FuturesAsyncWriter { FuturesAsyncWriter::new(self.inner) } /// Convert writer into [`FuturesBytesSink`] which implements [`futures::Sink`]. /// /// # Notes /// /// FuturesBytesSink is a zero-cost abstraction. The underlying writer /// will reuse the Bytes and won't perform any copy operation. /// /// # Examples /// /// ## Basic Usage /// /// ``` /// use std::io; /// /// use bytes::Bytes; /// use futures::SinkExt; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> io::Result<()> { /// let mut w = op.writer("hello.txt").await?.into_bytes_sink(); /// let bs = "Hello, World!".as_bytes(); /// w.send(Bytes::from(bs)).await?; /// w.close().await?; /// /// Ok(()) /// } /// ``` /// /// ## Concurrent Write /// /// ``` /// use std::io; /// /// use bytes::Bytes; /// use futures::SinkExt; /// use opendal::Operator; /// use opendal::Result; /// /// async fn test(op: Operator) -> io::Result<()> { /// let mut w = op /// .writer_with("hello.txt") /// .concurrent(8) /// .chunk(256) /// .await? /// .into_bytes_sink(); /// let bs = "Hello, World!".as_bytes(); /// w.send(Bytes::from(bs)).await?; /// w.close().await?; /// /// Ok(()) /// } /// ``` pub fn into_bytes_sink(self) -> FuturesBytesSink { FuturesBytesSink::new(self.inner) } } #[cfg(test)] mod tests { use bytes::Bytes; use rand::rngs::ThreadRng; use rand::Rng; use rand::RngCore; use crate::services; use crate::Operator; fn gen_random_bytes() -> Vec { let mut rng = ThreadRng::default(); // Generate size between 1B..16MB. let size = rng.gen_range(1..16 * 1024 * 1024); let mut content = vec![0; size]; rng.fill_bytes(&mut content); content } #[tokio::test] async fn test_writer_write() { let op = Operator::new(services::Memory::default()).unwrap().finish(); let path = "test_file"; let content = gen_random_bytes(); let mut writer = op.writer(path).await.unwrap(); writer .write(content.clone()) .await .expect("write must succeed"); writer.close().await.expect("close must succeed"); let buf = op.read(path).await.expect("read to end mut succeed"); assert_eq!(buf.to_bytes(), content); } #[tokio::test] async fn test_writer_write_from() { let op = Operator::new(services::Memory::default()).unwrap().finish(); let path = "test_file"; let content = gen_random_bytes(); let mut writer = op.writer(path).await.unwrap(); writer .write_from(Bytes::from(content.clone())) .await .expect("write must succeed"); writer.close().await.expect("close must succeed"); let buf = op.read(path).await.expect("read to end mut succeed"); assert_eq!(buf.to_bytes(), content); } }