syntect-4.6.0/.cargo_vcs_info.json0000644000000001120000000000100125260ustar { "git": { "sha1": "cadc2bd66b2be1bbf1931f387ea1018f8a9e0e89" } } syntect-4.6.0/.gitattributes000064400000000000000000000032430072674642500142510ustar 00000000000000# Sublime Text # tabs indentation, no trailing *.stTheme eol=lf whitespace=-tab-in-indent,trailing-space,tabwidth=4 # spaces indentation, no trailing *.sublime-color-scheme eol=lf whitespace=tab-in-indent,trailing-space,tabwidth=4 *.hidden-color-scheme eol=lf whitespace=tab-in-indent,trailing-space,tabwidth=4 *.sublime-settings eol=lf whitespace=tab-in-indent,trailing-space,tabwidth=4 *.sublime-syntax eol=lf whitespace=tab-in-indent,trailing-space,tabwidth=2 *.sublime-theme eol=lf whitespace=tab-in-indent,trailing-space,tabwidth=4 # TextMate # tabs indentation, no trailing *.tmLanguage eol=lf whitespace=-tab-in-indent,trailing-space,tabwidth=4 *.hidden-tmLanguage eol=lf whitespace=-tab-in-indent,trailing-space,tabwidth=4 *.tmPreferences eol=lf whitespace=-tab-in-indent,trailing-space,tabwidth=4 *.tmTheme eol=lf whitespace=-tab-in-indent,trailing-space,tabwidth=4 *.hidden-tmTheme eol=lf whitespace=-tab-in-indent,trailing-space,tabwidth=4 # syntect testdata/* linguist-vendored # General # spaces indentation, no trailing *.md eol=lf whitespace=tab-in-indent,trailing-space,tabwidth=4 # binary *.png binary *.tex diff=tex *.pdf binary diff=astextplain syntect-4.6.0/.github/workflows/CI.yml000064400000000000000000000025010072674642500157650ustar 00000000000000name: CI on: workflow_dispatch: push: branches: [ master ] pull_request: branches: [ master ] env: CARGO_TERM_COLOR: always jobs: build-and-test: name: Build and test runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 with: submodules: true - name: Build run: | cargo build - name: Run tests run: | cargo test --features metadata - name: Run tests with fancy run: | # Run these tests in release mode since they're slow as heck otherwise cargo test --features default-fancy --no-default-features --release - name: make stuff run: | make assets make syntest make syntest-fancy rm -Rf examples - name: Docs run: | # Only run doc build on stable until this is fixed: https://github.com/rust-lang/rust/issues/51661 cargo doc - name: Run tests without default features run: | # default features are required for examples to build - so remove them from sight. # Doc-tests may also use default features rm -Rf examples && cargo test --lib --no-default-features - name: Run Xi tests run: | # Test the build configuration that Xi uses cargo test --lib --no-default-features --features "assets dump-load-rs" syntect-4.6.0/.gitignore000064400000000000000000000002370072674642500133460ustar 00000000000000# macOS .DS_Store Thumbs.db # Windows [Dd]esktop.ini # cargo target/ Cargo.lock # IDEA .idea/ *.iml # Sublime Text *.sublime-workspace # VS Code .vscode/ syntect-4.6.0/.gitmodules000064400000000000000000000010600072674642500135260ustar 00000000000000[submodule "testdata/Packages"] branch = st3 # TODO: this line should be removed once st4 functionality has been implemented path = testdata/Packages url = https://github.com/sublimehq/Packages [submodule "testdata/InspiredGitHub.tmtheme"] path = testdata/InspiredGitHub.tmtheme url = https://github.com/sethlopezme/InspiredGitHub.tmtheme.git [submodule "testdata/Solarized"] path = testdata/Solarized url = https://github.com/braver/Solarized.git [submodule "testdata/spacegray"] path = testdata/spacegray url = https://github.com/kkga/spacegray.git syntect-4.6.0/CHANGELOG.md000064400000000000000000000332550072674642500131750ustar 00000000000000# Changelog ## [Version 4.6.0](https://github.com/trishume/syntect/compare/v4.5.0...v4.6.0) (2021-08-01) - Add `html::line_tokens_to_classed_spans` to also take a mutable ScopeStack, deprecate `tokens_to_classed_spans`, to avoid panics and incorrect highlighting. - Derive Hash for Color and Style - Add `find_unlinked_contexts` to `SyntaxSet` - Add `syntaxes` method to `SyntaxSetBuilder` - Bump `fancy-regex` to v0.7 and `yaml-rust` to v0.4.5 ## [Version 4.5.0](https://github.com/trishume/syntect/compare/v4.4.0...v4.5.0) (2020-12-09) - Added a new function for producing classed HTML which handles newlines correctly and deprecated old one. [#307](https://github.com/trishume/syntect/pull/307) ## [Version 4.4.0](https://github.com/trishume/syntect/compare/v4.3.0...v4.4.0) (2020-08-19) - Errors are now `Send + Sync + 'static` [#304](https://github.com/trishume/syntect/pull/304) ## [Version 4.3.0](https://github.com/trishume/syntect/compare/v4.2.0...v4.3.0) (2020-08-01) - Fixes unnecesary dependency of the `html` feature on the `assets` feature. [#300](https://github.com/trishume/syntect/pull/300) - Adds ability to add prefixes to `html` module CSS class names. [#296](https://github.com/trishume/syntect/pull/296) ## [Version 4.2.0](https://github.com/trishume/syntect/compare/v4.1.1...v4.2.0) (2020-04-20) - Updates to new versions of `onig` and `plist`. The new `onig` version doesn't require `bindgen` thus making compilation easier. [#293](https://github.com/trishume/syntect/pull/293) ## [Version 4.1.1](https://github.com/trishume/syntect/compare/v4.1.0...v4.1.1) (2020-04-20) - Properly handle backreferences in included contexts [#288](https://github.com/trishume/syntect/pull/288) ## [Version 4.1.0](https://github.com/trishume/syntect/compare/v4.0.0...v4.1.0) (2020-03-30) - Make sure errors implement `Send` [#285](https://github.com/trishume/syntect/pull/285) - Fix errors to not use the deprecated `description()` [#286](https://github.com/trishume/syntect/pull/286) Thanks @sharkdp for the bug fixes! Bumping second part of semver since `Send` is adding functionality (back). ## [Version 4.0.0](https://github.com/trishume/syntect/compare/v3.3.0...v4.0.0) (2020-03-29) ### Headline feature: pure-Rust `fancy-regex` engine option Users can now opt in to a pure-Rust regex engine using Cargo features, making compilation easier in general. People experiencing difficulty compiling for Windows and Wasm should try switching to `fancy-regex`. Note this currently approximately halves highlighting speed. See the Readme and [#270](https://github.com/trishume/syntect/pull/270) for details. Thanks to @robinst for implementing this! ### Other changes - Ability to generate CSS for a theme for use with classed HTML generation (won't always be correct) [#274](https://github.com/trishume/syntect/pull/274/files) - Don't generate empty spans in classed HTML [#276](https://github.com/trishume/syntect/pull/276) - Miscellaneous dependency bumps and cleanup ### Breaking changes and upgrading Upgrading should cause no errors for nearly all users. Users using more unusual APIs may have a small amount of tweaking to do. - If you use `default-features = false` you may need to update your features to choose a regex engine - A bunch of technically public APIs that I don't know if anyone uses changed due to the regex engine refactor, common uses shouldn't break ## [Version 3.3.0](https://github.com/trishume/syntect/compare/v3.2.1...v3.3.0) (2019-09-22) > Bug fixes and new utilities - Fixes multiple bugs - Add RangedHighlightIterator - Add `as_latex_escaped` util ## [Version 3.2.1](https://github.com/trishume/syntect/compare/v3.2.0...v3.2.1) (2019-08-10) - Bump onig dependency - inconsequential patches ## [Version 3.2.0](https://github.com/trishume/syntect/compare/v3.1.0...v3.2.0) (2019-03-09) - Actually make `tokens_to_classed_spans` public like intended ## [Version 3.1.0](https://github.com/trishume/syntect/compare/v3.0.2...v3.1.0) (2019-02-24) > Metadata and new classed HTML generation - Add support for loading metadata ([#223](https://github.com/trishume/syntect/pull/223) [#225](https://github.com/trishume/syntect/pull/225) [#230](https://github.com/trishume/syntect/pull/230)) - Improve support for generating classed HTML and fix a bug, old function is deprecated because it's impossible to use correctly ([#235](https://github.com/trishume/syntect/pull/235)) - Update `plist` to `v0.4` and `pretty_assertions` to `v0.6` ([#232](https://github.com/trishume/syntect/pull/232) [#236](https://github.com/trishume/syntect/pull/236)) ## [Version 3.0.2](https://github.com/trishume/syntect/compare/v3.0.1...v3.0.2) (2018-11-11) > Bug fixes - Fix application of multiple `with_prototype`s ([#220](https://github.com/trishume/syntect/pull/220), fixes [#160](https://github.com/trishume/syntect/issues/160), [#178](https://github.com/trishume/syntect/issues/178), ASP highlighting) - Fix prototype marking logic ([#221](https://github.com/trishume/syntect/pull/221), fixes [#219](https://github.com/trishume/syntect/issues/219)) ## [Version 3.0.1](https://github.com/trishume/syntect/compare/v3.0.0...v3.0.1) (2018-10-16) > Minor bug fixes - Fix a bug with syntaxes that used captures in lookarounds ([#176](https://github.com/trishume/syntect/issues/176) [#215](https://github.com/trishume/syntect/pull/215)) - Fix the precedence order of syntaxes to match Sublime ([#217](https://github.com/trishume/syntect/pull/217) [#216](https://github.com/trishume/syntect/pull/216)) ## [Version 3.0.0](https://github.com/trishume/syntect/compare/v2.1.0...v3.0.0) (2018-10-09) > Breaking changes and major new features This is a major release with multiple breaking API changes, although upgrading shouldn't be too difficult. It fixes bugs and comes with some nice new features. ### Breaking changes and upgrading - The `SyntaxSet` API has been revamped to use a builder and an arena of contexts. See [example usage](https://github.com/trishume/syntect/blob/51208d35a6d98c07468fbe044d5c6f37eb129205/examples/gendata.rs#L25-L28). - Many functions now need to be passed the `SyntaxSet` that goes with the rest of their arguments because of this new arena. - Filename added to `LoadingError::ParseSyntax` - Many functions in the `html` module now take the `newlines` version of syntaxes. - These methods have also been renamed, partially so that code that needs updating doesn't break without a compile error. - The HTML they output also treats newlines slightly differently and I think more correctly but uglier when you look at the HTML. #### Breaking rename upgrade guide - `SyntaxSet::add_syntax -> SyntaxSetBuilder::add` - `SyntaxSet::load_syntaxes -> SyntaxSetBuilder::add_from_folder` - `SyntaxSet::load_plain_text_syntax -> SyntaxSetBuilder::add_plain_text_syntax` - `html::highlighted_snippet_for_string -> html::highlighted_html_for_string`: also change to `newlines` `SyntaxSet` - `html::highlighted_snippet_for_file -> html::highlighted_html_for_file`: also change to `newlines` `SyntaxSet` - `html::styles_to_coloured_html -> html::styled_line_to_highlighted_html`: also change to `newlines` `SyntaxSet` - `html::start_coloured_html_snippet -> html::start_highlighted_html_snippet`: return type also changed ### Major changes and new features - Use arena for contexts ([#182](https://github.com/trishume/syntect/pull/182) [#186](https://github.com/trishume/syntect/pull/186) [#187](https://github.com/trishume/syntect/pull/187) [#190](https://github.com/trishume/syntect/pull/190) [#195](https://github.com/trishume/syntect/pull/195)): This makes the code cleaner, enables use of syntaxes from multiple threads, and prevents accidental misuse. - This involves a new `SyntaxSetBuilder` API for constructing new `SyntaxSet`s - See the revamped [parsyncat example](https://github.com/trishume/syntect/blob/51208d35a6d98c07468fbe044d5c6f37eb129205/examples/parsyncat.rs). - Encourage use of newlines ([#197](https://github.com/trishume/syntect/pull/197) [#207](https://github.com/trishume/syntect/pull/207) [#196](https://github.com/trishume/syntect/issues/196)): The `nonewlines` mode is often buggy so we made it easier to use the `newlines` mode. - Added a `LinesWithEndings` utility for iterating over the lines of a string with `\n` characters. - Reengineer the `html` module to use `newlines` syntaxes. - Add helpers for modifying highlighted lines ([#198](https://github.com/trishume/syntect/pull/198)): For use cases like highlighting a piece of text in a blog code snippet or debugger. This allows you to reach into the highlighted spans and add styles. - Check out `split_at` and `modify_range` in the `util` module. - New `ThemeSet::add_from_folder` function ([#200](https://github.com/trishume/syntect/pull/200)): For modifying existing theme sets. ### Bug Fixes - Improve nonewlines regex rewriting: [#212](https://github.com/trishume/syntect/pull/212) [#211](https://github.com/trishume/syntect/issues/211) - Reengineer theme application to match Sublime: [#209](https://github.com/trishume/syntect/pull/209) - Also mark contexts referenced by name as "no prototype" (same as ST): [#180](https://github.com/trishume/syntect/pull/180) - keep with_prototype when switching contexts with `set`: [#177](https://github.com/trishume/syntect/pull/177) [#166](https://github.com/trishume/syntect/pull/166) - Fix unused import warning: [#174](https://github.com/trishume/syntect/pull/174) - Ignore trailing dots in selectors: [#173](https://github.com/trishume/syntect/pull/173) - Fix `embed` to not include prototypes: [#172](https://github.com/trishume/syntect/pull/172) [#160](https://github.com/trishume/syntect/issues/160) ### Upgraded dependencies - plist: `0.2 -> 0.3` - regex: `0.2 -> 1.0` - onig: `3.2.1 -> 4.1` ## [Version 2.1.0](https://github.com/trishume/syntect/compare/v2.0.1...v2.1.0) (2018-05-31) > Regex checking and plain file names * Check regexes compile upon loading from YAML (There's technically a small breaking change here if you match on the previously unused regex error, but I don't think anyone does) * Can detect the correct syntax on full file names like `CMakeLists.txt` * Make `nonewlines` mode marginally less buggy (still prefer using `newlines` mode) * Better error types * Better examples and tests ## [Version 2.0.1](https://github.com/trishume/syntect/compare/v2.0.0...v2.0.1) (2018-04-28) > More robust parsing * Parsing now abandons a regex after reaching a recursion depth limit instead of taking forever * Loop detection better matches Sublime Text * Parsing is faster! * Dependency upgrades * Other minor tweaks Thanks to [@robinst](https://github.com/ronbinst) for the headline features of this release! ## [Version 2.0.0](https://github.com/trishume/syntect/compare/v1.8.2...v2.0.0) (2018-01-02) > Breaking Changes and New Stuff ### Breaking changes * The `static-onig` feature was removed, static linking is now the default * Font styles and color constants now use associated consts because of bitflags upgrade * `SyntaxDefinition::load_from_str` now has an extra parameter ### Other notable changes * Support for new `embed` syntax, see [#124](https://github.com/trishume/syntect/issues/124) * Updates to many dependencies * Updated dumps * More compact HTML output ## [Version 1.8.2](https://github.com/trishume/syntect/compare/v1.8.0...v1.8.2) (2017-11-11) > New Inspired GitHub and libonig ## [Version 1.8.0](https://github.com/trishume/syntect/compare/v1.7.3...v1.8.0) (2017-10-14) > Update bitflags & packages This release changes how the constants for `FontStyle` and `Color`, relying on the new associated consts feature in `Rust 1.20`. The old constants are still available but are deprecated and will be removed in `v2.0`. Packages were also updated to newer versions. ## [Version 1.7.3](https://github.com/trishume/syntect/compare/v1.7.2...v1.7.3) (2017-09-15) > Enable comparison of parse states Fixes comparisons of parse states so they are fast and don't recurse infinitely. Thanks [@raphlinus](https://github.com/raphlinus) ## [Version 1.7.2](https://github.com/trishume/syntect/compare/v1.7.0...v1.7.2) (2017-09-05) > Bug fixes and package updates * Fixes [#101](https://github.com/trishume/syntect/issues/101), which caused some syntaxes like PHP to behave incorrectly. * Updates Packages with new syntax versions * Adds new handy flags to the `syncat` example ## [Version 1.7.0](https://github.com/trishume/syntect/compare/v1.6.0...v1.7.0) (2017-06-30) > Pure Rust dump loading / creation features ## [Version 1.6.0](https://github.com/trishume/syntect/compare/v1.5.0...v1.6.0) (2017-06-21) > Helper methods and more theme attributes ## [Version 1.5.0](https://github.com/trishume/syntect/compare/v1.4.0...v1.5.0) (2017-05-31) > Highlighting stacks Small release, adds a convenience method for highlighting an entire stack, and derives some more things on `Scope`. ## [Version 1.4.0](https://github.com/trishume/syntect/compare/v1.3.0...v1.4.0) (2017-05-25) > Serde and optional parsing This release switches the dump format from `rustc-serialize` to `Serde`, anyone using custom dumps will have to update them. It also makes the parsing part of the library optional behind a feature flag, anyone not using the default feature flags probably will want to add the `parsing` flag. ## [Version 1.3.0](https://github.com/trishume/syntect/tree/v1.3.0) (2017-04-05) > Bug fixes, tests, updates and feature flags * Syntax tests: there is a new `syntest` example for running Sublime Text syntax tests * Bug fixes: there's a ton of bugs fixed in this release, mostly found via the syntax tests. These mostly affected certain syntaxes which pushed/set multiple contexts at once. * Updated packages: The Sublime packages have been updated to the latest version * Feature flags: there's now Cargo feature flags for disabling some parts of syntect if you don't want unnecessary binary and dependency bloat. syntect-4.6.0/Cargo.lock0000644000000506230000000000100105150ustar # This file is automatically @generated by Cargo. # It is not intended for manual editing. version = 3 [[package]] name = "adler32" version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5d2e7343e7fc9de883d1b0341e0b13970f764c14101234857d2ddafa1cb1cac2" [[package]] name = "aho-corasick" version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "58fb5e95d83b38284460a5fda7d6470aa0b8844d283a0b614b8535e880800d2d" dependencies = [ "memchr", ] [[package]] name = "ansi_term" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ee49baf6cb617b853aa8d93bf420db2383fab46d314482ca2803b40d5fde979b" dependencies = [ "winapi", ] [[package]] name = "atty" version = "0.2.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1803c647a3ec87095e7ae7acfca019e98de5ec9a7d01343f611cf3152ed71a90" dependencies = [ "libc", "winapi", ] [[package]] name = "autocfg" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d49d90015b3c36167a20fe2810c5cd875ad504b39cff3d4eae7977e6b7c1cb2" [[package]] name = "autocfg" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d" [[package]] name = "base64" version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3441f0f7b02788e948e47f457ca01f1d7e6d92c693bc132c22b087d3141c03ff" [[package]] name = "bincode" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5753e2a71534719bf3f4e57006c3a4f0d2c672a4b676eec84161f763eca87dbf" dependencies = [ "byteorder", "serde", ] [[package]] name = "bit-set" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e84c238982c4b1e1ee668d136c510c67a13465279c0cb367ea6baf6310620a80" dependencies = [ "bit-vec", ] [[package]] name = "bit-vec" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f59bbe95d4e52a6398ec21238d31577f2b28a9d86807f06ca59d191d8440d0bb" [[package]] name = "bitflags" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693" [[package]] name = "bstr" version = "0.2.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8d6c2c5b58ab920a4f5aeaaca34b4488074e8cc7596af94e6f8c6ff247c60245" dependencies = [ "lazy_static", "memchr", "regex-automata", "serde", ] [[package]] name = "byteorder" version = "1.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a7c3dd8985a7111efc5c80b44e23ecdd8c007de8ade3b96595387e812b957cf5" [[package]] name = "cast" version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4b9434b9a5aa1450faa3f9cb14ea0e8c53bb5d2b3c1bfd1ab4fc03e9f33fbfb0" dependencies = [ "rustc_version", ] [[package]] name = "cc" version = "1.0.48" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f52a465a666ca3d838ebbf08b241383421412fe7ebb463527bba275526d89f76" [[package]] name = "cfg-if" version = "0.1.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822" [[package]] name = "chrono" version = "0.4.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "80094f509cf8b5ae86a4966a39b3ff66cd7e2a3e594accec3743ff3fabeab5b2" dependencies = [ "num-integer", "num-traits", ] [[package]] name = "clap" version = "2.33.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5067f5bb2d80ef5d68b4c87db81601f0b75bca627bc2ef76b141d7b846a3c6d9" dependencies = [ "bitflags", "textwrap", "unicode-width", ] [[package]] name = "crc32fast" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba125de2af0df55319f41944744ad91c71113bf74a4646efff39afe1f6842db1" dependencies = [ "cfg-if", ] [[package]] name = "criterion" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "938703e165481c8d612ea3479ac8342e5615185db37765162e762ec3523e2fc6" dependencies = [ "atty", "cast", "clap", "criterion-plot", "csv", "itertools", "lazy_static", "num-traits", "rand_core", "rand_os", "rand_xoshiro", "rayon", "serde", "serde_derive", "serde_json", "tinytemplate", "walkdir", ] [[package]] name = "criterion-plot" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eccdc6ce8bbe352ca89025bee672aa6d24f4eb8c53e3a8b5d1bc58011da072a2" dependencies = [ "cast", "itertools", ] [[package]] name = "crossbeam-deque" version = "0.7.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3aa945d63861bfe624b55d153a39684da1e8c0bc8fba932f7ee3a3c16cea3ca" dependencies = [ "crossbeam-epoch", "crossbeam-utils", ] [[package]] name = "crossbeam-epoch" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5064ebdbf05ce3cb95e45c8b086f72263f4166b29b97f6baff7ef7fe047b55ac" dependencies = [ "autocfg 0.1.7", "cfg-if", "crossbeam-utils", "lazy_static", "memoffset", "scopeguard", ] [[package]] name = "crossbeam-queue" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c695eeca1e7173472a32221542ae469b3e9aac3a4fc81f7696bcad82029493db" dependencies = [ "cfg-if", "crossbeam-utils", ] [[package]] name = "crossbeam-utils" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ce446db02cdc3165b94ae73111e570793400d0794e46125cc4056c81cbb039f4" dependencies = [ "autocfg 0.1.7", "cfg-if", "lazy_static", ] [[package]] name = "csv" version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37519ccdfd73a75821cac9319d4fce15a81b9fcf75f951df5b9988aa3a0af87d" dependencies = [ "bstr", "csv-core", "itoa", "ryu", "serde", ] [[package]] name = "csv-core" version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b5cadb6b25c77aeff80ba701712494213f4a8418fcda2ee11b6560c3ad0bf4c" dependencies = [ "memchr", ] [[package]] name = "ctor" version = "0.1.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cd8ce37ad4184ab2ce004c33bf6379185d3b1c95801cab51026bd271bf68eedc" dependencies = [ "quote", "syn", ] [[package]] name = "difference" version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "524cbf6897b527295dff137cec09ecf3a05f4fddffd7dfcd1585403449e74198" [[package]] name = "either" version = "1.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bb1f6b1ce1c140482ea30ddd3335fc0024ac7ee112895426e0a629a6c20adfe3" [[package]] name = "fancy-regex" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9d6b8560a05112eb52f04b00e5d3790c0dd75d9d980eb8a122fb23b92a623ccf" dependencies = [ "bit-set", "regex", ] [[package]] name = "flate2" version = "1.0.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6bd6d6f4752952feb71363cffc9ebac9411b75b87c6ab6058c40c8900cf43c0f" dependencies = [ "cfg-if", "crc32fast", "libc", "miniz_oxide", ] [[package]] name = "fnv" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2fad85553e09a6f881f739c29f0b00b0f01357c743266d478b68951ce23285f3" [[package]] name = "getopts" version = "0.2.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "14dbbfd5c71d70241ecf9e6f13737f7b5ce823821063188d7e46c41d371eebd5" dependencies = [ "unicode-width", ] [[package]] name = "getrandom" version = "0.1.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7abc8dd8451921606d809ba32e95b6111925cd2906060d2dcc29c070220503eb" dependencies = [ "cfg-if", "libc", "wasi", ] [[package]] name = "hermit-abi" version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eff2656d88f158ce120947499e971d743c05dbcbed62e5bd2f38f1698bbc3772" dependencies = [ "libc", ] [[package]] name = "indexmap" version = "1.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b54058f0a6ff80b6803da8faf8997cde53872b38f4023728f6830b06cd3c0dc" dependencies = [ "autocfg 1.0.0", ] [[package]] name = "itertools" version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f56a2d0bc861f9165be4eb3442afd3c236d8a98afd426f65d92324ae1091a484" dependencies = [ "either", ] [[package]] name = "itoa" version = "0.4.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "501266b7edd0174f8530248f87f99c88fbe60ca4ef3dd486835b8d8d53136f7f" [[package]] name = "lazy_static" version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" [[package]] name = "lazycell" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b294d6fa9ee409a054354afc4352b0b9ef7ca222c69b8812cbea9e7d2bf3783f" [[package]] name = "libc" version = "0.2.66" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d515b1f41455adea1313a4a2ac8a8a477634fbae63cc6100e3aebb207ce61558" [[package]] name = "line-wrap" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f30344350a2a51da54c1d53be93fade8a237e545dbcc4bdbe635413f2117cab9" dependencies = [ "safemem", ] [[package]] name = "linked-hash-map" version = "0.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7fb9b38af92608140b86b693604b9ffcc5824240a484d1ecd4795bacb2fe88f3" [[package]] name = "memchr" version = "2.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "88579771288728879b57485cc7d6b07d648c9f0141eb955f8ab7f9d45394468e" dependencies = [ "libc", ] [[package]] name = "memoffset" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "75189eb85871ea5c2e2c15abbdd541185f63b408415e5051f5cac122d8c774b9" dependencies = [ "rustc_version", ] [[package]] name = "miniz_oxide" version = "0.3.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6f3f74f726ae935c3f514300cc6773a0c9492abc5e972d42ba0c0ebb88757625" dependencies = [ "adler32", ] [[package]] name = "num-integer" version = "0.1.42" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f6ea62e9d81a77cd3ee9a2a5b9b609447857f3d358704331e4ef39eb247fcba" dependencies = [ "autocfg 1.0.0", "num-traits", ] [[package]] name = "num-traits" version = "0.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c62be47e61d1842b9170f0fdeec8eba98e60e90e5446449a0545e5152acd7096" dependencies = [ "autocfg 1.0.0", ] [[package]] name = "num_cpus" version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76dac5ed2a876980778b8b85f75a71b6cbf0db0b1232ee12f826bccb00d09d72" dependencies = [ "hermit-abi", "libc", ] [[package]] name = "onig" version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bd91ccd8a02fce2f7e8a86655aec67bc6c171e6f8e704118a0e8c4b866a05a8a" dependencies = [ "bitflags", "lazy_static", "libc", "onig_sys", ] [[package]] name = "onig_sys" version = "69.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3814583fad89f3c60ae0701d80e87e1fd3028741723deda72d0d4a0ecf0cb0db" dependencies = [ "cc", "pkg-config", ] [[package]] name = "output_vt100" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "53cdc5b785b7a58c5aad8216b3dfa114df64b0b06ae6e1501cef91df2fbdf8f9" dependencies = [ "winapi", ] [[package]] name = "pkg-config" version = "0.3.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "05da548ad6865900e60eaba7f589cc0783590a92e940c26953ff81ddbab2d677" [[package]] name = "plist" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b336d94e8e4ce29bf15bba393164629764744c567e8ad306cc1fdd0119967fd" dependencies = [ "base64", "chrono", "indexmap", "line-wrap", "serde", "xml-rs", ] [[package]] name = "pretty_assertions" version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f81e1644e1b54f5a68959a29aa86cde704219254669da328ecfdf6a1f09d427" dependencies = [ "ansi_term", "ctor", "difference", "output_vt100", ] [[package]] name = "proc-macro2" version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0319972dcae462681daf4da1adeeaa066e3ebd29c69be96c6abb1259d2ee2bcc" dependencies = [ "unicode-xid", ] [[package]] name = "quote" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2bdc6c187c65bca4260c9011c9e3132efe4909da44726bad24cf7572ae338d7f" dependencies = [ "proc-macro2", ] [[package]] name = "rand_core" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "90bde5296fc891b0cef12a6d03ddccc162ce7b2aff54160af9338f8d40df6d19" dependencies = [ "getrandom", ] [[package]] name = "rand_os" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a788ae3edb696cfcba1c19bfd388cc4b8c21f8a408432b199c072825084da58a" dependencies = [ "getrandom", "rand_core", ] [[package]] name = "rand_xoshiro" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0e18c91676f670f6f0312764c759405f13afb98d5d73819840cf72a518487bff" dependencies = [ "rand_core", ] [[package]] name = "rayon" version = "1.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db6ce3297f9c85e16621bb8cca38a06779ffc31bb8184e1be4bed2be4678a098" dependencies = [ "crossbeam-deque", "either", "rayon-core", ] [[package]] name = "rayon-core" version = "1.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "08a89b46efaf957e52b18062fb2f4660f8b8a4dde1807ca002690868ef2c85a9" dependencies = [ "crossbeam-deque", "crossbeam-queue", "crossbeam-utils", "lazy_static", "num_cpus", ] [[package]] name = "regex" version = "1.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc220bd33bdce8f093101afe22a037b8eb0e5af33592e6a9caafff0d4cb81cbd" dependencies = [ "aho-corasick", "memchr", "regex-syntax", "thread_local", ] [[package]] name = "regex-automata" version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "92b73c2a1770c255c240eaa4ee600df1704a38dc3feaa6e949e7fcd4f8dc09f9" dependencies = [ "byteorder", ] [[package]] name = "regex-syntax" version = "0.6.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "11a7e20d1cce64ef2fed88b66d347f88bd9babb82845b2b858f3edbf59a4f716" [[package]] name = "rustc_version" version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "138e3e0acb6c9fb258b19b67cb8abd63c00679d2851805ea151465464fe9030a" dependencies = [ "semver", ] [[package]] name = "ryu" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bfa8506c1de11c9c4e4c38863ccbe02a305c8188e85a05a784c9e11e1c3910c8" [[package]] name = "safemem" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ef703b7cb59335eae2eb93ceb664c0eb7ea6bf567079d843e09420219668e072" [[package]] name = "same-file" version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "585e8ddcedc187886a30fa705c47985c3fa88d06624095856b36ca0b82ff4421" dependencies = [ "winapi-util", ] [[package]] name = "scopeguard" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b42e15e59b18a828bbf5c58ea01debb36b9b096346de35d941dcb89009f24a0d" [[package]] name = "semver" version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d7eb9ef2c18661902cc47e535f9bc51b78acd254da71d375c2f6720d9a40403" dependencies = [ "semver-parser", ] [[package]] name = "semver-parser" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "388a1df253eca08550bef6c72392cfe7c30914bf41df5269b68cbd6ff8f570a3" [[package]] name = "serde" version = "1.0.104" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "414115f25f818d7dfccec8ee535d76949ae78584fc4f79a6f45a904bf8ab4449" [[package]] name = "serde_derive" version = "1.0.104" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "128f9e303a5a29922045a830221b8f78ec74a5f544944f3d5984f8ec3895ef64" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "serde_json" version = "1.0.44" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "48c575e0cc52bdd09b47f330f646cf59afc586e9c4e3ccd6fc1f625b8ea1dad7" dependencies = [ "itoa", "ryu", "serde", ] [[package]] name = "syn" version = "1.0.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e4ff033220a41d1a57d8125eab57bf5263783dfdcc18688b1dacc6ce9651ef8" dependencies = [ "proc-macro2", "quote", "unicode-xid", ] [[package]] name = "syntect" version = "4.6.0" dependencies = [ "bincode", "bitflags", "criterion", "fancy-regex", "flate2", "fnv", "getopts", "lazy_static", "lazycell", "onig", "plist", "pretty_assertions", "rayon", "regex", "regex-syntax", "serde", "serde_derive", "serde_json", "walkdir", "yaml-rust", ] [[package]] name = "textwrap" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060" dependencies = [ "unicode-width", ] [[package]] name = "thread_local" version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c6b53e329000edc2b34dbe8545fd20e55a333362d0a321909685a19bd28c3f1b" dependencies = [ "lazy_static", ] [[package]] name = "tinytemplate" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "57a3c6667d3e65eb1bc3aed6fd14011c6cbc3a0665218ab7f5daf040b9ec371a" dependencies = [ "serde", "serde_json", ] [[package]] name = "unicode-width" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "caaa9d531767d1ff2150b9332433f32a24622147e5ebb1f26409d5da67afd479" [[package]] name = "unicode-xid" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "826e7639553986605ec5979c7dd957c7895e93eabed50ab2ffa7f6128a75097c" [[package]] name = "walkdir" version = "2.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9658c94fa8b940eab2250bd5a457f9c48b748420d71293b165c8cdbe2f55f71e" dependencies = [ "same-file", "winapi", "winapi-util", ] [[package]] name = "wasi" version = "0.9.0+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cccddf32554fecc6acb585f82a32a72e28b48f8c4c1883ddfeeeaa96f7d8e519" [[package]] name = "winapi" version = "0.3.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8093091eeb260906a183e6ae1abdba2ef5ef2257a21801128899c3fc699229c6" dependencies = [ "winapi-i686-pc-windows-gnu", "winapi-x86_64-pc-windows-gnu", ] [[package]] name = "winapi-i686-pc-windows-gnu" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" [[package]] name = "winapi-util" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "70ec6ce85bb158151cae5e5c87f95a8e97d2c0c4b001223f33a334e3ce5de178" dependencies = [ "winapi", ] [[package]] name = "winapi-x86_64-pc-windows-gnu" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "xml-rs" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "541b12c998c5b56aa2b4e6f18f03664eef9a4fd0a246a55594efae6cc2d964b5" [[package]] name = "yaml-rust" version = "0.4.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "56c1936c4cc7a1c9ab21a1ebb602eb942ba868cbd44a99cb7cdc5892335e1c85" dependencies = [ "linked-hash-map", ] syntect-4.6.0/Cargo.toml0000644000000054540000000000100105420ustar # THIS FILE IS AUTOMATICALLY GENERATED BY CARGO # # When uploading crates to the registry Cargo will automatically # "normalize" Cargo.toml files for maximal compatibility # with all versions of Cargo and also rewrite `path` dependencies # to registry (e.g., crates.io) dependencies # # If you believe there's an error in this file please file an # issue against the rust-lang/cargo repository. If you're # editing this file be aware that the upstream Cargo.toml # will likely look very different (and much more reasonable) [package] edition = "2018" name = "syntect" version = "4.6.0" authors = ["Tristan Hume "] exclude = ["testdata/*", "/scripts/*", "/Makefile", "/codecov.yml"] description = "library for high quality syntax highlighting and code intelligence using Sublime Text's grammars" documentation = "https://docs.rs/syntect" readme = "Readme.md" keywords = ["syntax", "highlighting", "highlighter", "colouring", "parsing"] categories = ["parser-implementations", "parsing", "text-processing"] license = "MIT" repository = "https://github.com/trishume/syntect" [lib] bench = false [[bench]] name = "highlighting" harness = false [[bench]] name = "loading" harness = false [[bench]] name = "parsing" harness = false [dependencies.bincode] version = "1.0" optional = true [dependencies.bitflags] version = "1.0.4" [dependencies.fancy-regex] version = "0.7" optional = true [dependencies.flate2] version = "1.0" optional = true default-features = false [dependencies.fnv] version = "1.0" optional = true [dependencies.lazy_static] version = "1.0" [dependencies.lazycell] version = "1.0" [dependencies.onig] version = "6.0" optional = true default-features = false [dependencies.plist] version = "1" [dependencies.regex-syntax] version = "0.6" optional = true [dependencies.serde] version = "1.0" [dependencies.serde_derive] version = "1.0" [dependencies.serde_json] version = "1.0" [dependencies.walkdir] version = "2.0" [dependencies.yaml-rust] version = "0.4.5" optional = true [dev-dependencies.criterion] version = "0.3" [dev-dependencies.getopts] version = "0.2" [dev-dependencies.pretty_assertions] version = "0.6" [dev-dependencies.rayon] version = "1.0.0" [dev-dependencies.regex] version = "1.0" [features] assets = [] default = ["default-onig"] default-fancy = ["parsing", "assets", "html", "yaml-load", "dump-load", "dump-create", "regex-fancy"] default-onig = ["parsing", "assets", "html", "yaml-load", "dump-load", "dump-create", "regex-onig"] dump-create = ["flate2/default", "bincode"] dump-create-rs = ["flate2/rust_backend", "bincode"] dump-load = ["flate2/default", "bincode"] dump-load-rs = ["flate2/rust_backend", "bincode"] html = ["parsing"] metadata = ["parsing"] parsing = ["regex-syntax", "fnv"] regex-fancy = ["fancy-regex"] regex-onig = ["onig"] yaml-load = ["yaml-rust", "parsing"] syntect-4.6.0/Cargo.toml.orig000064400000000000000000000055640072674642500142550ustar 00000000000000[package] name = "syntect" description = "library for high quality syntax highlighting and code intelligence using Sublime Text's grammars" documentation = "https://docs.rs/syntect" repository = "https://github.com/trishume/syntect" keywords = ["syntax", "highlighting", "highlighter", "colouring", "parsing"] categories = ["parser-implementations", "parsing", "text-processing"] readme = "Readme.md" license = "MIT" version = "4.6.0" # remember to update html_root_url authors = ["Tristan Hume "] edition = "2018" exclude = [ "testdata/*", "/scripts/*", "/Makefile", "/codecov.yml" ] [dependencies] yaml-rust = { version = "0.4.5", optional = true } onig = { version = "6.0", optional = true, default-features = false } fancy-regex = { version = "0.7", optional = true } walkdir = "2.0" regex-syntax = { version = "0.6", optional = true } lazy_static = "1.0" lazycell = "1.0" bitflags = "1.0.4" plist = "1" bincode = { version = "1.0", optional = true } flate2 = { version = "1.0", optional = true, default-features = false } fnv = { version = "1.0", optional = true } serde = "1.0" serde_derive = "1.0" serde_json = "1.0" [dev-dependencies] criterion = "0.3" rayon = "1.0.0" regex = "1.0" getopts = "0.2" pretty_assertions = "0.6" [features] # If you enable two dump loading features or two creation features the build # will fail, choose one of each. If you are using both creation and loading, # your binary will be smaller if you choose the same one for each. # Pure Rust dump loading, slower than regular `dump-load` dump-load-rs = ["flate2/rust_backend", "bincode"] # Dump loading using flate2, which depends on the miniz C library. dump-load = ["flate2/default", "bincode"] # Dump creation using flate2, which depends on the miniz C library. dump-create = ["flate2/default", "bincode"] # Pure Rust dump creation, worse compressor so produces larger dumps than dump-create dump-create-rs = ["flate2/rust_backend", "bincode"] regex-fancy = ["fancy-regex"] regex-onig = ["onig"] parsing = ["regex-syntax", "fnv"] # Support for .tmPreferenes metadata files (indentation, comment syntax, etc) metadata = ["parsing"] # The `assets` feature enables inclusion of the default theme and syntax packages. # For `assets` to do anything, it requires one of `dump-load-rs` or `dump-load` to be set. assets = [] html = ["parsing"] yaml-load = ["yaml-rust", "parsing"] default-onig = ["parsing", "assets", "html", "yaml-load", "dump-load", "dump-create", "regex-onig"] # In order to switch to the fancy-regex engine, disable default features then add the default-fancy feature default-fancy = ["parsing", "assets", "html", "yaml-load", "dump-load", "dump-create", "regex-fancy"] default = ["default-onig"] # [profile.release] # debug = true [lib] bench = false [[bench]] name = "highlighting" harness = false [[bench]] name = "loading" harness = false [[bench]] name = "parsing" harness = false syntect-4.6.0/DESIGN.md000064400000000000000000000123510072674642500126510ustar 00000000000000# Optimization/Design notes This is my scratch pad for optimization ideas. Some of this I will implement, some I have implemented, some are just speculative. # Scopes ## Representation ideas: - Normal arrays of strings - array of 32-bit or 64-bit atoms (maybe using Servo's atom library) - Atoms packed into one or two u64s - fast equality checking - potentially fast prefix checking - needs unsafe code ## Potential packings: - variable width atoms, either 7 bits and a tag bit for top 128 or 13 bits and 3 tagging bits for rest - can fit all but 33 of the scopes present - tagged pointer (taking advantage of alignment), either a pointer to a slow path, or the first 4 bits set then a packed representation, one of others mentioned - 6 10-bit atoms referencing unique things by position (see by-position stats below) - 5 11-bit atoms and one 8-bit one for the first atom (2^11 = 2048, 2^8 = 256), one remaining bit for tag marker ## Stats: - 7000 scopes referenced in sublime, 3537 unique ones, all stats after this are based on non-unique data - all but 33 scopes in default packages could fit in 64 with combination 8bit or 16bit atom encoding - there are only 1219 unique atoms in the default package set - the top 128 atoms make up ~90% of all unique atoms referenced in syntax files - there are 26 unique first atoms, 145 unique last atoms - every position (1st atom, 2nd atom, ...) has under 878 possibilities, only 2nd,3rd and 4th have >256 - 99.8% of scopes have 6 or fewer atoms, 97% have 5 or fewer, 70% have 4 or fewer - for unique scopes: {2=>81, 4=>1752, 3=>621, 5=>935, 7=>8, 6=>140} ----> 95% of uniques <= 6 - for non-unique scopes: {2=>125, 4=>3383, 3=>1505, 5=>1891, 7=>9, 6=>202} # Checking prefix operation: `fn extent_matched(potential_prefix: Scope, s: Scope) -> u8` idea: any differences are beyond the length of the prefix. figure this out by xor and then ctz/clz then a compare to the length (however that works). ```bash XXXXYYYY00000000 # prefix XXXXYYYYZZZZ0000 # testee 00000000ZZZZ0000 # = xored XXXXYYYYQQQQ0000 # non-prefix XXXXYYYYZZZZ0000 # testee 00000000GGGG0000 # = xored XXXXQQQQ00000000 # non-prefix XXXXYYYYZZZZ0000 # testee 0000BBBBZZZZ0000 # = xored ``` # Parsing * Problem: need to reduce number of regex search calls * Solution: cache better ## Stats ```bash # On stats branch $cargo run --release --example syncat testdata/jquery.js | grep cmiss | wc -l Running `target/release/examples/syncat testdata/jquery.js` 61266 $cargo run --release --example syncat testdata/jquery.js | grep ptoken | wc -l Compiling syntect v0.1.0 (file:///Users/tristan/Box/Dev/Projects/syntect) Running `target/release/examples/syncat testdata/jquery.js` 98714 $wc -l testdata/jquery.js 9210 testdata/jquery.js $cargo run --release --example syncat testdata/jquery.js | grep cclear | wc -l Compiling syntect v0.1.0 (file:///Users/tristan/Box/Dev/Projects/syntect) Running `target/release/examples/syncat testdata/jquery.js` 71302 $cargo run --release --example syncat testdata/jquery.js | grep freshcachetoken | wc -l Compiling syntect v0.1.0 (file:///Users/tristan/Box/Dev/Projects/syntect) Running `target/release/examples/syncat testdata/jquery.js` 80512 # On stats-2 branch $cargo run --example syncat testdata/jquery.js | grep cachehit | wc -l Running `target/debug/examples/syncat testdata/jquery.js` 527774 $cargo run --example syncat testdata/jquery.js | grep regsearch | wc -l Running `target/debug/examples/syncat testdata/jquery.js` 2862948 $cargo run --example syncat testdata/jquery.js | grep regmatch | wc -l Compiling syntect v0.6.0 (file:///Users/tristan/Box/Dev/Projects/syntect) Running `target/debug/examples/syncat testdata/jquery.js` 296127 $cargo run --example syncat testdata/jquery.js | grep leastmatch | wc -l Compiling syntect v0.6.0 (file:///Users/tristan/Box/Dev/Projects/syntect) Running `target/debug/examples/syncat testdata/jquery.js` 137842 # With search caching $cargo run --example syncat testdata/jquery.js | grep searchcached | wc -l Compiling syntect v0.6.0 (file:///Users/tristan/Box/Dev/Projects/syntect) Running `target/debug/examples/syncat testdata/jquery.js` 2440527 $cargo run --example syncat testdata/jquery.js | grep regsearch | wc -l Running `target/debug/examples/syncat testdata/jquery.js` 950195 ``` Average unique regexes per line is 87.58, average non-unique is regsearch/lines = 317 Ideally we should have only a couple fresh cache searches per line, not `~10` like the stats show (freshcachetoken/linecount). In a fantabulous world these stats mean a possible 10x speed improvement, but since caching does have a cost and we can't always cache it likely will be nice but not that high. ## Issues - Stack transitions always bust cache, even when for example JS just pushes another group - Doesn't cache actual matches, only if it matched or not ## Attacks - cache based on actual context, only search if it is a prototype we haven't searched before - hash maps based on casting RC ref to pointer and hashing? (there is a Hash impl for pointers) - for new searches, store matched regexes for context in BTreeMap like textmate - for subsequent tokens in same context, just pop off btreemap and re-search if before curpos - cache per Regex syntect-4.6.0/LICENSE.txt000064400000000000000000000021340072674642500131770ustar 00000000000000MIT License Copyright (c) 2017 Tristan Hume, Keith Hall, Google Inc and other contributors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. syntect-4.6.0/Readme.md000064400000000000000000000425640072674642500131060ustar 00000000000000# syntect [![Crates.io](https://img.shields.io/crates/v/syntect.svg?maxAge=2591000)](https://crates.io/crates/syntect) [![Documentation](https://docs.rs/syntect/badge.svg)](https://docs.rs/syntect) [![Crates.io](https://img.shields.io/crates/l/syntect.svg?maxAge=2591000)]() [![Build Status](https://github.com/trishume/syntect/actions/workflows/CI.yml/badge.svg)](https://github.com/trishume/syntect/actions) [![codecov](https://codecov.io/gh/trishume/syntect/branch/master/graph/badge.svg)](https://codecov.io/gh/trishume/syntect) `syntect` is a syntax highlighting library for Rust that uses [Sublime Text syntax definitions](http://www.sublimetext.com/docs/3/syntax.html#include-syntax). It aims to be a good solution for any Rust project that needs syntax highlighting, including deep integration with text editors written in Rust. It's used in production by at least two companies, and by [many open source projects](#projects-using-syntect). If you are writing a text editor (or something else needing highlighting) in Rust and this library doesn't fit your needs, I consider that a bug and you should file an issue or email me. I consider this project mostly complete, I still maintain it and review PRs, but it's not under heavy development. ## Important Links - API docs with examples: - [Changelogs and upgrade notes for past releases](https://github.com/trishume/syntect/releases) ## Getting Started `syntect` is [available on crates.io](https://crates.io/crates/syntect). You can install it by adding this line to your `Cargo.toml`: ```toml syntect = "4.4" ``` After that take a look at the [documentation](https://docs.rs/syntect) and the [examples](https://github.com/trishume/syntect/tree/master/examples). If you've cloned this repository, be sure to run ```bash git submodule update --init ``` to fetch all the required dependencies for running the tests. ## Features/Goals - [x] Work with many languages (accomplished through using existing grammar formats) - [x] Highlight super quickly, faster than nearly all text editors - [x] Include easy to use API for basic cases - [x] API allows use in fancy text editors with piece tables and incremental re-highlighting and the like. - [x] Expose internals of the parsing process so text editors can do things like cache parse states and use semantic info for code intelligence - [x] High quality highlighting, supporting things like heredocs and complex syntaxes (like Rust's). - [x] Include a compressed dump of all the default syntax definitions in the library binary so users don't have to manage a folder of syntaxes. - [x] Well documented, I've tried to add a useful documentation comment to everything that isn't utterly self explanatory. - [x] Built-in output to coloured HTML `
` tags or 24-bit colour ANSI terminal escape sequences.
- [x] Nearly complete compatibility with Sublime Text 3, including lots of edge cases. Passes nearly all of Sublime's syntax tests, see [issue 59](https://github.com/trishume/syntect/issues/59).
- [x] Load up quickly, currently in around 23ms but could potentially be even faster.

## Screenshots

There's currently an example program called `syncat` that prints one of the source files using hard-coded themes and syntaxes using 24-bit terminal escape sequences supported by many newer terminals.
These screenshots don't look as good as they could for two reasons:
first the sRGB colours aren't corrected properly, and second the Rust syntax definition uses some fancy labels that these themes don't have highlighting for.

![Nested languages](http://i.imgur.com/bByxb1E.png)
![Base 16 Ocean Dark](http://i.imgur.com/CwiPOwZ.png)
![Solarized Light](http://i.imgur.com/l3zcO4J.png)
![InspiredGithub](http://i.imgur.com/a7U1r2j.png)

## Example Code

Prints highlighted lines of a string to the terminal.
See the [easy](https://docs.rs/syntect/latest/syntect/easy/index.html) and [html](https://docs.rs/syntect/latest/syntect/html/index.html) module docs for more basic use case examples.

```rust
use syntect::easy::HighlightLines;
use syntect::parsing::SyntaxSet;
use syntect::highlighting::{ThemeSet, Style};
use syntect::util::{as_24_bit_terminal_escaped, LinesWithEndings};

// Load these once at the start of your program
let ps = SyntaxSet::load_defaults_newlines();
let ts = ThemeSet::load_defaults();

let syntax = ps.find_syntax_by_extension("rs").unwrap();
let mut h = HighlightLines::new(syntax, &ts.themes["base16-ocean.dark"]);
let s = "pub struct Wow { hi: u64 }\nfn blah() -> u64 {}";
for line in LinesWithEndings::from(s) {
    let ranges: Vec<(Style, &str)> = h.highlight(line, &ps);
    let escaped = as_24_bit_terminal_escaped(&ranges[..], true);
    println!("{}", escaped);
}
```

## Performance

Currently `syntect` is one of the faster syntax highlighting engines, but not the fastest. The following perf features are done:

- [x] Pre-link references between languages (e.g `