gitlab-labkit-0.34.0/0000755000004100000410000000000014535551620014342 5ustar www-datawww-datagitlab-labkit-0.34.0/.env.example.sh0000644000004100000410000000050514535551620017176 0ustar www-datawww-dataexport CI_PROJECT_ID=10947578 export CI_API_V4_URL="https://gitlab.com/api/v4" export CI_PROJECT_URL="https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby" # Don't keep secrets in plaintext files. Use a keyring or 1password to load # it instead and export it as an env var. token=$(load-your-token) export GITLAB_TOKEN=$token gitlab-labkit-0.34.0/.rspec0000644000004100000410000000004514535551620015456 0ustar www-datawww-data--color --require ./spec/spec_helper gitlab-labkit-0.34.0/README.md0000644000004100000410000000522014535551620015620 0ustar www-datawww-data# LabKit-Ruby 🔬🔬🔬🔬🔬 LabKit-Ruby is minimalist library to provide functionality for Ruby services at GitLab. LabKit-Ruby is the Ruby companion for [LabKit](https://gitlab.com/gitlab-org/labkit), a minimalist library to provide functionality for Go services at GitLab. LabKit-Ruby and LabKit are intended to provide similar functionality, but use the semantics of their respective languages, so are not intended to provide identical APIS. ## Documentation API Documentation is available at [the Rubydoc site](https://www.rubydoc.info/gems/gitlab-labkit/). ## Changelog The changelog is available via [**tagged release notes**](https://gitlab.com/gitlab-org/labkit-ruby/tags) ## Functionality LabKit-Ruby provides functionality in a number of areas: 1. `Labkit::Context` used for providing context information to log messages. 1. `Labkit::Correlation` For accessing the correlation id. (Generated and propagated by `Labkit::Context`) 1. `Labkit::FIPS` for checking for FIPS mode and using FIPS-compliant algorithms. 1. `Labkit::Logging` for sanitizing log messages. 1. `Labkit::Tracing` for handling and propagating distributed traces. ## Developing Anyone can contribute! ```console $ git clone git@gitlab.com:gitlab-org/labkit-ruby.git $ cd labkit-ruby $ bundle install $ # Autoformat code and auto-correct linters $ bundle exec rake fix $ # Run tests, linters $ bundle exec rake verify ``` Note that LabKit-Ruby uses the [`rufo`](https://github.com/ruby-formatter/rufo) for auto-formatting. Please run `bundle exec rake fix` to auto-format your code before pushing. Please also review the [development section of the LabKit (go) README](https://gitlab.com/gitlab-org/labkit#developing-labkit) for details of the LabKit architectural philosophy. To work on some of the scripts we use for releasing a new version, make sure to add a new `.env.sh`. ```console cp .env.example.sh .env.sh` ``` Inside `.env.sh`, add a personal acccess token for the `GITLAB_TOKEN` environment variable. Next source the file: ```console . .env.sh ``` ### Releasing a new version Releasing a new version can be done by pushing a new tag, or creating it from the [interface](https://gitlab.com/gitlab-org/labkit-ruby/-/tags). A new changelog will automatically be added to the release on Gitlab. The new version will automatically be published to `gitlab-labkit` on [rubygems](https://rubygems.org/gems/gitlab-labkit) when the pipeline for the tag completes. It might take a few minutes before the update is available. A gem called [`labkit-ruby`](https://rubygems.org/gems/labkit-ruby) is also published to RubyGems.org as a placeholder. The same bot that pushes this gem has access. gitlab-labkit-0.34.0/.gitlab/0000755000004100000410000000000014535551620015662 5ustar www-datawww-datagitlab-labkit-0.34.0/.gitlab/CODEOWNERS0000644000004100000410000000005114535551620017251 0ustar www-datawww-data* @andrewn @ayufan @reprazent @mkaeppler gitlab-labkit-0.34.0/.rubocop.yml0000644000004100000410000000220414535551620016612 0ustar www-datawww-datainherit_from: .rubocop_todo.yml inherit_gem: gitlab-styles: - rubocop-default.yml AllCops: TargetRubyVersion: 2.7 Style/HashSyntax: EnforcedStyle: no_mixed_keys Style/SymbolLiteral: Enabled: No Style/TrailingCommaInArguments: Enabled: No # Delegated to rufo Style/TrailingCommaInHashLiteral: Enabled: No # Delegated to rufo Style/FrozenStringLiteralComment: EnforcedStyle: always Style/TrailingCommaInArrayLiteral: EnforcedStyleForMultiline: comma Style/StringLiterals: EnforcedStyle: double_quotes Style/StringLiteralsInInterpolation: EnforcedStyle: double_quotes Style/Lambda: Enabled: false Layout/LineLength: Enabled: false Layout/SpaceInLambdaLiteral: Enabled: No Layout/SpaceInsideBlockBraces: Enabled: No Layout/FirstParameterIndentation: Enabled: No RSpec/ExampleWording: Enabled: false CustomTransform: be: is have: has not: does not IgnoredWords: [] Lint/HashCompareByIdentity: # (new in 0.93) Enabled: true Lint/RedundantSafeNavigation: # (new in 0.93) Enabled: true Style/ClassEqualityComparison: # (new in 0.93) Enabled: true CodeReuse/ActiveRecord: Enabled: false gitlab-labkit-0.34.0/.gitignore0000644000004100000410000000006114535551620016327 0ustar www-datawww-dataGemfile.lock *.gem node_modules .bundle /.env.sh gitlab-labkit-0.34.0/gitlab-labkit.gemspec0000644000004100000410000000423714535551620020423 0ustar www-datawww-data# frozen_string_literal: true lib = File.expand_path("lib", __dir__) $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) Gem::Specification.new do |spec| spec.name = "gitlab-labkit" spec.version = `git describe --tags`.chomp.gsub(/^v/, "") spec.authors = ["Andrew Newdigate"] spec.email = ["andrew@gitlab.com"] spec.summary = "Instrumentation for GitLab" spec.homepage = "https://gitlab.com/gitlab-org/labkit-ruby" spec.metadata = { "source_code_uri" => "https://gitlab.com/gitlab-org/labkit-ruby" } spec.license = "MIT" spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec|tools)/}) } spec.require_paths = ["lib"] spec.required_ruby_version = ">= 2.6.0" # Please maintain alphabetical order for dependencies spec.add_runtime_dependency "actionpack", ">= 5.0.0", "< 8.0.0" spec.add_runtime_dependency "activesupport", ">= 5.0.0", "< 8.0.0" spec.add_runtime_dependency "grpc", ">= 1.37" # Be sure to update the "grpc-tools" dev_dependency too spec.add_runtime_dependency "jaeger-client", "~> 1.1.0" spec.add_runtime_dependency "opentracing", "~> 0.4" spec.add_runtime_dependency "pg_query", "~> 4.2.3" spec.add_runtime_dependency "redis", ">3.0.0", "<6.0.0" # Please maintain alphabetical order for dev dependencies spec.add_development_dependency "excon", "~> 0.78.1" spec.add_development_dependency "faraday", "~> 1.0" spec.add_development_dependency "gitlab-dangerfiles", "~> 2.11.0" spec.add_development_dependency "gitlab-styles", "~> 6.2.0" spec.add_development_dependency "grpc-tools", ">= 1.37" spec.add_development_dependency "httparty", "~> 0.17.3" spec.add_development_dependency "httpclient", "~> 2.8.3" spec.add_development_dependency "pry", "~> 0.12" spec.add_development_dependency "rack", "~> 2.0" spec.add_development_dependency "rake", "~> 12.3" spec.add_development_dependency "rest-client", "~> 2.1.0" spec.add_development_dependency "rspec", "~> 3.12.0" spec.add_development_dependency "rspec-parameterized", "~> 1.0" spec.add_development_dependency "rufo", "0.9.0" spec.add_development_dependency "sidekiq", ">= 5.2", "< 7" spec.add_development_dependency "webrick", "~> 1.7.0" end gitlab-labkit-0.34.0/LICENSE0000644000004100000410000000207314535551620015351 0ustar www-datawww-dataThe MIT License (MIT) Copyright (c) 2016-2019 GitLab B.V. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. gitlab-labkit-0.34.0/.rufo0000644000004100000410000000010514535551620015312 0ustar www-datawww-dataalign_chained_calls true parens_in_def :dynamic trailing_commas true gitlab-labkit-0.34.0/.rubocop_todo.yml0000644000004100000410000001712014535551620017642 0ustar www-datawww-data# This configuration was generated by # `rubocop --auto-gen-config` # on 2021-04-18 00:00:00 UTC using RuboCop version 0.93.1. # The point is for the user to remove these configuration records # one by one as the offenses are removed from the code base. # Note that changes in the inspected code, or installation of new # versions of RuboCop, may require this file to be generated again. # Offense count: 51 # Cop supports --auto-correct. CodeReuse/ActiveRecord: Exclude: - 'spec/labkit/correlation/grpc/client_interceptor_spec.rb' - 'spec/labkit/correlation/grpc/server_interceptor_spec.rb' - 'spec/labkit/middleware/rack_spec.rb' - 'spec/labkit/middleware/sidekiq/client_spec.rb' - 'spec/labkit/middleware/sidekiq/context/server_spec.rb' - 'spec/labkit/middleware/sidekiq/server_spec.rb' - 'spec/labkit/middleware/sidekiq/tracing/client_spec.rb' - 'spec/labkit/middleware/sidekiq/tracing/server_spec.rb' - 'spec/labkit/tracing/factory_spec.rb' - 'spec/labkit/tracing/grpc/client_interceptor_spec.rb' - 'spec/labkit/tracing/grpc/server_interceptor_spec.rb' - 'spec/labkit/tracing_spec.rb' - 'spec/support/tracing/shared_examples.rb' # Offense count: 1 # Configuration parameters: Include. # Include: **/*.gemspec Gemspec/RequiredRubyVersion: Exclude: - 'gitlab-labkit.gemspec' # Offense count: 1 # Cop supports --auto-correct. # Configuration parameters: AllowMultipleStyles, EnforcedHashRocketStyle, EnforcedColonStyle, EnforcedLastArgumentHashStyle. # SupportedHashRocketStyles: key, separator, table # SupportedColonStyles: key, separator, table # SupportedLastArgumentHashStyles: always_inspect, always_ignore, ignore_implicit, ignore_explicit Layout/HashAlignment: Exclude: - 'spec/labkit/context_spec.rb' # Offense count: 1 Lint/BinaryOperatorWithIdenticalOperands: Exclude: - 'spec/labkit/tracing/redis/redis_interceptor_helper_spec.rb' # Offense count: 4 Lint/ConstantDefinitionInBlock: Exclude: - 'spec/labkit/middleware/sidekiq/client_spec.rb' - 'spec/labkit/middleware/sidekiq/context/client_spec.rb' - 'spec/labkit/middleware/sidekiq/context/server_spec.rb' - 'spec/labkit/middleware/sidekiq/server_spec.rb' # Offense count: 1 # Cop supports --auto-correct. # Configuration parameters: AllowedMethods. # AllowedMethods: instance_of?, kind_of?, is_a?, eql?, respond_to?, equal? Lint/RedundantSafeNavigation: Exclude: - 'lib/labkit/tracing.rb' # Offense count: 1 # Cop supports --auto-correct. # Configuration parameters: PreferredName. Naming/RescuedExceptionsVariableName: Exclude: - 'lib/labkit/logging/grpc/server_interceptor.rb' # Offense count: 2 Performance/MethodObjectAsBlock: Exclude: - 'lib/labkit/context.rb' # Offense count: 1 # Configuration parameters: Prefixes. # Prefixes: when, with, without RSpec/ContextWording: Exclude: - 'spec/labkit/middleware/sidekiq/server_spec.rb' # Offense count: 3 # Cop supports --auto-correct. # Configuration parameters: SkipBlocks, EnforcedStyle. # SupportedStyles: described_class, explicit RSpec/DescribedClass: Exclude: - 'spec/labkit/tracing_spec.rb' # Offense count: 2 # Configuration parameters: CustomTransform, IgnoreMethods, SpecSuffixOnly. RSpec/FilePath: Exclude: - 'spec/labkit/httpclient_publisher_spec.rb' - 'spec/logging/sanitizer_spec.rb' # Offense count: 3 # Cop supports --auto-correct. # Configuration parameters: EnforcedStyle. # SupportedStyles: single_line_only, single_statement_only, disallow RSpec/ImplicitSubject: Exclude: - 'spec/labkit/correlation/correlation_id_spec.rb' # Offense count: 4 RSpec/LeakyConstantDeclaration: Exclude: - 'spec/labkit/middleware/sidekiq/client_spec.rb' - 'spec/labkit/middleware/sidekiq/context/client_spec.rb' - 'spec/labkit/middleware/sidekiq/context/server_spec.rb' - 'spec/labkit/middleware/sidekiq/server_spec.rb' # Offense count: 2 # Cop supports --auto-correct. RSpec/LetBeforeExamples: Exclude: - 'spec/labkit/tracing_spec.rb' # Offense count: 18 # Configuration parameters: AllowSubject. RSpec/MultipleMemoizedHelpers: Max: 7 # Offense count: 15 # Cop supports --auto-correct. RSpec/ScatteredLet: Exclude: - 'spec/labkit/excon_publisher_spec.rb' - 'spec/labkit/httpclient_publisher_spec.rb' - 'spec/labkit/net_http_publisher_spec.rb' # Offense count: 1 # Cop supports --auto-correct. Rails/NegateInclude: Exclude: - 'spec/labkit/correlation/grpc/client_interceptor_spec.rb' # Offense count: 3 # Cop supports --auto-correct. # Configuration parameters: Include. # Include: **/Rakefile, **/*.rake Rails/RakeEnvironment: Exclude: - 'lib/capistrano/tasks/**/*.rake' - 'Rakefile' # Offense count: 1 # Configuration parameters: AllowedChars. Style/AsciiComments: Exclude: - 'spec/support/grpc_service/test_service_impl.rb' # Offense count: 1 # Cop supports --auto-correct. Style/CaseLikeIf: Exclude: - 'lib/labkit/excon_publisher.rb' # Offense count: 23 # Cop supports --auto-correct. Style/ExplicitBlockArgument: Exclude: - 'lib/labkit/correlation/grpc/server_interceptor.rb' - 'lib/labkit/logging/grpc/server_interceptor.rb' - 'lib/labkit/middleware/sidekiq/client.rb' - 'lib/labkit/middleware/sidekiq/context/server.rb' - 'lib/labkit/middleware/sidekiq/server.rb' - 'lib/labkit/middleware/sidekiq/tracing/server.rb' - 'lib/labkit/tracing/grpc/client_interceptor.rb' - 'lib/labkit/tracing/grpc/server_interceptor.rb' - 'lib/labkit/tracing/redis/redis_interceptor_helper.rb' # Offense count: 8 # Cop supports --auto-correct. # Configuration parameters: EnforcedStyle. # SupportedStyles: always, always_true, never Style/FrozenStringLiteralComment: Exclude: - 'spec/labkit/tracing/rails/active_record/sql_instrumenter_spec.rb' - 'spec/labkit/tracing/rails/active_support/cache_delete_instrumenter_spec.rb' - 'spec/labkit/tracing/rails/active_support/cache_fetch_hit_instrumenter_spec.rb' - 'spec/labkit/tracing/rails/active_support/cache_generate_instrumenter_spec.rb' - 'spec/labkit/tracing/rails/active_support/cache_read_instrumenter_spec.rb' - 'spec/labkit/tracing/rails/active_support/cache_write_instrumenter_spec.rb' - 'spec/support/grpc_service/test_pb.rb' - 'tools/update-changelog.rb' # Offense count: 2 # Cop supports --auto-correct. Style/GlobalStdStream: Exclude: - 'lib/labkit/tracing/jaeger_factory.rb' # Offense count: 3 # Cop supports --auto-correct. # Configuration parameters: EnforcedStyle, UseHashRocketsWithSymbolValues, PreferHashRocketsForNonAlnumEndingSymbols. # SupportedStyles: ruby19, hash_rockets, no_mixed_keys, ruby19_no_mixed_keys Style/HashSyntax: Exclude: - 'spec/labkit/excon_publisher_spec.rb' # Offense count: 16 # Cop supports --auto-correct. # Configuration parameters: PreferredDelimiters. Style/PercentLiteralDelimiters: Exclude: - 'spec/labkit/tracing/redis/redis_interceptor_helper_spec.rb' # Offense count: 2 # Cop supports --auto-correct. Style/RedundantBegin: Exclude: - 'Rakefile' - 'spec/labkit/net_http_publisher_spec.rb' # Offense count: 1 # Cop supports --auto-correct. Style/RedundantRegexpEscape: Exclude: - 'lib/labkit/logging/sanitizer.rb' # Offense count: 1 # Cop supports --auto-correct. Style/StringConcatenation: Exclude: - 'lib/labkit/tracing/redis/redis_interceptor_helper.rb' # Offense count: 28 # Cop supports --auto-correct. # Configuration parameters: EnforcedStyleForMultiline. # SupportedStylesForMultiline: comma, consistent_comma, no_comma Style/TrailingCommaInArrayLiteral: Exclude: - 'spec/labkit/excon_publisher_spec.rb' - 'spec/labkit/httpclient_publisher_spec.rb' - 'spec/labkit/net_http_publisher_spec.rb' gitlab-labkit-0.34.0/Rakefile0000644000004100000410000000230014535551620016002 0ustar www-datawww-data# frozen_string_literal: true require "bundler/gem_tasks" require "rufo" require "rspec/core/rake_task" RSpec::Core::RakeTask.new(:spec) require "rubocop/rake_task" RuboCop::RakeTask.new(:rubocop) do |task| task.options = %w[--parallel] end desc "Alias for `rake rufo:run`" task :format => ["rufo:run"] namespace :rufo do require "rufo" def rufo_command(*switches, rake_args) files_or_dirs = rake_args[:files_or_dirs] || "." args = switches + files_or_dirs.split(" ") Rufo::Command.run(args) end desc "Format Ruby code in current directory" task :run, [:files_or_dirs] do |_task, rake_args| rufo_command(rake_args) end desc "Check that no formatting changes are produced" task :check, [:files_or_dirs] do |_task, rake_args| rufo_command("--check", rake_args) end end desc "Generate test protobuf stubs" task :gen_test_proto do system "grpc_tools_ruby_protoc --ruby_out=. --grpc_out=. spec/support/grpc_service/test.proto" Rufo::Command.run(["spec/support/grpc_service/test_pb.rb", "spec/support/grpc_service/test_services_pb.rb"]) end task :fix => %w[rufo:run rubocop:auto_correct] task :verify => %w[spec rufo:check rubocop] task :default => %w[verify build] gitlab-labkit-0.34.0/lib/0000755000004100000410000000000014535551620015110 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/gitlab-labkit.rb0000644000004100000410000000343714535551620020152 0ustar www-datawww-data# rubocop:disable Naming/FileName # frozen_string_literal: true # LabKit is a module for handling cross-project # infrastructural concerns, partcularly related to # observability. module Labkit autoload :System, "labkit/system" autoload :Correlation, "labkit/correlation" autoload :Context, "labkit/context" autoload :FIPS, "labkit/fips" autoload :Tracing, "labkit/tracing" autoload :Logging, "labkit/logging" autoload :Middleware, "labkit/middleware" # Publishers to publish notifications whenever a HTTP reqeust is made. # A broadcasted notification's payload in topic "request.external_http" includes: # + method (String): "GET" # + code (String): "200" # This is the status code read directly from HTTP response # + duration (Float - seconds): 0.234 # + host (String): "gitlab.com" # + port (Integer): 80, # + path (String): "/gitlab-org/gitlab" # + scheme (String): "https" # + query (String): "field_a=1&field_b=2" # + fragment (String): "issue-number-1" # + proxy_host (String - Optional): "proxy.gitlab.com" # + proxy_port (Integer - Optional): 80 # + exception (Array - Optional): ["Net::ReadTimeout", "Net::ReadTimeout with #"] # + exception_object (Error Object - Optional): # # # Usage: # # ActiveSupport::Notifications.subscribe "request.external_http" do |name, started, finished, unique_id, data| # puts "#{name} | #{started} | #{finished} | #{unique_id} | #{data.inspect}" # end # EXTERNAL_HTTP_NOTIFICATION_TOPIC = "request.external_http" autoload :NetHttpPublisher, "labkit/net_http_publisher" autoload :ExconPublisher, "labkit/excon_publisher" autoload :HTTPClientPublisher, "labkit/httpclient_publisher" end # rubocop:enable Naming/FileName gitlab-labkit-0.34.0/lib/labkit/0000755000004100000410000000000014535551620016356 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/net_http_publisher.rb0000644000004100000410000000525614535551620022615 0ustar www-datawww-data# frozen_string_literal: true module Labkit ## # Prepend to Ruby's Net/HTTP standard HTTP library to publish a notification # whenever a HTTP request is triggered. Net::HTTP has different class methods # for each http method. Those methods are delegated to corresponding instance # methods. Eventually, `request` method is call to dispatch the HTTP request. # Therefore, a prepender that override `request` method covers all HTTP # calls. # # For more information: # https://github.com/ruby/ruby/blob/9b9cbbbc17bb5840581c7da37fd0feb0a7d4c1f3/lib/net/http.rb#L1510 # # Note: some use cases to take care of # - Create a request from input URI # - Create a request from input host, port, and path string # - Create a singular request and closes the connection immediately # - Create a persistent connection and perform multiple HTTP requests # - Notification payload must separate URI components # - Create a post request with a body # - Create a post request with form data # - Create a request with basic authentication # - Make a request via a proxy server # - Streaming module NetHttpPublisher @prepend_mutex = Mutex.new def self.labkit_prepend! @prepend_mutex.synchronize do return if @prepended require "net/http" Net::HTTP.prepend(self) @prepended = true end end def request(request, *args, &block) return super unless started? start_time = ::Labkit::System.monotonic_time ActiveSupport::Notifications.instrument ::Labkit::EXTERNAL_HTTP_NOTIFICATION_TOPIC, create_request_payload(request) do |payload| response = begin super ensure payload[:duration] = (::Labkit::System.monotonic_time - start_time).to_f end payload[:code] = response.code response end end private def create_request_payload(request) payload = { method: request.method, } if request.uri.nil? path_uri = URI(request.path) payload[:host] = address payload[:path] = path_uri.path payload[:port] = port payload[:scheme] = use_ssl? ? "https" : "http" payload[:query] = path_uri.query payload[:fragment] = path_uri.fragment else payload[:host] = request.uri.host payload[:path] = request.uri.path payload[:port] = request.uri.port payload[:scheme] = request.uri.scheme payload[:query] = request.uri.query payload[:fragment] = request.uri.fragment end if proxy? payload[:proxy_host] = proxy_address payload[:proxy_port] = proxy_port end payload end end end gitlab-labkit-0.34.0/lib/labkit/logging/0000755000004100000410000000000014535551620020004 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/logging/sanitizer.rb0000644000004100000410000000453014535551620022343 0ustar www-datawww-data# frozen_string_literal: true require "pg_query" module Labkit module Logging # Sanitizer provides log message sanitization, removing # confidential information from log messages class Sanitizer SCP_URL_REGEXP = %r{ (?:((?:[\-_.!~*()a-zA-Z\d;&=+$,]|%[a-fA-F\d]{2})+)(:(?:(?:[\-_.!~*()a-zA-Z\d;:&=+$,]|%[a-fA-F\d]{2})*))?@) (?# 1: username, 2: password) (?:((?:(?:[a-zA-Z0-9\-._])+|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|\[(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})|(?:(?:[a-fA-F\d]{1,4}:)*[a-fA-F\d]{1,4})?::(?:(?:[a-fA-F\d]{1,4}:)*(?:[a-fA-F\d]{1,4}|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}))?)\]))) (?# 3: host) : ((?:[\-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*(?:;(?:[\-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*)*(?:\/(?:[\-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*(?:;(?:[\-_.!~*'()a-zA-Z\d:@&=+$,]|%[a-fA-F\d]{2})*)*)*)? (?# 4: path) }x.freeze SCP_ANCHORED_URL_REGEXP = /^#{SCP_URL_REGEXP}$/x.freeze ALLOWED_SCHEMES = %w[http https ssh git].freeze URL_REGEXP = URI::DEFAULT_PARSER.make_regexp(ALLOWED_SCHEMES).freeze def self.sanitize_field(content) content = content.gsub(URL_REGEXP) { |url| mask_url(url) } content.gsub(SCP_URL_REGEXP) { |scp_url| mask_scp_url(scp_url) } end def self.sanitize_sql(sql) PgQuery.normalize(sql) rescue PgQuery::ParseError "" end def self.sql_fingerprint(normalized_sql) PgQuery.parse(normalized_sql)&.fingerprint rescue PgQuery::ParseError "" end # Ensures that URLS are sanitized to hide credentials def self.mask_url(url) url = url.to_s.strip p = URI::DEFAULT_PARSER.parse(url) p.password = "*****" if p.password.present? p.user = "*****" if p.user.present? p.to_s rescue URI::InvalidURIError "" end # Ensures that URLs of the form user:password@hostname:project.git are # sanitized to hide credentials def self.mask_scp_url(scp_url) scp_url = scp_url.to_s.strip m = SCP_ANCHORED_URL_REGEXP.match(scp_url) return "" unless m password = m[2] host = m[3] path = m[4] return "*****@#{host}:#{path}" unless password.present? "*****:*****@#{host}:#{path}" end end end end gitlab-labkit-0.34.0/lib/labkit/logging/json_logger.rb0000644000004100000410000000404314535551620022642 0ustar www-datawww-data# frozen_string_literal: true require "time" require "logger" require "json" module Labkit module Logging class JsonLogger < ::Logger # We should also reject log keys coming from Labkit::Context, but we cannot # do this without breaking clients currently. This is tracked in # https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/issues/35 RESERVED_LOG_KEYS = [ :environment, :host, :shard, :stage, :subcomponent, :tier, :type, ].freeze def self.log_level(fallback: ::Logger::DEBUG) ENV.fetch("GITLAB_LOG_LEVEL", fallback) end def self.exclude_context! @exclude_context = true self end def self.exclude_context? !!@exclude_context end def initialize(path, level: JsonLogger.log_level) super end def format_message(severity, timestamp, progname, message) data = default_attributes data[:severity] = severity data[:time] = timestamp.utc.iso8601(3) if self.class.exclude_context? data[Labkit::Correlation::CorrelationId::LOG_KEY] = Labkit::Correlation::CorrelationId.current_id else data.merge!(Labkit::Context.current.to_h) end case message when String data[:message] = message when Hash reject_reserved_log_keys!(message) data.merge!(message) end dump_json(data) << "\n" end private def default_attributes {} end def dump_json(data) JSON.generate(data) end def reject_reserved_log_keys!(hash) return if ENV["RAILS_ENV"] == "production" reserved_keys_used = hash.transform_keys(&:to_sym).slice(*RESERVED_LOG_KEYS) if reserved_keys_used.any? raise "The following log keys used are reserved: #{reserved_keys_used.keys.join(", ")}" + "\n\nUse key names that are descriptive e.g. by using a prefix." end end end end end gitlab-labkit-0.34.0/lib/labkit/logging/grpc/0000755000004100000410000000000014535551620020737 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/logging/grpc/server_interceptor.rb0000644000004100000410000000631114535551620025211 0ustar www-datawww-data# frozen_string_literal: true require "active_support/time_with_zone" require "active_support/values/time_zone" require "grpc" require "json" module Labkit module Logging module GRPC class ServerInterceptor < ::GRPC::ServerInterceptor include Labkit::Correlation::GRPC::GRPCCommon CODE_STRINGS = { ::GRPC::Core::StatusCodes::OK => "OK", ::GRPC::Core::StatusCodes::CANCELLED => "Canceled", ::GRPC::Core::StatusCodes::UNKNOWN => "Unknown", ::GRPC::Core::StatusCodes::INVALID_ARGUMENT => "InvalidArgument", ::GRPC::Core::StatusCodes::DEADLINE_EXCEEDED => "DeadlineExceeded", ::GRPC::Core::StatusCodes::NOT_FOUND => "NotFound", ::GRPC::Core::StatusCodes::ALREADY_EXISTS => "AlreadyExists", ::GRPC::Core::StatusCodes::PERMISSION_DENIED => "PermissionDenied", ::GRPC::Core::StatusCodes::RESOURCE_EXHAUSTED => "ResourceExhausted", ::GRPC::Core::StatusCodes::FAILED_PRECONDITION => "FailedPrecondition", ::GRPC::Core::StatusCodes::ABORTED => "Aborted", ::GRPC::Core::StatusCodes::OUT_OF_RANGE => "OutOfRange", ::GRPC::Core::StatusCodes::UNIMPLEMENTED => "Unimplemented", ::GRPC::Core::StatusCodes::INTERNAL => "Internal", ::GRPC::Core::StatusCodes::UNAVAILABLE => "Unavailable", ::GRPC::Core::StatusCodes::DATA_LOSS => "DataLoss", ::GRPC::Core::StatusCodes::UNAUTHENTICATED => "Unauthenticated", }.freeze def initialize(log_file, default_tags) @log_file = log_file @log_file.sync = true @default_tags = default_tags super() end def request_response(request: nil, call: nil, method: nil) log_request(method, call) { yield } end def server_streamer(request: nil, call: nil, method: nil) log_request(method, call) { yield } end def client_streamer(call: nil, method: nil) log_request(method, call) { yield } end def bidi_streamer(requests: nil, call: nil, method: nil) log_request(method, call) { yield } end private def log_request(method, _call) start = Time.now code = ::GRPC::Core::StatusCodes::OK yield rescue StandardError => ex code = ex.is_a?(::GRPC::BadStatus) ? ex.code : ::GRPC::Core::StatusCodes::UNKNOWN raise ensure service_name, method_name = rpc_split(method) message = @default_tags.merge( 'grpc.start_time': start.utc.rfc3339, 'grpc.time_ms': ((Time.now - start) * 1000.0).truncate(3), 'grpc.code': CODE_STRINGS.fetch(code, code.to_s), 'grpc.method': method_name, 'grpc.service': service_name, pid: Process.pid, correlation_id: Labkit::Correlation::CorrelationId.current_id.to_s, time: Time.now.utc.strftime("%Y-%m-%dT%H:%M:%S.%LZ"), ) if ex message["exception"] = ex.message message["exception_backtrace"] = ex.backtrace[0..5] if ex.backtrace end @log_file.puts(JSON.dump(message)) end end end end end gitlab-labkit-0.34.0/lib/labkit/logging/grpc.rb0000644000004100000410000000025414535551620021265 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Logging module GRPC autoload :ServerInterceptor, "labkit/logging/grpc/server_interceptor" end end end gitlab-labkit-0.34.0/lib/labkit/logging.rb0000644000004100000410000000044614535551620020335 0ustar www-datawww-data# frozen_string_literal: true module Labkit # Logging provides functionality for logging, such as # sanitization module Logging autoload :GRPC, "labkit/logging/grpc" autoload :Sanitizer, "labkit/logging/sanitizer" autoload :JsonLogger, "labkit/logging/json_logger" end end gitlab-labkit-0.34.0/lib/labkit/excon_publisher.rb0000644000004100000410000001116214535551620022075 0ustar www-datawww-data# frozen_string_literal: true module Labkit ## # A middleware for Excon HTTP library to publish a notification # whenever a HTTP request is triggered. # # Excon supports a middleware system that allows request/response # interception freely. Whenever a new Excon connection is created, a list of # default middlewares is injected. This list of middlewares can be altered # thanks to Excon.defaults accessor. ExconPublisher is inserted into this # list. It affects all connections created in future. There is a limitation # that this approach doesn't work if a user decides to override the default # middleware list. It is unlikely though, at least in the dependency tree of # GitLab. # # ExconPublisher instance is created once and shared between all Excon # connections later. Each connection may be triggered by different threads in # parallel. In such cases, a connection objects creates multiple sockets for # each thread. Therfore in the implementation of this middleware, the # instrumation payload for each connection is stored inside a thread-isolated # storage. # # For more information: # https://github.com/excon/excon/blob/81a0130537f2f8cd00d6daafb05d02d9a90dc9f7/lib/excon/middlewares/base.rb # https://github.com/excon/excon/blob/fa3ec51e9bb062a12846a1cfff09534e76c99f4b/lib/excon/constants.rb#L146 # https://github.com/excon/excon/blob/fa3ec51e9bb062a12846a1cfff09534e76c99f4b/lib/excon/connection.rb#L474 class ExconPublisher @prepend_mutex = Mutex.new def self.labkit_prepend! @prepend_mutex.synchronize do return if !defined?(Excon) || @prepended defaults = Excon.defaults defaults[:middlewares] << ExconPublisher @prepended = true end end def initialize(stack) @stack = stack @instrumenter = ActiveSupport::Notifications.instrumenter end def request_call(datum) payload = start_payload(datum) store_connection_payload(datum, payload) @instrumenter.start(::Labkit::EXTERNAL_HTTP_NOTIFICATION_TOPIC, payload) @stack.request_call(datum) end def response_call(datum) payload = fetch_connection_payload(datum) return @stack.response_call(datum) if payload.nil? calculate_duration(payload) payload[:code] = datum[:response][:status].to_s @instrumenter.finish(::Labkit::EXTERNAL_HTTP_NOTIFICATION_TOPIC, payload) @stack.response_call(datum) ensure remove_connection_payload(datum) end def error_call(datum) payload = fetch_connection_payload(datum) return @stack.error_call(datum) if payload.nil? calculate_duration(payload) if datum[:error].is_a?(Exception) payload[:exception] = [datum[:error].class.name, datum[:error].message] payload[:exception_object] = datum[:error] elsif datum[:error].is_a?(String) exception = StandardError.new(datum[:error]) payload[:exception] = [exception.class.name, exception.message] payload[:exception_object] = exception end @instrumenter.finish(::Labkit::EXTERNAL_HTTP_NOTIFICATION_TOPIC, payload) @stack.error_call(datum) ensure remove_connection_payload(datum) end private def start_payload(datum) payload = { method: datum[:method].to_s.upcase, host: nil_or_string(datum[:host]), path: nil_or_string(datum[:path]), port: nil_or_int(datum[:port]), scheme: nil_or_string(datum[:scheme]), query: generate_query_string(datum[:query]), start_time: ::Labkit::System.monotonic_time, } unless datum[:proxy].nil? payload[:proxy_host] = datum[:proxy][:host] payload[:proxy_port] = datum[:proxy][:port] end payload end def calculate_duration(payload) start_time = payload.delete(:start_time) || ::Labkit::System.monotonic_time payload[:duration] = (::Labkit::System.monotonic_time - start_time).to_f end def connection_payload Thread.current[:__labkit_http_excon_payload] ||= {} end def store_connection_payload(datum, payload) connection_payload[datum[:connection]] = payload end def fetch_connection_payload(datum) connection_payload.fetch(datum[:connection], nil) end def remove_connection_payload(datum) connection_payload.delete(datum[:connection]) end def nil_or_string(str) str&.to_s end def nil_or_int(int) int&.to_i rescue StandardError nil end def generate_query_string(query) if query.is_a?(Hash) query.to_query else nil_or_string(query) end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/0000755000004100000410000000000014535551620020005 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/tracing/abstract_instrumenter.rb0000644000004100000410000000240614535551620024756 0ustar www-datawww-data# frozen_string_literal: true require "opentracing" module Labkit module Tracing # https://edgeapi.rubyonrails.org/classes/ActiveSupport/Notifications/Instrumenter.html#method-c-new class AbstractInstrumenter def start(_name, _id, payload) scope = OpenTracing.start_active_span(span_name(payload)) scope_stack.push scope end def finish(_name, _id, payload) scope = scope_stack.pop span = scope.span Labkit::Tracing::TracingUtils.log_common_fields_on_span(span, span_name(payload)) # exception_object is the standard exception payload from ActiveSupport::Notifications # https://github.com/rails/rails/blob/v6.0.3.1/activesupport/lib/active_support/notifications/instrumenter.rb#L26 exception = payload[:exception_object].presence || payload[:exception].presence Labkit::Tracing::TracingUtils.log_exception_on_span(span, exception) tags(payload).each do |k, v| span.set_tag(k, v) end scope.close end def scope_stack Thread.current[:_labkit_trace_scope_stack] ||= [] end def span_name(_payload) raise "span_name not implemented" end def tags(_payload) {} end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/tracing_utils.rb0000644000004100000410000000531714535551620023207 0ustar www-datawww-data# frozen_string_literal: true require "active_support/core_ext/string/starts_ends_with" require "opentracing" module Labkit module Tracing # Internal methods for tracing. This is not part of the LabKit public API. # For internal usage only class TracingUtils # Convience method for running a block with a span def self.with_tracing(operation_name:, tags:, child_of: nil) scope = tracer.start_active_span(operation_name, child_of: child_of, tags: tags) span = scope.span log_common_fields_on_span(span, operation_name) begin yield span rescue StandardError => e log_exception_on_span(span, e) raise e ensure scope.close end end # Obtain a tracer instance def self.tracer OpenTracing.global_tracer end # Generate a span retrospectively def self.postnotify_span(operation_name, start_time, end_time, tags: nil, child_of: nil, exception: nil) span = OpenTracing.start_span(operation_name, start_time: start_time, tags: tags, child_of: child_of) log_common_fields_on_span(span, operation_name) log_exception_on_span(span, exception) if exception span.finish(end_time: end_time) end # Add common fields to a span def self.log_common_fields_on_span(span, operation_name) correlation_id = Labkit::Correlation::CorrelationId.current_id span.set_tag("correlation_id", correlation_id) if correlation_id span.log_kv(stack: caller.join('\n')) if include_stacktrace?(operation_name) end # Add exception logging to a span def self.log_exception_on_span(span, exception) return if exception.blank? span.set_tag("error", true) span.log_kv(**kv_tags_for_exception(exception)) end # Generate key-value tags for an exception def self.kv_tags_for_exception(exception) case exception when Exception { :"event" => "error", :"error.kind" => exception.class.to_s, :"message" => Labkit::Logging::Sanitizer.sanitize_field(exception.message), :"stack" => exception.backtrace&.join('\n'), } else { :"event" => "error", :"error.kind" => exception.class.to_s, :"error.object" => Labkit::Logging::Sanitizer.sanitize_field(exception.to_s) } end end def self.include_stacktrace?(operation_name) @include_stacktrace ||= Hash.new do |result, name| result[name] = Tracing.stacktrace_operations.any? { |stacktrace_operation| name.starts_with?(stacktrace_operation) } end @include_stacktrace[operation_name] end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/grpc_interceptor.rb0000644000004100000410000000027514535551620023707 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing # GRPCInterceptor is the deprecated name for GRPCClientInterceptor GRPCInterceptor = GRPC::ClientInterceptor end end gitlab-labkit-0.34.0/lib/labkit/tracing/rack_middleware.rb0000644000004100000410000000237414535551620023455 0ustar www-datawww-data# frozen_string_literal: true require "opentracing" require "action_dispatch" module Labkit module Tracing # RackMiddleware is a rack middleware component for # instrumenting incoming http requests into a Rails/Rack # server class RackMiddleware REQUEST_METHOD = "REQUEST_METHOD" def initialize(app) @app = app end def call(env) method = env[REQUEST_METHOD] context = TracingUtils.tracer.extract(OpenTracing::FORMAT_RACK, env) tags = { "component" => "rack", "span.kind" => "server", "http.method" => method, "http.url" => self.class.build_sanitized_url_from_env(env) } TracingUtils.with_tracing(operation_name: "http:#{method}", child_of: context, tags: tags) do |span| @app.call(env).tap { |status_code, _headers, _body| span.set_tag("http.status_code", status_code) } end end # Generate a sanitized (safe) request URL from the rack environment def self.build_sanitized_url_from_env(env) request = ::ActionDispatch::Request.new(env) original_url = request.original_url uri = URI.parse(original_url) uri.query = request.filtered_parameters.to_query if uri.query.present? uri.to_s end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/redis/0000755000004100000410000000000014535551620021113 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/tracing/redis/redis_interceptor.rb0000644000004100000410000000147114535551620025167 0ustar www-datawww-data# frozen_string_literal: true require "redis" module Labkit module Tracing module Redis # RedisInterceptor is an interceptor for Redis to add distributed tracing. # It should be installed using the `Labkit::Tracing.instrument` method module RedisInterceptor def call(command) RedisInterceptorHelper.call_with_tracing(command, self) do # Note: when used without any arguments super uses the arguments given to the subclass method. super end end def call_pipeline(pipeline) RedisInterceptorHelper.call_pipeline_with_tracing(pipeline, self) do # Note: when used without any arguments super uses the arguments given to the subclass method. super end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/redis/redis_interceptor_helper.rb0000644000004100000410000000705114535551620026526 0ustar www-datawww-data# frozen_string_literal: true require "redis" module Labkit module Tracing module Redis # RedisInterceptorHelper is a helper for the RedisInterceptor. This is not a public API class RedisInterceptorHelper # For optimization, compile this once MASK_REDIS_RE = /^([\w{}-]+(?:\W+[\w{}-]+(?:\W+[\w{}-]+)?)?)(.?)/.freeze def self.call_with_tracing(command, client) Labkit::Tracing::TracingUtils.with_tracing(operation_name: "redis.call", tags: tags_from_command(command, client)) do |_span| yield end end def self.call_pipeline_with_tracing(pipeline, client) Labkit::Tracing::TracingUtils.with_tracing(operation_name: "redis.call_pipeline", tags: tags_from_pipeline(pipeline, client)) do |_span| yield end end def self.common_tags_for_client(client) { "component" => "redis", "span.kind" => "client", "redis.scheme" => client.scheme, "redis.host" => client.host, "redis.port" => client.port, "redis.path" => client.path, } end def self.tags_from_command(command, client) tags = common_tags_for_client(client) tags["redis.command"] = command_serialized(command) tags end def self.command_serialized(command) return "" unless command.is_a?(Array) return "" if command.empty? command_name, *arguments = command command_name ||= "nil" info = [command_name] info << sanitize_argument_for_command(command_name, arguments.first) unless arguments.empty? info << "...#{arguments.size - 1} more value(s)" if arguments.size > 1 info.join(" ") end def self.tags_from_pipeline(pipeline, client) tags = common_tags_for_client(client) commands = pipeline.commands # Limit to the first 5 commands commands.first(5).each_with_index do |command, index| tags["redis.command.#{index}"] = command_serialized(command) end tags["redis.pipeline.commands.length"] = commands.length tags end # get_first_argument_for_command returns a masked value representing the first argument # from a redis command, taking care of certain sensitive commands def self.sanitize_argument_for_command(command_name, first_argument) return "*****" if command_is_sensitive?(command_name) return "nil" if first_argument.nil? return first_argument if first_argument.is_a?(Numeric) return "*****" unless first_argument.is_a?(String) mask_redis_arg(first_argument) end # Returns true if the arguments for the command should be masked def self.command_is_sensitive?(command_name) command_is?(command_name, :auth) || command_is?(command_name, :eval) end # Returns true if the command is equivalent to the command_symbol symbol def self.command_is?(command_name, command_symbol) if command_name.is_a?(Symbol) command_name == command_symbol else command_name.to_s.casecmp(command_symbol.to_s).zero? end end def self.mask_redis_arg(argument) return "" if argument.empty? matches = argument.match(MASK_REDIS_RE) matches[2].empty? ? matches[0] : matches[0] + "*****" end private_class_method :mask_redis_arg end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/external_http/0000755000004100000410000000000014535551620022666 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/tracing/external_http/request_instrumenter.rb0000644000004100000410000000176514535551620027533 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module ExternalHttp # For more information on the payloads: lib/labkit/net_http_publisher.rb class RequestInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(_payload) "external_http:request" end def tags(payload) # Duration is calculated by start and end time # Exception is already captured in lib/labkit/tracing/tracing_utils.rb tags = { "component" => "external_http", "method" => payload[:method], "code" => payload[:code], "host" => payload[:host], "port" => payload[:port], "path" => payload[:path], "scheme" => payload[:scheme], } unless payload[:proxy_host].nil? tags["proxy_host"] = payload[:proxy_host] tags["proxy_port"] = payload[:proxy_port] end tags end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/factory.rb0000644000004100000410000000323214535551620022001 0ustar www-datawww-data# frozen_string_literal: true require "cgi" module Labkit module Tracing # Factory provides tools for setting up and configuring the # distributed tracing system within the process, given the # tracing connection string class Factory OPENTRACING_SCHEME = "opentracing" def self.create_tracer(service_name, connection_string) return unless connection_string.present? begin opentracing_details = parse_connection_string(connection_string) driver_name = opentracing_details[:driver_name] case driver_name when "jaeger" JaegerFactory.create_tracer(service_name, opentracing_details[:options]) else raise "Unknown driver: #{driver_name}" end # Can't create the tracer? Warn and continue sans tracer rescue StandardError => e warn "Unable to instantiate tracer: #{e}" nil end end def self.parse_connection_string(connection_string) parsed = URI.parse(connection_string) raise "Invalid tracing connection string" unless valid_uri?(parsed) { driver_name: parsed.host, options: parse_query(parsed.query) } end private_class_method :parse_connection_string def self.parse_query(query) return {} unless query CGI.parse(query).symbolize_keys.transform_values(&:first) end private_class_method :parse_query def self.valid_uri?(uri) return false unless uri uri.scheme == OPENTRACING_SCHEME && uri.host.to_s =~ /^[a-z0-9_]+$/ && uri.path.empty? end private_class_method :valid_uri? end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/0000755000004100000410000000000014535551620021117 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_record.rb0000644000004100000410000000052414535551620024256 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveRecord autoload :SqlInstrumenter, "labkit/tracing/rails/active_record/sql_instrumenter" autoload :Subscriber, "labkit/tracing/rails/active_record/subscriber" COMPONENT_TAG = "ActiveRecord" end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/action_view/0000755000004100000410000000000014535551620023426 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/tracing/rails/action_view/render_template_instrumenter.rb0000644000004100000410000000136714535551620031753 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActionView # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class RenderTemplateInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(payload) identifier = ActionView.template_identifier(payload) if identifier.nil? "render_template" else "render_template:#{identifier}" end end def tags(payload) { "component" => COMPONENT_TAG, "template.id" => payload[:identifier], "template.layout" => payload[:layout] } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/action_view/subscriber.rb0000644000004100000410000000230114535551620026112 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActionView # ActionView bridges action view notifications to # the distributed tracing subsystem class Subscriber include Labkit::Tracing::TracingCommon RENDER_TEMPLATE_NOTIFICATION_TOPIC = "render_template.action_view" RENDER_COLLECTION_NOTIFICATION_TOPIC = "render_collection.action_view" RENDER_PARTIAL_NOTIFICATION_TOPIC = "render_partial.action_view" # Instruments Rails ActionView events for opentracing. # Returns a lambda, which, when called will unsubscribe from the notifications def self.instrument subscriptions = [ ::ActiveSupport::Notifications.subscribe(RENDER_TEMPLATE_NOTIFICATION_TOPIC, RenderTemplateInstrumenter.new), ::ActiveSupport::Notifications.subscribe(RENDER_COLLECTION_NOTIFICATION_TOPIC, RenderCollectionInstrumenter.new), ::ActiveSupport::Notifications.subscribe(RENDER_PARTIAL_NOTIFICATION_TOPIC, RenderPartialInstrumenter.new), ] create_unsubscriber subscriptions end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/action_view/render_partial_instrumenter.rb0000644000004100000410000000131514535551620031565 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActionView # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class RenderPartialInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(payload) identifier = ActionView.template_identifier(payload) if identifier.nil? "render_partial" else "render_partial:#{identifier}" end end def tags(payload) { "component" => COMPONENT_TAG, "template.id" => payload[:identifier] } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/action_view/render_collection_instrumenter.rb0000644000004100000410000000157114535551620032270 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActionView # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class RenderCollectionInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(payload) identifier = ActionView.template_identifier(payload) if identifier.nil? "render_collection" else "render_collection:#{identifier}" end end def tags(payload) { "component" => COMPONENT_TAG, "template.id" => payload[:identifier], "template.count" => payload[:count] || 0, "template.cache.hits" => payload[:cache_hits] || 0, } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_support.rb0000644000004100000410000000143014535551620024511 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveSupport autoload :CacheDeleteInstrumenter, "labkit/tracing/rails/active_support/cache_delete_instrumenter" autoload :CacheFetchHitInstrumenter, "labkit/tracing/rails/active_support/cache_fetch_hit_instrumenter" autoload :CacheGenerateInstrumenter, "labkit/tracing/rails/active_support/cache_generate_instrumenter" autoload :CacheReadInstrumenter, "labkit/tracing/rails/active_support/cache_read_instrumenter" autoload :CacheWriteInstrumenter, "labkit/tracing/rails/active_support/cache_write_instrumenter" autoload :Subscriber, "labkit/tracing/rails/active_support/subscriber" COMPONENT_TAG = "ActiveSupport" end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_support/0000755000004100000410000000000014535551620024166 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_support/cache_fetch_hit_instrumenter.rb0000644000004100000410000000103014535551620032404 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveSupport # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class CacheFetchHitInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(payload) "cache_fetch_hit" end def tags(payload) { "component" => COMPONENT_TAG, "cache.key" => payload[:key] } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_support/cache_write_instrumenter.rb0000644000004100000410000000102114535551620031601 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveSupport # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class CacheWriteInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(payload) "cache_write" end def tags(payload) { "component" => COMPONENT_TAG, "cache.key" => payload[:key] } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_support/cache_generate_instrumenter.rb0000644000004100000410000000102714535551620032247 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveSupport # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class CacheGenerateInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(payload) "cache_generate" end def tags(payload) { "component" => COMPONENT_TAG, "cache.key" => payload[:key] } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_support/cache_read_instrumenter.rb0000644000004100000410000000125014535551620031366 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveSupport # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class CacheReadInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(payload) "cache_read" end def tags(payload) { "component" => COMPONENT_TAG, "cache.key" => payload[:key], "cache.hit" => payload[:hit], "cache.super_operation" => payload[:super_operation], } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_support/cache_delete_instrumenter.rb0000644000004100000410000000102314535551620031713 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveSupport # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class CacheDeleteInstrumenter < Labkit::Tracing::AbstractInstrumenter def span_name(payload) "cache_delete" end def tags(payload) { "component" => COMPONENT_TAG, "cache.key" => payload[:key] } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_support/subscriber.rb0000644000004100000410000000267214535551620026665 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveSupport # ActiveSupport bridges action active support notifications to # the distributed tracing subsystem class Subscriber include Labkit::Tracing::TracingCommon CACHE_READ_TOPIC = "cache_read.active_support" CACHE_GENERATE_TOPIC = "cache_generate.active_support" CACHE_FETCH_HIT_TOPIC = "cache_fetch_hit.active_support" CACHE_WRITE_TOPIC = "cache_write.active_support" CACHE_DELETE_TOPIC = "cache_delete.active_support" # Instruments Rails ActiveSupport events for opentracing. # Returns a lambda, which, when called will unsubscribe from the notifications def self.instrument subscriptions = [ ::ActiveSupport::Notifications.subscribe(CACHE_READ_TOPIC, CacheReadInstrumenter.new), ::ActiveSupport::Notifications.subscribe(CACHE_GENERATE_TOPIC, CacheGenerateInstrumenter.new), ::ActiveSupport::Notifications.subscribe(CACHE_FETCH_HIT_TOPIC, CacheFetchHitInstrumenter.new), ::ActiveSupport::Notifications.subscribe(CACHE_WRITE_TOPIC, CacheWriteInstrumenter.new), ::ActiveSupport::Notifications.subscribe(CACHE_DELETE_TOPIC, CacheDeleteInstrumenter.new), ] create_unsubscriber subscriptions end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_record/0000755000004100000410000000000014535551620023730 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_record/sql_instrumenter.rb0000644000004100000410000000224314535551620027674 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveRecord # For more information on the payloads: https://guides.rubyonrails.org/active_support_instrumentation.html class SqlInstrumenter < Labkit::Tracing::AbstractInstrumenter OPERATION_NAME_PREFIX = "active_record:" DEFAULT_OPERATION_NAME = "sqlquery" def span_name(payload) OPERATION_NAME_PREFIX + (payload[:name].presence || DEFAULT_OPERATION_NAME) end def tags(payload) if Labkit::Tracing.sampled? && payload[:sql] sql = Labkit::Logging::Sanitizer.sanitize_sql(payload[:sql]) fingerprint = Labkit::Logging::Sanitizer.sql_fingerprint(sql) end { "component" => COMPONENT_TAG, "span.kind" => "client", "db.type" => "sql", "db.connection_id" => payload[:connection_id], "db.cached" => payload[:cached] || false, "db.statement" => sql, "db.statement_fingerprint" => fingerprint, } end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/active_record/subscriber.rb0000644000004100000410000000137314535551620026424 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActiveRecord # ActiveRecord bridges active record notifications to # the distributed tracing subsystem class Subscriber include Labkit::Tracing::TracingCommon ACTIVE_RECORD_NOTIFICATION_TOPIC = "sql.active_record" # Instruments Rails ActiveRecord events for opentracing. # Returns a lambda, which, when called will unsubscribe from the notifications def self.instrument subscription = ::ActiveSupport::Notifications.subscribe(ACTIVE_RECORD_NOTIFICATION_TOPIC, SqlInstrumenter.new) create_unsubscriber [subscription] end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails/action_view.rb0000644000004100000410000000263314535551620023757 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing module Rails module ActionView autoload :RenderCollectionInstrumenter, "labkit/tracing/rails/action_view/render_collection_instrumenter" autoload :RenderPartialInstrumenter, "labkit/tracing/rails/action_view/render_partial_instrumenter" autoload :RenderTemplateInstrumenter, "labkit/tracing/rails/action_view/render_template_instrumenter" autoload :Subscriber, "labkit/tracing/rails/action_view/subscriber" COMPONENT_TAG = "ActionView" # Returns identifier relative to Rails.root. Rails supports different template types and returns corresponding identifiers: # - Text template: the identifier is "text template" # - Html template: the identifier is "html template" # - Inline template: the identifier is "inline template" # - Raw template: the identifier is the file path of the template # Therefore, the amount of returned identifiers is static. def self.template_identifier(payload) return if !defined?(::Rails.root) || payload[:identifier].nil? # Rails.root returns a Pathname object, whose `to_s` methods returns an absolute path without ending "/" # Source: https://github.com/rails/rails/blob/v6.0.3.1/railties/lib/rails.rb#L64 payload[:identifier].sub("#{::Rails.root}/", "") end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/grpc/0000755000004100000410000000000014535551620020740 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/tracing/grpc/client_interceptor.rb0000644000004100000410000000275514535551620025172 0ustar www-datawww-data# frozen_string_literal: true # Disable the UnusedMethodArgument linter, since we need to declare the kwargs # in the methods, but we don't actually use them. require "opentracing" require "grpc" require "singleton" module Labkit module Tracing module GRPC # GRPCClientInterceptor is a client-side GRPC interceptor # for instrumenting GRPC calls with distributed tracing class ClientInterceptor < ::GRPC::ClientInterceptor include Singleton def request_response(request:, call:, method:, metadata:) wrap_with_tracing(method, "unary", metadata) { yield } end def client_streamer(requests:, call:, method:, metadata:) wrap_with_tracing(method, "client_stream", metadata) { yield } end def server_streamer(request:, call:, method:, metadata:) wrap_with_tracing(method, "server_stream", metadata) { yield } end def bidi_streamer(requests:, call:, method:, metadata:) wrap_with_tracing(method, "bidi_stream", metadata) { yield } end private def wrap_with_tracing(method, grpc_type, metadata) tags = { "component" => "grpc", "span.kind" => "client", "grpc.method" => method, "grpc.type" => grpc_type } TracingUtils.with_tracing(operation_name: "grpc:#{method}", tags: tags) do |span| OpenTracing.inject(span.context, OpenTracing::FORMAT_TEXT_MAP, metadata) yield end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/grpc/server_interceptor.rb0000644000004100000410000000336314535551620025216 0ustar www-datawww-data# frozen_string_literal: true # Disable the UnusedMethodArgument linter, since we need to declare the kwargs # in the methods, but we don't actually use them. require "opentracing" require "grpc" module Labkit module Tracing module GRPC # GRPCServerInterceptor is a server-side GRPC interceptor # for instrumenting GRPC calls with distributed tracing # in a GRPC Ruby server class ServerInterceptor < ::GRPC::ServerInterceptor include Labkit::Correlation::GRPC::GRPCCommon def request_response(request: nil, call: nil, method: nil) wrap_with_tracing(call, method, "unary") do yield end end def client_streamer(call: nil, method: nil) wrap_with_tracing(call, method, "client_stream") do yield end end def server_streamer(request: nil, call: nil, method: nil) wrap_with_tracing(call, method, "server_stream") do yield end end def bidi_streamer(requests: nil, call: nil, method: nil) wrap_with_tracing(call, method, "bidi_stream") do yield end end private def wrap_with_tracing(call, method, grpc_type) context = TracingUtils.tracer.extract(OpenTracing::FORMAT_TEXT_MAP, call.metadata) method_name = "/#{rpc_split(method).join("/")}" tags = { "component" => "grpc", "span.kind" => "server", "grpc.method" => method_name, "grpc.type" => grpc_type, } TracingUtils.with_tracing(operation_name: "grpc:#{method_name}", child_of: context, tags: tags) do |_span| yield end end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/tracing_common.rb0000644000004100000410000000107614535551620023335 0ustar www-datawww-data# frozen_string_literal: true require "active_support/concern" require "active_support/notifications" module Labkit module Tracing # TracingCommon is a mixin for providing instrumentation # functionality for the instrumentation classes based on # ActiveSupport::Notifications module TracingCommon extend ::ActiveSupport::Concern class_methods do def create_unsubscriber(subscriptions) -> { subscriptions.each { |subscriber| ::ActiveSupport::Notifications.unsubscribe(subscriber) } } end end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/redis.rb0000644000004100000410000000072514535551620021444 0ustar www-datawww-data# frozen_string_literal: true require "redis" module Labkit module Tracing # The Redis interceptor will intercept all calls to Redis and instrument them for distributed tracing module Redis autoload :RedisInterceptor, "labkit/tracing/redis/redis_interceptor" autoload :RedisInterceptorHelper, "labkit/tracing/redis/redis_interceptor_helper" def self.instrument ::Redis::Client.prepend RedisInterceptor end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/rails.rb0000644000004100000410000000101314535551620021437 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing # Rails provides classes for instrumenting Rails events module Rails autoload :ActionView, "labkit/tracing/rails/action_view" autoload :ActiveRecord, "labkit/tracing/rails/active_record" autoload :ActiveSupport, "labkit/tracing/rails/active_support" ActionViewSubscriber = ActionView::Subscriber ActiveRecordSubscriber = ActiveRecord::Subscriber ActiveSupportSubscriber = ActiveSupport::Subscriber end end end gitlab-labkit-0.34.0/lib/labkit/tracing/external_http.rb0000644000004100000410000000151614535551620023216 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing # Instrument external HTTP calls made by the HTTP client libraries. This # tracing instrumenter listens to the events broadcasted from the # publishers injected into the libraries whenever there is a request. module ExternalHttp include Labkit::Tracing::TracingCommon autoload :RequestInstrumenter, "labkit/tracing/external_http/request_instrumenter" def self.instrument Labkit::NetHttpPublisher.labkit_prepend! Labkit::ExconPublisher.labkit_prepend! Labkit::HTTPClientPublisher.labkit_prepend! subscriptions = [ ::ActiveSupport::Notifications.subscribe(::Labkit::EXTERNAL_HTTP_NOTIFICATION_TOPIC, RequestInstrumenter.new), ] create_unsubscriber subscriptions end end end end gitlab-labkit-0.34.0/lib/labkit/tracing/jaeger_factory.rb0000644000004100000410000000734214535551620023324 0ustar www-datawww-data# frozen_string_literal: true require "active_support" require "active_support/core_ext" require "jaeger/client" module Labkit module Tracing # JaegerFactory will configure Jaeger distributed tracing class JaegerFactory # When the probabilistic sampler is used, by default 0.1% of requests will be traced DEFAULT_PROBABILISTIC_RATE = 0.001 # The default port for the Jaeger agent UDP listener DEFAULT_UDP_PORT = 6831 # Reduce this from default of 10 seconds as the Ruby jaeger # client doesn't have overflow control, leading to very large # messages which fail to send over UDP (max packet = 64k) # Flush more often, with smaller packets FLUSH_INTERVAL = 5 def self.create_tracer(service_name, options) # The service_name parameter from GITLAB_TRACING takes precedence over the application one service_name = options[:service_name] if options[:service_name] # parse reporter headers as necessary headers = build_headers(options) kwargs = { service_name: service_name, sampler: get_sampler(options[:sampler], options[:sampler_param]), reporter: get_reporter(service_name, options[:http_endpoint], options[:udp_endpoint], headers), }.compact extra_params = options.except(:sampler, :sampler_param, :http_endpoint, :udp_endpoint, :strict_parsing, :debug) if extra_params.present? message = "jaeger tracer: invalid option: #{extra_params.keys.join(", ")}" raise message if options[:strict_parsing] warn message end Jaeger::Client.build(**kwargs) end def self.build_headers(options) return unless options&.key?(:http_endpoint) http_endpoint = options[:http_endpoint] parsed = URI.parse(http_endpoint) headers = {} # add basic auth header only when both user and password are setup correctly user = parsed.user password = parsed.password if user.present? && password.present? headers["Authorization"] = "Basic " + Base64.strict_encode64("#{user}:#{password}") end return headers end private_class_method :build_headers def self.get_sampler(sampler_type, sampler_param) case sampler_type when "probabilistic" sampler_rate = sampler_param ? sampler_param.to_f : DEFAULT_PROBABILISTIC_RATE Jaeger::Samplers::Probabilistic.new(rate: sampler_rate) when "const" const_value = sampler_param == "1" Jaeger::Samplers::Const.new(const_value) end end private_class_method :get_sampler def self.get_reporter(service_name, http_endpoint, udp_endpoint, headers) encoder = Jaeger::Encoders::ThriftEncoder.new(service_name: service_name) if http_endpoint.present? sender = get_http_sender(encoder, http_endpoint, headers) elsif udp_endpoint.present? sender = get_udp_sender(encoder, udp_endpoint) else return nil end Jaeger::Reporters::RemoteReporter.new(sender: sender, flush_interval: FLUSH_INTERVAL) end private_class_method :get_reporter def self.get_http_sender(encoder, address, headers) Jaeger::HttpSender.new(url: address, headers: headers, encoder: encoder, logger: Logger.new(STDOUT)) end private_class_method :get_http_sender def self.get_udp_sender(encoder, address) pair = address.split(":", 2) host = pair[0] port = pair[1] ? pair[1].to_i : DEFAULT_UDP_PORT Jaeger::UdpSender.new(host: host, port: port, encoder: encoder, logger: Logger.new(STDOUT)) end private_class_method :get_udp_sender end end end gitlab-labkit-0.34.0/lib/labkit/tracing/grpc.rb0000644000004100000410000000050214535551620021262 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Tracing # The GRPC module contains functionality for instrumenting GRPC calls module GRPC autoload :ClientInterceptor, "labkit/tracing/grpc/client_interceptor" autoload :ServerInterceptor, "labkit/tracing/grpc/server_interceptor" end end end gitlab-labkit-0.34.0/lib/labkit/context.rb0000644000004100000410000000637314535551620020400 0ustar www-datawww-data# frozen_string_literal: true require "securerandom" require "active_support/core_ext/module/delegation" require "active_support/core_ext/string/starts_ends_with" require "active_support/core_ext/string/inflections" module Labkit # A context can be used to provide structured information on what resources # GitLab is working on within a service. # # Values can be provided by passing a hash. If one of the values is a Proc # the proc will only be called when the value is actually needed. # # Multiple contexts can be nested, the nested context will inherit the values # from the closest outer one. # All contexts will have the same correlation id. # # Usage: # Labkit::Context.with_context(user: 'username', root_namespace: -> { get_root_namespace } do |context| # logger.info(context.to_h) # end # class Context LOG_KEY = "meta" CORRELATION_ID_KEY = "correlation_id" RAW_KEYS = [CORRELATION_ID_KEY].freeze class << self def with_context(attributes = {}) context = push(attributes) begin yield(context) ensure pop(context) end end def push(new_attributes = {}) new_context = current&.merge(new_attributes) || new(new_attributes) contexts.push(new_context) new_context end def pop(context) contexts.pop while contexts.include?(context) end def correlation_id contexts.last&.correlation_id end def current contexts.last end def log_key(key) key = key.to_s return key if RAW_KEYS.include?(key) return key if key.starts_with?("#{LOG_KEY}.") "#{LOG_KEY}.#{key}" end def known_log_keys @known_log_keys ||= (KNOWN_KEYS.map(&method(:log_key)) + RAW_KEYS).freeze end private def contexts Thread.current[:labkit_contexts] ||= [] end end def initialize(values = {}) @data = {} assign_attributes(values) end def merge(new_attributes) new_context = self.class.new(data.dup) new_context.assign_attributes(new_attributes) new_context end def to_h expand_data end def correlation_id data[CORRELATION_ID_KEY] end def get_attribute(attribute) raw = call_or_value(data[log_key(attribute)]) call_or_value(raw) end protected def assign_attributes(attributes) attributes = attributes.transform_keys(&method(:log_key)) data.merge!(attributes) # Remove keys that had their values set to `nil` in the new attributes data.keep_if { |_, value| valid_data?(value) } # Assign a correlation if it was missing in the first context or when # explicitly removed data[CORRELATION_ID_KEY] ||= new_id data end private delegate :log_key, to: :class attr_reader :data def call_or_value(value) value.respond_to?(:call) ? value.call : value end def expand_data data.transform_values do |value| value = call_or_value(value) value if valid_data?(value) end.compact end def new_id SecureRandom.hex end def valid_data?(value) value == false || value.present? end end end gitlab-labkit-0.34.0/lib/labkit/fips.rb0000644000004100000410000000264514535551620017653 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Digest module SHA2 def new(*args, &block) bitlen = args.first || 256 ::OpenSSL::Digest.const_get("SHA#{bitlen}").new end end end class FIPS OPENSSL_DIGESTS = %i[SHA1 SHA256 SHA384 SHA512].freeze class << self # Returns whether we should be running in FIPS mode or not # # @return [Boolean] def enabled? # Check if it set manually to false return false if %w[0 false no].include?(ENV["FIPS_MODE"]) # Otherwise allow it to be set manually via the env vars return true if %w[1 true yes].include?(ENV["FIPS_MODE"]) # Otherwise, attempt to auto-detect FIPS mode from OpenSSL return true if OpenSSL.fips_mode false end # Swap Ruby's Digest::SHAx implementations for OpenSSL::Digest::SHAx. def enable_fips_mode! require "digest" require "digest/sha1" require "digest/sha2" ::Digest::SHA2.singleton_class.prepend(Labkit::Digest::SHA2) OPENSSL_DIGESTS.each { |alg| use_openssl_digest(alg, alg) } end private def use_openssl_digest(ruby_algorithm, openssl_algorithm) ::Digest.send(:remove_const, ruby_algorithm) # rubocop:disable GitlabSecurity/PublicSend ::Digest.const_set(ruby_algorithm, OpenSSL::Digest.const_get(openssl_algorithm, false)) end end end end gitlab-labkit-0.34.0/lib/labkit/httpclient_publisher.rb0000644000004100000410000000414614535551620023143 0ustar www-datawww-data# frozen_string_literal: true module Labkit ## # Prepend to HTTPClient class to publish an ActiveSupport::Notifcation # whenever a HTTP request is triggered. # # Similar to Net::HTTP, this HTTP client redirects all calls to # HTTPClient#do_get_block. HTTPClient is prepended with HTTPClientPublisher. # Although HTTPClient supports request filter (a kind of middleware), its # support is strictly limited. The request and response passed into the # filter don't contain connection information. The response doesn't even # contain any link to the request object. It's impossible to fit this filter # mechanism into our subscribing model. # # For more information; # https://github.com/nahi/httpclient/blob/d3091b095a1b29f65f4531a70a8e581e75be035e/lib/httpclient.rb#L1233 module HTTPClientPublisher @prepend_mutex = Mutex.new def self.labkit_prepend! @prepend_mutex.synchronize do return if !defined?(HTTPClient) || @prepended HTTPClient.prepend(self) @prepended = true end end def do_get_block(req, proxy, conn, &block) start_time = ::Labkit::System.monotonic_time ActiveSupport::Notifications.instrument ::Labkit::EXTERNAL_HTTP_NOTIFICATION_TOPIC, create_request_payload(req, proxy) do |payload| response = begin super ensure payload[:duration] = (::Labkit::System.monotonic_time - start_time).to_f end payload[:code] = response.status_code.to_s response end end private def create_request_payload(request, proxy) http_header = request.http_header payload = { method: http_header.request_method, host: http_header.request_uri.host, path: http_header.request_uri.path, port: http_header.request_uri.port, scheme: http_header.request_uri.scheme, query: http_header.request_uri.query, fragment: http_header.request_uri.fragment, } unless proxy.nil? payload[:proxy_host] = proxy.host payload[:proxy_port] = proxy.port end payload end end end gitlab-labkit-0.34.0/lib/labkit/tracing.rb0000644000004100000410000000477114535551620020343 0ustar www-datawww-data# frozen_string_literal: true module Labkit # Tracing provides distributed tracing functionality module Tracing autoload :AbstractInstrumenter, "labkit/tracing/abstract_instrumenter" autoload :TracingCommon, "labkit/tracing/tracing_common" autoload :Factory, "labkit/tracing/factory" autoload :GRPC, "labkit/tracing/grpc" autoload :GRPCInterceptor, "labkit/tracing/grpc_interceptor" # Deprecated autoload :JaegerFactory, "labkit/tracing/jaeger_factory" autoload :RackMiddleware, "labkit/tracing/rack_middleware" autoload :Rails, "labkit/tracing/rails" autoload :Redis, "labkit/tracing/redis" autoload :ExternalHttp, "labkit/tracing/external_http" autoload :Sidekiq, "labkit/tracing/sidekiq" autoload :TracingUtils, "labkit/tracing/tracing_utils" # Tracing is only enabled when the `GITLAB_TRACING` env var is configured. def self.enabled? connection_string.present? end def self.connection_string ENV["GITLAB_TRACING"] end def self.tracing_url_template ENV["GITLAB_TRACING_URL"] end # Check if the current request is being traced. def self.sampled? context = OpenTracing.active_span&.context context&.respond_to?(:sampled?) && context&.sampled? end def self.stacktrace_operations @stacktrace_operations ||= Set.new(ENV["GITLAB_TRACING_INCLUDE_STACKTRACE"].to_s.split(",").map(&:strip)) end def self.tracing_url_enabled? enabled? && tracing_url_template.present? end # This will provide a link into the distributed tracing for the current trace, # if it has been captured. def self.tracing_url(service_name) return unless tracing_url_enabled? correlation_id = Labkit::Correlation::CorrelationId.current_id.to_s # Avoid using `format` since it can throw TypeErrors # which we want to avoid on unsanitised env var input tracing_url_template.to_s .gsub("{{ correlation_id }}", correlation_id) .gsub("{{ service }}", service_name) end # This will run a block with a span # @param operation_name [String] The operation name for the span # @param tags [Hash] Tags to assign to the span # @param child_of [SpanContext, Span] SpanContext that acts as a parent to # the newly-started span. If a span instance is provided, its # context is automatically substituted. def self.with_tracing(**kwargs, &block) TracingUtils.with_tracing(**kwargs, &block) end end end gitlab-labkit-0.34.0/lib/labkit/correlation/0000755000004100000410000000000014535551620020677 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/correlation/grpc/0000755000004100000410000000000014535551620021632 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/correlation/grpc/grpc_common.rb0000644000004100000410000000115614535551620024465 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Correlation module GRPC # This module is shared between the client and server interceptor middlewares. # It is not part of the public API module GRPCCommon CORRELATION_METADATA_KEY = "x-gitlab-correlation-id" def rpc_split(method) owner = method.owner method_name, = owner.rpc_descs.find do |k, _| ::GRPC::GenericService.underscore(k.to_s) == method.name.to_s end method_name ||= "(unknown)" [owner.service_name, method_name] end end end end end gitlab-labkit-0.34.0/lib/labkit/correlation/grpc/client_interceptor.rb0000644000004100000410000000256514535551620026063 0ustar www-datawww-data# frozen_string_literal: true # Disable the UnusedMethodArgument linter, since we need to declare the kwargs # in the methods, but we don't actually use them. require "grpc" require "singleton" module Labkit module Correlation module GRPC # ClientInterceptor is used to inject the correlation_id into the metadata # or a GRPC call for onward propagation to the server class ClientInterceptor < ::GRPC::ClientInterceptor include Labkit::Correlation::GRPC::GRPCCommon include Singleton def request_response(request:, call:, method:, metadata:) inject_correlation_id_into_metadata(metadata) yield end def client_streamer(requests:, call:, method:, metadata:) inject_correlation_id_into_metadata(metadata) yield end def server_streamer(request:, call:, method:, metadata:) inject_correlation_id_into_metadata(metadata) yield end def bidi_streamer(requests:, call:, method:, metadata:) inject_correlation_id_into_metadata(metadata) yield end private def inject_correlation_id_into_metadata(metadata, &block) metadata[CORRELATION_METADATA_KEY] = Labkit::Correlation::CorrelationId.current_id if Labkit::Correlation::CorrelationId.current_id end end end end end gitlab-labkit-0.34.0/lib/labkit/correlation/grpc/server_interceptor.rb0000644000004100000410000000265714535551620026115 0ustar www-datawww-data# frozen_string_literal: true # Disable the UnusedMethodArgument linter, since we need to declare the kwargs # in the methods, but we don't actually use them. require "grpc" module Labkit module Correlation module GRPC # ServerInterceptor is a server-side GRPC interceptor # for injecting GRPC calls with a correlation-id passed from # a GRPC client to the GRPC Ruby Service class ServerInterceptor < ::GRPC::ServerInterceptor include Labkit::Correlation::GRPC::GRPCCommon def request_response(request: nil, call: nil, method: nil) wrap_with_correlation_id(call) do yield end end def client_streamer(call: nil, method: nil) wrap_with_correlation_id(call) do yield end end def server_streamer(request: nil, call: nil, method: nil) wrap_with_correlation_id(call) do yield end end def bidi_streamer(requests: nil, call: nil, method: nil) wrap_with_correlation_id(call) do yield end end private def wrap_with_correlation_id(call, &block) correlation_id = call.metadata[CORRELATION_METADATA_KEY] correlation_id ||= Labkit::Correlation::CorrelationId.current_or_new_id Labkit::Correlation::CorrelationId.use_id(correlation_id, &block) end end end end end gitlab-labkit-0.34.0/lib/labkit/correlation/grpc.rb0000644000004100000410000000062014535551620022155 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Correlation # The GRPC module contains functionality for instrumenting GRPC calls module GRPC autoload :ClientInterceptor, "labkit/correlation/grpc/client_interceptor" autoload :GRPCCommon, "labkit/correlation/grpc/grpc_common" autoload :ServerInterceptor, "labkit/correlation/grpc/server_interceptor" end end end gitlab-labkit-0.34.0/lib/labkit/correlation/correlation_id.rb0000644000004100000410000000120514535551620024217 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Correlation # CorrelationId module provides access the Correlation-ID # of the current request module CorrelationId LOG_KEY = Labkit::Context::CORRELATION_ID_KEY class << self def use_id(correlation_id) Labkit::Context.with_context(LOG_KEY => correlation_id) do |context| yield(context.correlation_id) end end def current_id Labkit::Context.correlation_id end def current_or_new_id current_id || Labkit::Context.push.correlation_id end end end end end gitlab-labkit-0.34.0/lib/labkit/system.rb0000644000004100000410000000061114535551620020225 0ustar www-datawww-data# frozen_string_literal: true module Labkit # A helper class to store system-related methods used in metrics, tracing, and logging module System # Returns the current monotonic clock time as seconds with microseconds precision. # # Returns the time as a Float. def self.monotonic_time Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_second) end end end gitlab-labkit-0.34.0/lib/labkit/correlation.rb0000644000004100000410000000035714535551620021231 0ustar www-datawww-data# frozen_string_literal: true module Labkit # Correlation provides correlation functionality module Correlation autoload :GRPC, "labkit/correlation/grpc" autoload :CorrelationId, "labkit/correlation/correlation_id" end end gitlab-labkit-0.34.0/lib/labkit/middleware/0000755000004100000410000000000014535551620020473 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq.rb0000644000004100000410000000056314535551620022455 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Middleware # Middleware for sidekiq module Sidekiq autoload :Client, "labkit/middleware/sidekiq/client" autoload :Server, "labkit/middleware/sidekiq/server" autoload :Context, "labkit/middleware/sidekiq/context" autoload :Tracing, "labkit/middleware/sidekiq/tracing" end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/0000755000004100000410000000000014535551620022124 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/client.rb0000644000004100000410000000142114535551620023725 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Middleware module Sidekiq # This is a wrapper around all the sidekiq client-middleware in labkit # The only middleware that needs to be added to the chain in GitLab-rails # # It uses a new `Sidekiq::Middleware::Chain` to string multiple middlewares # together. class Client def self.chain @chain ||= ::Sidekiq::Middleware::Chain.new do |chain| chain.add Labkit::Middleware::Sidekiq::Context::Client chain.add Labkit::Middleware::Sidekiq::Tracing::Client if Labkit::Tracing.enabled? end end def call(*args) self.class.chain.invoke(*args) do yield end end end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/tracing/0000755000004100000410000000000014535551620023553 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/tracing/client.rb0000644000004100000410000000147314535551620025363 0ustar www-datawww-data# frozen_string_literal: true require "opentracing" module Labkit module Middleware module Sidekiq module Tracing # Client provides a sidekiq client middleware for # instrumenting distributed tracing calls made from the client # application class Client include SidekiqCommon SPAN_KIND = "client" def call(_worker_class, job, _queue, _redis_pool) Labkit::Tracing::TracingUtils.with_tracing(operation_name: "sidekiq:#{job_class(job)}", tags: tags_from_job(job, SPAN_KIND)) do |span| # Inject the details directly into the job Labkit::Tracing::TracingUtils.tracer.inject(span.context, OpenTracing::FORMAT_TEXT_MAP, job) yield end end end end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/tracing/sidekiq_common.rb0000644000004100000410000000211314535551620027076 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Middleware module Sidekiq module Tracing # SidekiqCommon is a mixin for the sidekiq middleware components module SidekiqCommon def job_class(job) # Active Job wrapping can be found at # https://github.com/rails/rails/blob/v6.0.3.1/activejob/lib/active_job/queue_adapters/sidekiq_adapter.rb job["wrapped"].presence || job["class"].presence || "undefined" end def wrapped?(job) job["wrapped"].present? end def tags_from_job(job, kind) tags = { "component" => "sidekiq", "span.kind" => kind, "sidekiq.wrapped" => wrapped?(job), "sidekiq.queue" => job["queue"], "sidekiq.jid" => job["jid"], "sidekiq.retry" => job["retry"].to_s, "sidekiq.args" => job["args"]&.join(", "), } tags["sidekiq.at"] = job["at"] if job["at"] tags end end end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/tracing/server.rb0000644000004100000410000000135514535551620025412 0ustar www-datawww-data# frozen_string_literal: true require "opentracing" module Labkit module Middleware module Sidekiq module Tracing # Server provides a sidekiq server middleware for # instrumenting distributed tracing calls when they are # executed by the Sidekiq server class Server include SidekiqCommon SPAN_KIND = "server" def call(_worker, job, _queue) context = Labkit::Tracing::TracingUtils.tracer.extract(OpenTracing::FORMAT_TEXT_MAP, job) Labkit::Tracing::TracingUtils.with_tracing(operation_name: "sidekiq:#{job_class(job)}", child_of: context, tags: tags_from_job(job, SPAN_KIND)) { |_span| yield } end end end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/context.rb0000644000004100000410000000055514535551620024142 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Middleware module Sidekiq # This module contains all the sidekiq middleware regarding application # context module Context autoload :Client, "labkit/middleware/sidekiq/context/client" autoload :Server, "labkit/middleware/sidekiq/context/server" end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/tracing.rb0000644000004100000410000000070614535551620024103 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Middleware module Sidekiq # Sidekiq provides classes for instrumenting Sidekiq client and server # functionality module Tracing autoload :Client, "labkit/middleware/sidekiq/tracing/client" autoload :Server, "labkit/middleware/sidekiq/tracing/server" autoload :SidekiqCommon, "labkit/middleware/sidekiq/tracing/sidekiq_common" end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/context/0000755000004100000410000000000014535551620023610 5ustar www-datawww-datagitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/context/client.rb0000644000004100000410000000112514535551620025412 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Middleware module Sidekiq module Context # This middleware for Sidekiq-client wraps scheduling jobs in a context # The context will also be added to the sidekiq job in redis so it can # be reinstantiated by Sidekiq-server when running the job. class Client def call(_worker_class, job, _queue, _redis_pool) Labkit::Context.with_context do |context| job.merge!(context.to_h) yield end end end end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/context/server.rb0000644000004100000410000000136714535551620025452 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Middleware module Sidekiq module Context # This middleware for Sidekiq-client uses the values stored on a job to # reinstantiate a context in which the job will run. class Server def call(_worker_class, job, _queue) worker_name = (job["wrapped"].presence || job["class"]).to_s data = job.merge(Labkit::Context.log_key(:caller_id) => worker_name) .select { |key, _| key.start_with?("#{Labkit::Context::LOG_KEY}.") || Labkit::Context::RAW_KEYS.include?(key.to_s) } Labkit::Context.with_context(data) do |_context| yield end end end end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/sidekiq/server.rb0000644000004100000410000000142114535551620023755 0ustar www-datawww-data# frozen_string_literal: true module Labkit module Middleware module Sidekiq # This is a wrapper around all the sidekiq server-middleware in labkit # The only middleware that needs to be added to the chain in GitLab-rails # # It uses a new `Sidekiq::Middleware::Chain` to string multiple middlewares # together. class Server def self.chain @chain ||= ::Sidekiq::Middleware::Chain.new do |chain| chain.add Labkit::Middleware::Sidekiq::Context::Server chain.add Labkit::Middleware::Sidekiq::Tracing::Server if Labkit::Tracing.enabled? end end def call(*args) self.class.chain.invoke(*args) do yield end end end end end end gitlab-labkit-0.34.0/lib/labkit/middleware/rack.rb0000644000004100000410000000222514535551620021741 0ustar www-datawww-data# frozen_string_literal: true require "action_dispatch" require "json" module Labkit module Middleware HEADER = "X-Gitlab-Meta" # This is a rack middleware to be inserted in GitLab-rails # It makes sure that there's always a root context containing the correlation # id. # Since this context always get's cleaned up by this middleware, we can be # sure that any nested contexts will also be cleaned up. class Rack def initialize(app) @app = app end def call(env) Labkit::Context.with_context(Labkit::Context::CORRELATION_ID_KEY => correlation_id(env)) do |context| status, headers, response = @app.call(env) headers[HEADER] = context_to_json(context) [status, headers, response] end end private def correlation_id(env) request(env).request_id end def request(env) ActionDispatch::Request.new(env) end def context_to_json(context) context .to_h .transform_keys { |k| k.delete_prefix("meta.") } .merge("version" => "1") .to_json end end end end gitlab-labkit-0.34.0/lib/labkit/middleware.rb0000644000004100000410000000033614535551620021022 0ustar www-datawww-data# frozen_string_literal: true module Labkit # Adds middlewares for using in rack and sidekiq module Middleware autoload :Rack, "labkit/middleware/rack" autoload :Sidekiq, "labkit/middleware/sidekiq" end end gitlab-labkit-0.34.0/CONTRIBUTING.md0000644000004100000410000000424114535551620016574 0ustar www-datawww-data## Developer Certificate of Origin and License By contributing to GitLab B.V., you accept and agree to the following terms and conditions for your present and future contributions submitted to GitLab B.V. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., you reserve all right, title, and interest in and to your Contributions. All contributions are subject to the Developer Certificate of Origin and license set out at [docs.gitlab.com/ce/legal/developer_certificate_of_origin](https://docs.gitlab.com/ce/legal/developer_certificate_of_origin). _This notice should stay as the first item in the CONTRIBUTING.md file._ ## Code of conduct As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities. We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion. Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct. Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team. This code of conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Instances of abusive, harassing, or otherwise unacceptable behavior can be reported by emailing contact@gitlab.com. This Code of Conduct is adapted from the [Contributor Covenant](https://contributor-covenant.org), version 1.1.0, available at [https://contributor-covenant.org/version/1/1/0/](https://contributor-covenant.org/version/1/1/0/). gitlab-labkit-0.34.0/Gemfile0000644000004100000410000000010614535551620015632 0ustar www-datawww-data# frozen_string_literal: true source "https://rubygems.org" gemspec gitlab-labkit-0.34.0/Dangerfile0000644000004100000410000000020514535551620016322 0ustar www-datawww-data# frozen_string_literal: true require 'gitlab-dangerfiles' Gitlab::Dangerfiles.for_project(self, 'labkit-ruby', &:import_defaults) gitlab-labkit-0.34.0/.tool-versions0000644000004100000410000000001314535551620017160 0ustar www-datawww-dataruby 2.7.7 gitlab-labkit-0.34.0/.ruby-version0000644000004100000410000000000614535551620017003 0ustar www-datawww-data2.7.7 gitlab-labkit-0.34.0/.gitlab-ci.yml0000644000004100000410000000226014535551620016776 0ustar www-datawww-datadefault: image: ruby:3.1 workflow: rules: # For merge requests, create a pipeline. - if: '$CI_MERGE_REQUEST_IID' # For `master` branch, create a pipeline (this includes on schedules, pushes, merges, etc.). - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' # For tags, create a pipeline. - if: '$CI_COMMIT_TAG' .test_template: &test_definition image: ruby:${RUBY_VERSION} stage: test script: - gem install bundler --no-document - bundle config --local path vendor - bundle install - bundle exec rake verify build install cache: key: ${CI_JOB_IMAGE} paths: - vendor/ruby ruby: <<: *test_definition parallel: matrix: - RUBY_VERSION: ["2.7", "3.0", "3.1", "3.2"] static-analysis: before_script: - bundle install script: - rake verify danger-review: image: ruby:3.0 stage: test except: - tags - master before_script: - bundle install script: - bundle exec danger --fail-on-errors=true --verbose deploy: stage: deploy script: - tools/deploy-rubygem.sh only: - tags release: stage: deploy script: - tools/update-changelog.rb "${CI_COMMIT_TAG}" only: - tags