gitlab-experiment-0.6.5/ 0000755 0000041 0000041 00000000000 14147762242 015203 5 ustar www-data www-data gitlab-experiment-0.6.5/README.md 0000644 0000041 0000041 00000060213 14147762242 016464 0 ustar www-data www-data GitLab Experiment
=================
Here at GitLab, we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing variant and promote it as the new default code path. Or revert back to the control if no variant outperformed it. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) docs if you're curious about how we use this gem internally.
This library provides a clean and elegant DSL (domain specific language) to define, run, and track your GitLab experiment.
When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
- `experiment` is any deviation of code paths we want to run sometimes and not others.
- `context` is used to identify a consistent experience we'll provide in an experiment.
- `control` is the default, or "original" code path.
- `candidate` defines that there's one experimental code path.
- `variant(s)` is used when more than one experimental code path exists.
Candidate and variant are the same concept, but simplify how we speak about experimental paths -- this concept is also widely referred to as the "experiment group".
[[_TOC_]]
## Installation
Add the gem to your Gemfile and then `bundle install`.
```ruby
gem 'gitlab-experiment'
```
If you're using Rails, you can install the initializer which provides basic configuration, documentation, and the base experiment class that all your experiments can inherit from.
```shell
$ rails generate gitlab:experiment:install
```
## Implementing an experiment
For the sake of an example let's make one up. Let's run an experiment on what we render for disabling desktop notifications.
In our control (current world) we show a simple toggle interface labeled, "Notifications." In our experiment we want a "Turn on/off desktop notifications" button with a confirmation.
The behavior will be the same, but the interface will be different and may involve more or fewer steps.
Our hypothesis is that this will make the action more clear and will help in making a choice about if that's what the user really wants to do.
We'll name our experiment `notification_toggle`. This name is prefixed based on configuration. If you've set `config.name_prefix = 'gitlab'`, the experiment name would be `gitlab_notification_toggle` elsewhere.
When you implement an experiment you'll need to provide a name, and a context. The name can show up in tracking calls, and potentially other aspects. The context determines the variant assigned, and should be consistent between calls. We'll discuss migrating context in later examples.
A context "key" represents the unique id of a context. It allows us to give the same experience between different calls to the experiment and can be used in caching. This is how an experiment remains "sticky" to a given context.
Now in our experiment we're going to render one of two views: the control will be our current view, and the candidate will be the new toggle button with a confirmation flow.
```ruby
class SubscriptionsController < ApplicationController
def show
experiment(:notification_toggle, actor: user) do |e|
e.use { render_toggle } # control
e.try { render_button } # candidate
end
end
end
```
You can define the experiment using simple control/candidate paths, or provide named variants.
Handling [multivariate](https://en.wikipedia.org/wiki/Multivariate_statistics) experiments is up to the configuration you provide around resolving variants. But in our example we may want to try with and without the confirmation. We can run any number of variations in our experiments this way.
```ruby
experiment(:notification_toggle, actor: user) do |e|
e.use { render_toggle } # control
e.try(:variant_one) { render_button(confirmation: true) }
e.try(:variant_two) { render_button(confirmation: false) }
end
```
You can specify what the experiment should be "sticky" to by providing a `:sticky_to` option. By default this will be the entire context provided, but this can be overridden manually if needed.
```ruby
experiment(:notification_toggle, actor: user, project: project, sticky_to: project) #...
```
Understanding how an experiment can change behavior is important in evaluating its performance.
To this end, we track events that are important by calling the same experiment elsewhere in code. By using the same context, you'll have consistent behavior and the ability to track events to it.
```ruby
experiment(:notification_toggle, actor: user).track(:clicked_button)
```
### Custom experiments
You can craft more advanced behaviors by defining custom experiments at a higher level. To do this you can define a class that inherits from `ApplicationExperiment` (or `Gitlab::Experiment`).
Let's say you want to do more advanced segmentation, or provide default behavior for the variants on the experiment we've already outlined above -- that way if the variants aren't defined in the block at the time the experiment is run, these methods will be used.
You can generate a custom experiment by running:
```shell
$ rails generate gitlab:experiment NotificationToggle control candidate
```
This will generate a file in `app/experiments/notification_toggle_experiment.rb`, as well as a test file for you to further expand on.
Here are some examples of what you can introduce once you have a custom experiment defined.
```ruby
class NotificationToggleExperiment < ApplicationExperiment
# Exclude any users that aren't me.
exclude :users_named_richard
# Segment any account older than 2 weeks into the candidate, without
# asking the variant resolver to decide which variant to provide.
segment :old_account?, variant: :candidate
# Define the default control behavior, which can be overridden at
# experiment time.
def control_behavior
# render_toggle
end
# Define the default candidate behavior, which can be overridden
# at experiment time.
def candidate_behavior
# render_button
end
private
def users_named_richard
context.actor.first_name == 'Richard'
end
def old_account?
context.actor.created_at < 2.weeks.ago
end
end
# The class will be looked up based on the experiment name provided.
exp = experiment(:notification_toggle, actor: user)
exp # => instance of NotificationToggleExperiment
# Run the experiment -- returning the result.
exp.run
# Track an event on the experiment we've defined.
exp.track(:clicked_button)
```
You can now also do things very similar to the simple examples and override the default variant behaviors defined in the custom experiment -- keeping in mind that this should be carefully considered within the scope of your experiment.
```ruby
experiment(:notification_toggle, actor: user) do |e|
e.use { render_special_toggle } # override default control behavior
end
```
You can also specify the variant to use for segmentation...
Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment. It's important to know what this might do to your data during rollout, so use this with careful consideration.
Any time a specific variant is provided (including `:control`) it will be cached for that context, if caching is enabled.
```ruby
experiment(:notification_toggle, :no_interface, actor: user) do |e|
e.use { render_toggle } # control
e.try { render_button } # candidate
e.try(:no_interface) { no_interface! } # no_interface variant
end
```
Or you can set the variant within the block. This allows using unique segmentation logic or variant resolution if you need it.
```ruby
experiment(:notification_toggle, actor: user) do |e|
# Variant selection must be done before calling run or track.
e.variant(:no_interface) # set the variant
# ...
end
```
Or it can be specified in the call to run if you call it from within the block.
```ruby
experiment(:notification_toggle, actor: user) do |e|
# ...
# Variant selection can be specified when calling run.
e.run(:no_interface)
end
```
### Segmentation rules
This library comes with the capability to segment contexts into a specific variant, before asking the variant resolver which variant to provide.
Segmentation can be achieved by using a custom experiment class and specifying the segmentation rules at a class level.
```ruby
class ExampleExperiment < ApplicationExperiment
segment(variant: :variant_one) { context.actor.first_name == 'Richard' }
segment :old_account?, variant: :variant_two
private
def old_account?
context.actor.created_at < 2.weeks.ago
end
end
```
In the previous examples, any user named `'Richard'` would always receive the experience defined in "variant_one". As well, any account older than 2 weeks old would get the alternate experience defined in "variant_two".
When an experiment is run, the segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped.
This means that any user named `'Richard'`, regardless of account age, will always be provided the experience as defined in "variant_one". If you wanted the opposite logic, you can flip the order.
### Exclusion rules
Exclusion rules are similar to segmentation rules, but are intended to determine if a context should even be considered as something we should track events towards. Exclusion means we don't care about the events in relation to the given context.
```ruby
class ExampleExperiment < ApplicationExperiment
exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
private
def old_account?
context.actor.created_at < 2.weeks.ago
end
end
```
The previous examples will exclude all users named `'Richard'` as well as any account older than 2 weeks old. Not only will they be given the control behavior, but no events will be tracked in these cases as well.
You may need to check exclusion in custom tracking logic by calling `should_track?`:
```ruby
def expensive_tracking_logic
return unless should_track?
track(:my_event, value: expensive_method_call)
end
```
Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can implement.
### Return value
By default the return value is a `Gitlab::Experiment` instance. In simple cases you may want only the results of the experiment though. You can call `run` within the block to get the return value of the assigned variant.
```ruby
experiment(:notification_toggle) do |e|
e.use { 'A' }
e.try { 'B' }
e.run
end # => 'A'
```
### Including the DSL
By default, `Gitlab::Experiment` injects itself into the controller and view layers. This exposes the `experiment` method application wide in those layers.
Some experiments may extend outside of those layers, so you may want to include it elsewhere. For instance in a mailer, service object, background job, or similar.
Note: In a lot of these contexts you may not have a reference to the request (unless you pass it in, or provide access to it) which may be needed if you want to enable cookie behaviors and track that through to user conversion.
```ruby
class WelcomeMailer < ApplicationMailer
include Gitlab::Experiment::Dsl # include the `experiment` method
def welcome
@user = params[:user]
ex = experiment(:project_suggestions, actor: @user) do |e|
e.use { 'welcome' }
e.try { 'welcome_with_project_suggestions' }
end
mail(to: @user.email, subject: 'Welcome!', template: ex.run)
end
end
```
### Context migrations
There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.
Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
```ruby
# Migrate just the `:version` portion of the previous context, `{ actor: project, version: 1 }`:
experiment(:example, actor: project, version: 2, migrated_with: { version: 1 })
```
You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
If you wanted to introduce a `version` to your context, provide the full previous context.
```ruby
# Migrate the full context from `{ actor: project }` to `{ actor: project, version: 1 }`:
experiment(:example, actor: project, version: 1, migrated_from: { actor: project })
```
This can impact an experience if you:
1. haven't implemented the concept of migrations in your variant resolver
1. haven't enabled a reasonable caching mechanism
### When there isn't an actor (cookie fallback)
When there isn't an identifying key in the context (this is `actor` by default), we fall back to cookies to provide a consistent experience for the client viewing them.
Once we assign a certain variant to a context, we need to always provide the same experience. We achieve this by setting a cookie for the experiment in question, but only when needed.
This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers.
You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
```ruby
experiment(:example, actor: user, request: request)
```
The cookie isn't set if the `actor` key isn't present at all in the context. Meaning that when no `actor` key is provided, the cookie will not be set.
```ruby
# actor is not present, so no cookie is set
experiment(:example, project: project)
# actor is present and is nil, so the cookie is set and used
experiment(:example, actor: nil, project: project)
# actor is present and set to a value, so no cookie is set
experiment(:example, actor: user, project: project)
```
For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_actor']`. The cookie name is the full experiment name (including any configured prefix) with `_actor` appended -- e.g. `gitlab_notification_toggle_actor` for the `:notification_toggle` experiment key with a configured prefix of `gitlab`.
## How it works
The way the gem works is best described using the following decision tree illustration. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and the context provided.
```mermaid
graph TD
GP[General Pool/Population] --> Enabled?
Enabled? -->|Yes| Cached?[Cached? / Pre-segmented?]
Enabled? -->|No| Excluded[Control / No Tracking]
Cached? -->|No| Excluded?
Cached? -->|Yes| Cached[Cached Value]
Excluded? -->|Yes / Cached| Excluded
Excluded? -->|No| Segmented?
Segmented? -->|Yes / Cached| VariantA
Segmented? -->|No| Included?[Experiment Group?]
Included? -->|Yes| Rollout
Included? -->|No| Control
Rollout -->|Cached| VariantA
Rollout -->|Cached| VariantB
Rollout -->|Cached| VariantC
classDef included fill:#380d75,color:#ffffff,stroke:none
classDef excluded fill:#fca121,stroke:none
classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
classDef default fill:#fff,stroke:#6e49cb
class VariantA,VariantB,VariantC included
class Control,Excluded excluded
class Cached cached
```
## Configuration
This gem needs to be configured before being used in a meaningful way.
The default configuration will always render the control, so it's important to configure your own logic for resolving variants.
Yes, the most important aspect of the gem -- that of determining which variant to render and when -- is up to you. Consider using [Unleash](https://github.com/Unleash/unleash-client-ruby) or [Flipper](https://github.com/jnunemaker/flipper) for this.
```ruby
Gitlab::Experiment.configure do |config|
# The block here is evaluated within the scope of the experiment instance,
# which is why we are able to access things like name and context.
config.variant_resolver = lambda do |requested_variant|
# Return the requested variant if a specific one has been provided in code.
return requested_variant unless requested_variant.nil?
# Ask Unleash to determine the variant, given the context we've built,
# using the control as the fallback.
fallback = Unleash::Variant.new(name: 'control', enabled: true)
UNLEASH.get_variant(name, context.value, fallback)
end
end
```
More examples for configuration are available in the provided [rails initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt).
### Client layer / JavaScript
This library doesn't attempt to provide any logic for the client layer.
Instead it allows you to do this yourself in configuration. Using [Gon](https://github.com/gazay/gon) to publish your experiment information to the client layer is pretty simple.
```ruby
Gitlab::Experiment.configure do |config|
config.publishing_behavior = lambda do |_result|
# Push the experiment knowledge into the front end. The signature contains
# the context key, and the variant that has been determined.
Gon.push({ experiment: { name => signature } }, true)
end
end
```
In the client you can now access `window.gon.experiment.notificationToggle`.
### Caching
Caching can be enabled in configuration, and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
It's important to understand that using caching can drastically change or override your rollout strategy logic.
```ruby
Gitlab::Experiment.configure do |config|
config.cache = Rails.cache
end
```
### Middleware
There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you won't be able to implement tracking. For these cases, gitlab-experiment comes with middleware that will redirect to a given URL while also tracking that the URL was visited.
In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`. If this path is empty the middleware won't be mounted at all.
Once mounted, the redirect URLs can be generated using the Rails route helpers. If not using Rails, mount the middleware and generate these URLs yourself.
```ruby
Gitlab::Experiment.configure do |config|
config.mount_at = '/experiment'
end
ex = experiment(:example, foo: :bar)
# using rails path/url helpers
experiment_redirect_path(ex, 'https//docs.gitlab.com/') # => /experiment/example:[context_key]?https//docs.gitlab.com/
# manually
"#{Gitlab::Experiment.configure.mount_at}/#{ex.to_param}?https//docs.gitlab.com/"
```
URLS that match the base path will be handled by the middleware and will redirect to the provided redirect path.
## Testing (rspec support)
This gem comes with some rspec helpers and custom matchers. These are in flux at the time of writing.
First, require the rspec support file:
```ruby
require 'gitlab/experiment/rspec'
```
This mixes in some of the basics, but the matchers and other aspects need to be included. This happens automatically for files in `spec/experiments`, but for other files and specs you want to include it in, you can specify the `:experiment` type:
```ruby
it "tests", :experiment do
end
```
### Stub helpers
You can stub experiments using `stub_experiments`. Pass it a hash using experiment names as the keys and the variants you want each to resolve to as the values:
```ruby
# Ensures the experiments named `:example` & `:example2` are both
# "enabled" and that each will resolve to the given variant
# (`:my_variant` & `:control` respectively).
stub_experiments(example: :my_variant, example2: :control)
experiment(:example) do |e|
e.enabled? # => true
e.variant.name # => 'my_variant'
end
experiment(:example2) do |e|
e.enabled? # => true
e.variant.name # => 'control'
end
```
### Exclusion and segmentation matchers
You can also easily test the exclusion and segmentation matchers.
```ruby
class ExampleExperiment < ApplicationExperiment
exclude { context.actor.first_name == 'Richard' }
segment(variant: :candidate) { context.actor.username == 'jejacks0n' }
end
excluded = double(username: 'rdiggitty', first_name: 'Richard')
segmented = double(username: 'jejacks0n', first_name: 'Jeremy')
# exclude matcher
expect(experiment(:example)).to exclude(actor: excluded)
expect(experiment(:example)).not_to exclude(actor: segmented)
# segment matcher
expect(experiment(:example)).to segment(actor: segmented).into(:candidate)
expect(experiment(:example)).not_to segment(actor: excluded)
```
### Tracking matcher
Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.
You can do this on the instance level or at an "any instance" level. At an instance level this is pretty straight forward:
```ruby
subject = experiment(:example)
expect(subject).to track(:my_event)
subject.track(:my_event)
```
You can use the `on_any_instance` chain method to specify that it could happen on any instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream:
```ruby
expect(experiment(:example)).to track(:my_event).on_any_instance
experiment(:example).track(:my_event)
```
And here's a full example of the methods that can be chained onto the `track` matcher:
```ruby
expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_')
.on_any_instance
.with_context(foo: :bar)
.for(:variant_name)
experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_')
```
## Tracking, anonymity and GDPR
We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.
This library attempts to be non-user-centric, in that a context can contain things like a user or a project.
If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.
Each of these approaches could be desirable given the objectives of your experiment.
## Development
After checking out the repo, run `bundle install` to install dependencies.
Then, run `bundle exec rake` to run the tests. You can also run `bundle exec pry` for an
interactive prompt that will allow you to experiment.
## Contributing
Bug reports and merge requests are welcome on GitLab at
https://gitlab.com/gitlab-org/gitlab-experiment. This project is intended to be a
safe, welcoming space for collaboration, and contributors are expected to adhere
to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
## Release process
Please refer to the [Release Process](docs/release_process.md).
## License
The gem is available as open source under the terms of the
[MIT License](http://opensource.org/licenses/MIT).
## Code of conduct
Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers,
chat rooms and mailing lists is expected to follow the
[code of conduct](CODE_OF_CONDUCT.md).
***Make code not war***
gitlab-experiment-0.6.5/gitlab-experiment.gemspec 0000644 0000041 0000041 00000010023 14147762242 022164 0 ustar www-data www-data #########################################################
# This file has been automatically generated by gem2tgz #
#########################################################
# -*- encoding: utf-8 -*-
# stub: gitlab-experiment 0.6.5 ruby lib
Gem::Specification.new do |s|
s.name = "gitlab-experiment".freeze
s.version = "0.6.5"
s.required_rubygems_version = Gem::Requirement.new(">= 0".freeze) if s.respond_to? :required_rubygems_version=
s.require_paths = ["lib".freeze]
s.authors = ["GitLab".freeze]
s.date = "2021-11-06"
s.email = ["gitlab_rubygems@gitlab.com".freeze]
s.files = ["LICENSE.txt".freeze, "README.md".freeze, "lib/generators/gitlab".freeze, "lib/generators/gitlab/experiment".freeze, "lib/generators/gitlab/experiment/USAGE".freeze, "lib/generators/gitlab/experiment/experiment_generator.rb".freeze, "lib/generators/gitlab/experiment/install".freeze, "lib/generators/gitlab/experiment/install/install_generator.rb".freeze, "lib/generators/gitlab/experiment/install/templates".freeze, "lib/generators/gitlab/experiment/install/templates/POST_INSTALL".freeze, "lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt".freeze, "lib/generators/gitlab/experiment/install/templates/initializer.rb.tt".freeze, "lib/generators/gitlab/experiment/templates".freeze, "lib/generators/gitlab/experiment/templates/experiment.rb.tt".freeze, "lib/generators/rspec".freeze, "lib/generators/rspec/experiment".freeze, "lib/generators/rspec/experiment/experiment_generator.rb".freeze, "lib/generators/rspec/experiment/templates".freeze, "lib/generators/rspec/experiment/templates/experiment_spec.rb.tt".freeze, "lib/generators/test_unit".freeze, "lib/generators/test_unit/experiment".freeze, "lib/generators/test_unit/experiment/experiment_generator.rb".freeze, "lib/generators/test_unit/experiment/templates".freeze, "lib/generators/test_unit/experiment/templates/experiment_test.rb.tt".freeze, "lib/gitlab/experiment".freeze, "lib/gitlab/experiment.rb".freeze, "lib/gitlab/experiment/base_interface.rb".freeze, "lib/gitlab/experiment/cache".freeze, "lib/gitlab/experiment/cache.rb".freeze, "lib/gitlab/experiment/cache/redis_hash_store.rb".freeze, "lib/gitlab/experiment/callbacks.rb".freeze, "lib/gitlab/experiment/configuration.rb".freeze, "lib/gitlab/experiment/context.rb".freeze, "lib/gitlab/experiment/cookies.rb".freeze, "lib/gitlab/experiment/dsl.rb".freeze, "lib/gitlab/experiment/engine.rb".freeze, "lib/gitlab/experiment/errors.rb".freeze, "lib/gitlab/experiment/middleware.rb".freeze, "lib/gitlab/experiment/nestable.rb".freeze, "lib/gitlab/experiment/rollout".freeze, "lib/gitlab/experiment/rollout.rb".freeze, "lib/gitlab/experiment/rollout/percent.rb".freeze, "lib/gitlab/experiment/rollout/random.rb".freeze, "lib/gitlab/experiment/rollout/round_robin.rb".freeze, "lib/gitlab/experiment/rspec.rb".freeze, "lib/gitlab/experiment/test_behaviors".freeze, "lib/gitlab/experiment/test_behaviors/trackable.rb".freeze, "lib/gitlab/experiment/variant.rb".freeze, "lib/gitlab/experiment/version.rb".freeze]
s.homepage = "https://gitlab.com/gitlab-org/gitlab-experiment".freeze
s.licenses = ["MIT".freeze]
s.required_ruby_version = Gem::Requirement.new(">= 2.6".freeze)
s.rubygems_version = "2.7.6.2".freeze
s.summary = "GitLab experiment library built on top of scientist.".freeze
if s.respond_to? :specification_version then
s.specification_version = 4
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
s.add_runtime_dependency(%q.freeze, [">= 3.0"])
s.add_runtime_dependency(%q.freeze, [">= 1.0"])
s.add_runtime_dependency(%q.freeze, [">= 1.6.0", "~> 1.6"])
else
s.add_dependency(%q.freeze, [">= 3.0"])
s.add_dependency(%q.freeze, [">= 1.0"])
s.add_dependency(%q.freeze, [">= 1.6.0", "~> 1.6"])
end
else
s.add_dependency(%q.freeze, [">= 3.0"])
s.add_dependency(%q.freeze, [">= 1.0"])
s.add_dependency(%q.freeze, [">= 1.6.0", "~> 1.6"])
end
end
gitlab-experiment-0.6.5/lib/ 0000755 0000041 0000041 00000000000 14147762242 015751 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/gitlab/ 0000755 0000041 0000041 00000000000 14147762242 017213 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/gitlab/experiment.rb 0000644 0000041 0000041 00000012317 14147762242 021724 0 ustar www-data www-data # frozen_string_literal: true
require 'scientist'
require 'request_store'
require 'active_support/callbacks'
require 'active_support/cache'
require 'active_support/concern'
require 'active_support/core_ext/object/blank'
require 'active_support/core_ext/string/inflections'
require 'active_support/core_ext/module/delegation'
require 'gitlab/experiment/errors'
require 'gitlab/experiment/base_interface'
require 'gitlab/experiment/cache'
require 'gitlab/experiment/callbacks'
require 'gitlab/experiment/rollout'
require 'gitlab/experiment/configuration'
require 'gitlab/experiment/cookies'
require 'gitlab/experiment/context'
require 'gitlab/experiment/dsl'
require 'gitlab/experiment/middleware'
require 'gitlab/experiment/nestable'
require 'gitlab/experiment/variant'
require 'gitlab/experiment/version'
require 'gitlab/experiment/engine' if defined?(Rails::Engine)
module Gitlab
class Experiment
include BaseInterface
include Cache
include Callbacks
include Nestable
class << self
def default_rollout(rollout = nil, options = {})
return @rollout ||= Configuration.default_rollout if rollout.blank?
@rollout = Rollout.resolve(rollout).new(options)
end
def exclude(*filter_list, **options, &block)
build_callback(:exclusion_check, filter_list.unshift(block), **options) do |target, callback|
throw(:abort) if target.instance_variable_get(:@excluded) || callback.call(target, nil) == true
end
end
def segment(*filter_list, variant:, **options, &block)
build_callback(:segmentation_check, filter_list.unshift(block), **options) do |target, callback|
target.variant(variant) if target.instance_variable_get(:@variant_name).nil? && callback.call(target, nil)
end
end
def published_experiments
RequestStore.store[:published_gitlab_experiments] || {}
end
end
def name
[Configuration.name_prefix, @name].compact.join('_')
end
def control(&block)
candidate(:control, &block)
end
alias_method :use, :control
def candidate(name = nil, &block)
name = (name || :candidate).to_s
behaviors[name] = block
end
alias_method :try, :candidate
def context(value = nil)
return @context if value.blank?
@context.value(value)
@context
end
def variant(value = nil)
@variant_name = cache_variant(value) if value.present?
return Variant.new(name: (@variant_name || :unresolved).to_s) if @variant_name || @resolving_variant
if enabled?
@resolving_variant = true
@variant_name = cached_variant_resolver(@variant_name)
end
run_callbacks(segmentation_callback_chain) do
@variant_name ||= :control
Variant.new(name: @variant_name.to_s)
end
ensure
@resolving_variant = false
end
def rollout(rollout = nil, options = {})
return @rollout ||= self.class.default_rollout(nil, options) if rollout.blank?
@rollout = Rollout.resolve(rollout).new(options)
end
def exclude!
@excluded = true
end
def run(variant_name = nil)
return @result if context.frozen?
@result = run_callbacks(:run) { super(variant(variant_name).name) }
rescue Scientist::BehaviorMissing => e
raise Error, e
end
def publish(result)
instance_exec(result, &Configuration.publishing_behavior)
(RequestStore.store[:published_gitlab_experiments] ||= {})[name] = signature.merge(excluded: excluded?)
end
def track(action, **event_args)
return unless should_track?
instance_exec(action, event_args, &Configuration.tracking_behavior)
end
def process_redirect_url(url)
return unless Configuration.redirect_url_validator&.call(url)
track('visited', url: url)
url # return the url, which allows for mutation
end
def enabled?
true
end
def excluded?
return @excluded if defined?(@excluded)
@excluded = !run_callbacks(:exclusion_check) { :not_excluded }
end
def experiment_group?
instance_exec(@variant_name, &Configuration.inclusion_resolver)
end
def should_track?
enabled? && @context.trackable? && !excluded?
end
def signature
{ variant: variant.name, experiment: name }.merge(context.signature)
end
def key_for(source, seed = name)
# TODO: Added deprecation in release 0.6.0
if (block = Configuration.instance_variable_get(:@__context_hash_strategy))
return instance_exec(source, seed, &block)
end
return source if source.is_a?(String)
source = source.keys + source.values if source.is_a?(Hash)
ingredients = Array(source).map { |v| identify(v) }
ingredients.unshift(seed).unshift(Configuration.context_key_secret)
Digest::SHA2.new(Configuration.context_key_bit_length).hexdigest(ingredients.join('|'))
end
protected
def identify(object)
(object.respond_to?(:to_global_id) ? object.to_global_id : object).to_s
end
def segmentation_callback_chain
return :segmentation_check if @variant_name.nil? && enabled? && !excluded?
:unsegmented
end
def resolve_variant_name
rollout.rollout_for(self) if experiment_group?
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/ 0000755 0000041 0000041 00000000000 14147762242 021373 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/gitlab/experiment/rollout/ 0000755 0000041 0000041 00000000000 14147762242 023073 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/gitlab/experiment/rollout/random.rb 0000644 0000041 0000041 00000000542 14147762242 024701 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Rollout
class Random < Base
# Pick a random variant if we're in the experiment group. It doesn't
# take into account small sample sizes but is useful and performant.
def execute
variant_names.sample
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/rollout/round_robin.rb 0000644 0000041 0000041 00000001230 14147762242 025734 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Rollout
class RoundRobin < Base
KEY_NAME = :last_round_robin_variant
# Requires a cache to be configured.
#
# Keeps track of the number of assignments into the experiment group,
# and uses this to rotate "round robin" style through the variants
# that are defined.
#
# Relatively performant, but requires a cache, and is dependent on the
# performance of that cache store.
def execute
variant_names[(cache.attr_inc(KEY_NAME) - 1) % variant_names.size]
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/rollout/percent.rb 0000644 0000041 0000041 00000002477 14147762242 025072 0 ustar www-data www-data # frozen_string_literal: true
require 'zlib'
module Gitlab
class Experiment
module Rollout
class Percent < Base
def execute
crc = normalized_id
total = 0
case distribution_rules
# run through the rules until finding an acceptable one
when Array then variant_names[distribution_rules.find_index { |percent| crc % 100 <= total += percent }]
# run through the variant names until finding an acceptable one
when Hash then distribution_rules.find { |_, percent| crc % 100 <= total += percent }.first
# when there are no rules, assume even distribution
else variant_names[crc % variant_names.length]
end
end
def validate!
case distribution_rules
when nil then nil
when Array, Hash
if distribution_rules.length != variant_names.length
raise InvalidRolloutRules, "the distribution rules don't match the number of variants defined"
end
else
raise InvalidRolloutRules, 'unknown distribution options type'
end
end
private
def normalized_id
Zlib.crc32(id, nil)
end
def distribution_rules
@options[:distribution]
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/version.rb 0000644 0000041 0000041 00000000140 14147762242 023400 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
VERSION = '0.6.5'
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/cache.rb 0000644 0000041 0000041 00000003620 14147762242 022764 0 ustar www-data www-data # frozen_string_literal: true
require 'active_support/cache'
module Gitlab
class Experiment
module Cache
autoload :RedisHashStore, 'gitlab/experiment/cache/redis_hash_store.rb'
class Interface
attr_reader :store, :key
def initialize(experiment, store)
@experiment = experiment
@store = store
@key = experiment.cache_key
end
def read
store.read(key)
end
def write(value = nil)
store.write(key, value || @experiment.variant.name)
end
def delete
store.delete(key)
end
def attr_get(name)
store.read(@experiment.cache_key(name, suffix: :attrs))
end
def attr_set(name, value)
store.write(@experiment.cache_key(name, suffix: :attrs), value)
end
def attr_inc(name, amount = 1)
store.increment(@experiment.cache_key(name, suffix: :attrs), amount)
end
end
def cache
@cache ||= Interface.new(self, Configuration.cache)
end
def cache_variant(specified = nil, &block)
return (specified.presence || yield) unless cache.store
result = migrated_cache_fetch(cache.store, &block)
return result unless specified.present?
cache.write(specified) if result != specified
specified
end
def cache_key(key = nil, suffix: nil)
"#{[name, suffix].compact.join('_')}:#{key || context.signature[:key]}"
end
private
def migrated_cache_fetch(store, &block)
migrations = context.signature[:migration_keys]&.map { |key| cache_key(key) } || []
migrations.find do |old_key|
next unless (value = store.read(old_key))
store.write(cache_key, value)
store.delete(old_key)
return value
end
store.fetch(cache_key, &block)
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/nestable.rb 0000644 0000041 0000041 00000001434 14147762242 023517 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Nestable
extend ActiveSupport::Concern
included do
set_callback :run, :around, :manage_nested_stack
end
def nest_experiment(other)
raise NestingError, "unable to nest the #{other.name} experiment within the #{name} experiment"
end
private
def manage_nested_stack
Stack.push(self)
yield
ensure
Stack.pop
end
class Stack
include Singleton
@stack = []
class << self
delegate :pop, :length, :size, :[], to: :@stack
def push(instance)
@stack.last&.nest_experiment(instance)
@stack.push(instance)
end
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/errors.rb 0000644 0000041 0000041 00000000276 14147762242 023241 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
Error = Class.new(StandardError)
InvalidRolloutRules = Class.new(Error)
NestingError = Class.new(Error)
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/context.rb 0000644 0000041 0000041 00000004341 14147762242 023406 0 ustar www-data www-data # frozen_string_literal: true
require 'gitlab/experiment/cookies'
module Gitlab
class Experiment
class Context
include Cookies
DNT_REGEXP = /^(true|t|yes|y|1|on)$/i.freeze
def initialize(experiment, **initial_value)
@experiment = experiment
@value = {}
@migrations = { merged: [], unmerged: [] }
value(initial_value)
end
def reinitialize(request)
@signature = nil # clear memoization
@request = request if request.respond_to?(:headers) && request.respond_to?(:cookie_jar)
end
def value(value = nil)
return @value if value.nil?
value = value.dup # dup so we don't mutate
reinitialize(value.delete(:request))
key(value.delete(:sticky_to))
@value.merge!(process_migrations(value))
end
def key(key = nil)
return @key || @experiment.key_for(value) if key.nil?
@key = @experiment.key_for(key)
end
def trackable?
!(@request && @request.headers['DNT'].to_s.match?(DNT_REGEXP))
end
def freeze
signature # finalize before freezing
super
end
def signature
@signature ||= { key: key, migration_keys: migration_keys }.compact
end
def method_missing(method_name, *)
@value.include?(method_name.to_sym) ? @value[method_name.to_sym] : super
end
def respond_to_missing?(method_name, *)
@value.include?(method_name.to_sym) ? true : super
end
private
def process_migrations(value)
add_unmerged_migration(value.delete(:migrated_from))
add_merged_migration(value.delete(:migrated_with))
migrate_cookie(value, "#{@experiment.name}_id")
end
def add_unmerged_migration(value = {})
@migrations[:unmerged] << value if value.is_a?(Hash)
end
def add_merged_migration(value = {})
@migrations[:merged] << value if value.is_a?(Hash)
end
def migration_keys
return nil if @migrations[:unmerged].empty? && @migrations[:merged].empty?
@migrations[:unmerged].map { |m| @experiment.key_for(m) } +
@migrations[:merged].map { |m| @experiment.key_for(@value.merge(m)) }
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/configuration.rb 0000644 0000041 0000041 00000006352 14147762242 024575 0 ustar www-data www-data # frozen_string_literal: true
require 'singleton'
require 'logger'
require 'digest'
require 'active_support/deprecation'
module Gitlab
class Experiment
class Configuration
include Singleton
# Prefix all experiment names with a given value. Use `nil` for none.
@name_prefix = nil
# The logger is used to log various details of the experiments.
@logger = Logger.new($stdout)
# The base class that should be instantiated for basic experiments.
@base_class = 'Gitlab::Experiment'
# The caching layer is expected to respond to fetch, like Rails.cache.
@cache = nil
# The domain to use on cookies.
@cookie_domain = :all
# The default rollout strategy only works for single variant experiments.
# It's expected that you use a more advanced rollout for multiple variant
# experiments.
@default_rollout = Rollout::Base.new
# Secret seed used in generating context keys.
@context_key_secret = nil
# Bit length used by SHA2 in generating context keys - (256, 384 or 512.)
@context_key_bit_length = 256
# The default base path that the middleware (or rails engine) will be
# mounted.
@mount_at = nil
# The middleware won't redirect to urls that aren't considered valid.
# Expected to return a boolean value.
@redirect_url_validator = ->(_redirect_url) { true }
# Logic this project uses to determine inclusion in a given experiment.
# Expected to return a boolean value.
@inclusion_resolver = ->(_requested_variant) { false }
# Tracking behavior can be implemented to link an event to an experiment.
@tracking_behavior = lambda do |event, args|
Configuration.logger.info("#{self.class.name}[#{name}] #{event}: #{args.merge(signature: signature)}")
end
# Called at the end of every experiment run, with the result.
@publishing_behavior = lambda do |_result|
track(:assignment)
end
class << self
# TODO: Added deprecation in release 0.6.0
def context_hash_strategy=(block)
ActiveSupport::Deprecation.warn('context_hash_strategy has been deprecated, instead configure' \
' `context_key_secret` and `context_key_bit_length`.')
@__context_hash_strategy = block
end
# TODO: Added deprecation in release 0.5.0
def variant_resolver
ActiveSupport::Deprecation.warn('variant_resolver is deprecated, instead use `inclusion_resolver` with a' \
' block that returns a boolean.')
@inclusion_resolver
end
def variant_resolver=(block)
ActiveSupport::Deprecation.warn('variant_resolver is deprecated, instead use `inclusion_resolver` with a' \
' block that returns a boolean.')
@inclusion_resolver = block
end
attr_accessor(
:name_prefix,
:logger,
:base_class,
:cache,
:cookie_domain,
:context_key_secret,
:context_key_bit_length,
:mount_at,
:default_rollout,
:redirect_url_validator,
:inclusion_resolver,
:tracking_behavior,
:publishing_behavior
)
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/engine.rb 0000644 0000041 0000041 00000002440 14147762242 023165 0 ustar www-data www-data # frozen_string_literal: true
require 'active_model'
module Gitlab
class Experiment
include ActiveModel::Model
# used for generating routes
def self.model_name
ActiveModel::Name.new(self, Gitlab)
end
class Engine < ::Rails::Engine
isolate_namespace Experiment
initializer('gitlab_experiment.include_dsl') { include_dsl }
initializer('gitlab_experiment.mount_engine') { |app| mount_engine(app, Configuration.mount_at) }
private
def include_dsl
Dsl.include_in(ActionController::Base, with_helper: true) if defined?(ActionController)
Dsl.include_in(ActionMailer::Base, with_helper: true) if defined?(ActionMailer)
end
def mount_engine(app, mount_at)
return if mount_at.blank?
engine = routes do
default_url_options app.routes.default_url_options.clone.without(:script_name)
resources :experiments, path: '/', only: :show
end
app.config.middleware.use(Middleware, mount_at)
app.routes.append do
mount Engine, at: mount_at, as: :experiment_engine
direct(:experiment_redirect) do |ex, options|
url = options[:url]
"#{engine.url_helpers.experiment_url(ex)}?#{url}"
end
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/dsl.rb 0000644 0000041 0000041 00000001331 14147762242 022500 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Dsl
def self.include_in(klass, with_helper: false)
klass.include(self).tap { |base| base.helper_method(:experiment) if with_helper }
end
def experiment(name, variant_name = nil, **context, &block)
raise ArgumentError, 'name is required' if name.nil?
context[:request] ||= request if respond_to?(:request)
base = Configuration.base_class.constantize
klass = base.constantize(name) || base
instance = klass.new(name, variant_name, **context, &block)
return instance unless block
instance.context.frozen? ? instance.run : instance.tap(&:run)
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/test_behaviors/ 0000755 0000041 0000041 00000000000 14147762242 024414 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/gitlab/experiment/test_behaviors/trackable.rb 0000644 0000041 0000041 00000002721 14147762242 026673 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module TestBehaviors
module Trackable
private
def manage_nested_stack
TrackedStructure.push(self)
super
ensure
TrackedStructure.pop
end
end
class TrackedStructure
include Singleton
# dependency tracking
@flat = {}
@stack = []
# structure tracking
@tree = { name: nil, count: 0, children: {} }
@node = @tree
class << self
def reset!
# dependency tracking
@flat = {}
@stack = []
# structure tracking
@tree = { name: nil, count: 0, children: {} }
@node = @tree
end
def hierarchy
@tree[:children]
end
def dependencies
@flat
end
def push(instance)
# dependency tracking
@flat[instance.name] = ((@flat[instance.name] || []) + @stack.map(&:name)).uniq
@stack.push(instance)
# structure tracking
@last = @node
@node = @node[:children][instance.name] ||= { name: instance.name, count: 0, children: {} }
@node[:count] += 1
end
def pop
# dependency tracking
@stack.pop
# structure tracking
@node = @last
end
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/variant.rb 0000644 0000041 0000041 00000000340 14147762242 023361 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
Variant = Struct.new(:name, :payload, keyword_init: true) do
def group
name == 'control' ? :control : :experiment
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/rollout.rb 0000644 0000041 0000041 00000002026 14147762242 023420 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Rollout
autoload :Percent, 'gitlab/experiment/rollout/percent.rb'
autoload :Random, 'gitlab/experiment/rollout/random.rb'
autoload :RoundRobin, 'gitlab/experiment/rollout/round_robin.rb'
def self.resolve(klass)
return "#{name}::#{klass.to_s.classify}".constantize if klass.is_a?(Symbol) || klass.is_a?(String)
klass
end
class Base
attr_reader :experiment
delegate :variant_names, :cache, :id, to: :experiment
def initialize(options = {})
@options = options
# validate! # we want to validate here, but we can't yet
end
def rollout_for(experiment)
@experiment = experiment
validate! # until we have variant registration we can only validate here
execute
end
def validate!
# base is always valid
end
def execute
variant_names.first
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/cookies.rb 0000644 0000041 0000041 00000002207 14147762242 023355 0 ustar www-data www-data # frozen_string_literal: true
require 'securerandom'
module Gitlab
class Experiment
module Cookies
private
def migrate_cookie(hash, cookie_name)
return hash if cookie_jar.nil?
resolver = [hash, :actor, cookie_name, cookie_jar.signed[cookie_name]]
resolve_cookie(*resolver) || generate_cookie(*resolver)
end
def cookie_jar
@request&.cookie_jar
end
def resolve_cookie(hash, key, cookie_name, cookie)
return if cookie.to_s.empty? && hash[key].nil?
return hash if cookie.to_s.empty?
return hash.merge(key => cookie) if hash[key].nil?
add_unmerged_migration(key => cookie)
cookie_jar.delete(cookie_name, domain: domain)
hash
end
def generate_cookie(hash, key, cookie_name, cookie)
return hash unless hash.key?(key)
cookie ||= SecureRandom.uuid
cookie_jar.permanent.signed[cookie_name] = {
value: cookie, secure: true, domain: domain, httponly: true
}
hash.merge(key => cookie)
end
def domain
Configuration.cookie_domain
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/cache/ 0000755 0000041 0000041 00000000000 14147762242 022436 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/gitlab/experiment/cache/redis_hash_store.rb 0000644 0000041 0000041 00000004342 14147762242 026313 0 ustar www-data www-data # frozen_string_literal: true
require 'active_support/notifications'
# This cache strategy is an implementation on top of the redis hash data type,
# that also adheres to the ActiveSupport::Cache::Store interface. It's a good
# example of how to build a custom caching strategy for Gitlab::Experiment, and
# is intended to be a long lived cache -- until the experiment is cleaned up.
#
# The data structure:
# key: experiment.name
# fields: context key => variant name
#
# Gitlab::Experiment::Configuration.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
# pool: -> { Gitlab::Redis::SharedState.with { |redis| yield redis } }
# )
module Gitlab
class Experiment
module Cache
class RedisHashStore < ActiveSupport::Cache::Store
# Clears the entire cache for a given experiment. Be careful with this
# since it would reset all resolved variants for the entire experiment.
def clear(key:)
key = hkey(key)[0] # extract only the first part of the key
pool do |redis|
case redis.type(key)
when 'hash', 'none'
redis.del(key) # delete the primary experiment key
redis.del("#{key}_attrs") # delete the experiment attributes key
else raise ArgumentError, 'invalid call to clear a non-hash cache key'
end
end
end
def increment(key, amount = 1)
pool { |redis| redis.hincrby(*hkey(key), amount) }
end
private
def pool(&block)
raise ArgumentError, 'missing block' unless block.present?
@options[:pool].call(&block)
end
def hkey(key)
key.to_s.split(':') # this assumes the default strategy in gitlab-experiment
end
def read_entry(key, **options)
value = pool { |redis| redis.hget(*hkey(key)) }
value.nil? ? nil : ActiveSupport::Cache::Entry.new(value)
end
def write_entry(key, entry, **options)
return false if entry.value.blank? # don't cache any empty values
pool { |redis| redis.hset(*hkey(key), entry.value) }
end
def delete_entry(key, **options)
pool { |redis| redis.hdel(*hkey(key)) }
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/base_interface.rb 0000644 0000041 0000041 00000005346 14147762242 024662 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module BaseInterface
extend ActiveSupport::Concern
include Scientist::Experiment
class_methods do
def configure
yield Configuration
end
def experiment_name(name = nil, suffix: true, suffix_word: 'experiment')
name = (name.presence || self.name).to_s.underscore.sub(%r{(?[_/]|)#{suffix_word}$}, '')
name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}"
suffix ? name : name.sub(/_#{suffix_word}$/, '')
end
def base?
self == Gitlab::Experiment || name == Configuration.base_class
end
def constantize(name = nil)
return self if name.nil?
experiment_name(name).classify.safe_constantize || Configuration.base_class.constantize
end
def from_param(id)
%r{/?(?.*):(?.*)$} =~ id
name = CGI.unescape(name) if name
constantize(name).new(name).tap { |e| e.context.key(key) }
end
end
def initialize(name = nil, variant_name = nil, **context)
raise ArgumentError, 'name is required' if name.blank? && self.class.base?
@name = self.class.experiment_name(name, suffix: false)
@context = Context.new(self, **context)
@variant_name = cache_variant(variant_name) { nil } if variant_name.present?
compare { false }
yield self if block_given?
end
def inspect
"#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} @name=#{name} @context=#{context.value}>"
end
def id
"#{name}:#{context.key}"
end
alias_method :session_id, :id
alias_method :to_param, :id
def flipper_id
"Experiment;#{id}"
end
def variant_names
@variant_names ||= behaviors.keys.map(&:to_sym) - [:control]
end
def behaviors
@behaviors ||= public_methods.each_with_object(super) do |name, behaviors|
next unless name.end_with?('_behavior')
behavior_name = name.to_s.sub(/_behavior$/, '')
behaviors[behavior_name] ||= -> { send(name) } # rubocop:disable GitlabSecurity/PublicSend
end
end
protected
def raise_on_mismatches?
false
end
def cached_variant_resolver(provided_variant)
return :control if excluded?
result = cache_variant(provided_variant) { resolve_variant_name }
result.to_sym if result.present?
end
def generate_result(variant_name)
observation = Scientist::Observation.new(variant_name, self, &behaviors[variant_name])
Scientist::Result.new(self, [observation], observation)
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/callbacks.rb 0000644 0000041 0000041 00000001612 14147762242 023637 0 ustar www-data www-data # frozen_string_literal: true
require 'active_support/callbacks'
module Gitlab
class Experiment
module Callbacks
extend ActiveSupport::Concern
include ActiveSupport::Callbacks
included do
define_callbacks(:run)
define_callbacks(:unsegmented)
define_callbacks(:segmentation_check)
define_callbacks(:exclusion_check, skip_after_callbacks_if_terminated: true)
end
class_methods do
private
def build_callback(chain, filters, **options)
filters = filters.compact.map do |filter|
result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
->(target) { yield(target, result_lambda) }
end
raise ArgumentError, 'no filters provided' if filters.empty?
set_callback(chain, *filters, **options)
end
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/middleware.rb 0000644 0000041 0000041 00000001336 14147762242 024040 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
class Middleware
def self.redirect(id, url)
raise Error, 'no url to redirect to' if url.blank?
experiment = Gitlab::Experiment.from_param(id)
[303, { 'Location' => experiment.process_redirect_url(url) || raise(Error, 'not redirecting') }, []]
end
def initialize(app, base_path)
@app = app
@matcher = %r{^#{base_path}/(?.+)}
end
def call(env)
return @app.call(env) if env['REQUEST_METHOD'] != 'GET' || (match = @matcher.match(env['PATH_INFO'])).nil?
Middleware.redirect(match[:id], env['QUERY_STRING'])
rescue Error
@app.call(env)
end
end
end
end
gitlab-experiment-0.6.5/lib/gitlab/experiment/rspec.rb 0000644 0000041 0000041 00000020373 14147762242 023041 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module TestBehaviors
autoload :Trackable, 'gitlab/experiment/test_behaviors/trackable.rb'
end
module RSpecHelpers
def stub_experiments(experiments, times = nil)
experiments.each { |experiment| wrapped_experiment(experiment, times) }
end
def wrapped_experiment(experiment, times = nil, expected = false, &block)
klass, experiment_name, variant_name = *experiment_details(experiment)
base_klass = Configuration.base_class.constantize
# Set expectations on experiment classes so we can and_wrap_original with more specific args
experiment_klasses = base_klass.descendants.reject { |k| k == klass }
experiment_klasses.push(base_klass).each do |k|
allow(k).to receive(:new).and_call_original
end
receiver = receive(:new)
# Be specific for BaseClass calls
receiver = receiver.with(experiment_name, any_args) if experiment_name && klass == base_klass
receiver.exactly(times).times if times
# Set expectations on experiment class of interest
allow_or_expect_klass = expected ? expect(klass) : allow(klass)
allow_or_expect_klass.to receiver.and_wrap_original do |method, *original_args, &original_block|
method.call(*original_args).tap do |e|
# Stub internal methods before calling the original_block
allow(e).to receive(:enabled?).and_return(true)
if variant_name == true # passing true allows the rollout to do its job
allow(e).to receive(:experiment_group?).and_return(true)
else
allow(e).to receive(:resolve_variant_name).and_return(variant_name.to_s)
end
# Stub/set expectations before calling the original_block
yield e if block
original_block.call(e) if original_block.present?
end
end
end
private
def experiment_details(experiment)
if experiment.is_a?(Symbol)
experiment_name = experiment
variant_name = nil
end
experiment_name, variant_name = *experiment if experiment.is_a?(Array)
base_klass = Configuration.base_class.constantize
variant_name = experiment.variant.name if experiment.is_a?(base_klass)
if experiment.class.name.nil? # Anonymous class instance
klass = experiment.class
elsif experiment.instance_of?(Class) # Class level stubbing, eg. "MyExperiment"
klass = experiment
else
experiment_name ||= experiment.instance_variable_get(:@name)
klass = base_klass.constantize(experiment_name)
end
if experiment_name && klass == base_klass
experiment_name = experiment_name.to_sym
# For experiment names like: "group/experiment-name"
experiment_name = experiment_name.to_s if experiment_name.inspect.include?('"')
end
[klass, experiment_name, variant_name]
end
end
module RSpecMatchers
extend RSpec::Matchers::DSL
def require_experiment(experiment, matcher_name, classes: false)
klass = experiment.instance_of?(Class) ? experiment : experiment.class
unless klass <= Gitlab::Experiment
raise(
ArgumentError,
"#{matcher_name} matcher is limited to experiment instances#{classes ? ' and classes' : ''}"
)
end
if experiment == klass && !classes
raise ArgumentError, "#{matcher_name} matcher requires an instance of an experiment"
end
experiment != klass
end
matcher :exclude do |context|
ivar = :'@excluded'
match do |experiment|
require_experiment(experiment, 'exclude')
experiment.context(context)
experiment.instance_variable_set(ivar, nil)
!experiment.run_callbacks(:exclusion_check) { :not_excluded }
end
failure_message do
%(expected #{context} to be excluded)
end
failure_message_when_negated do
%(expected #{context} not to be excluded)
end
end
matcher :segment do |context|
ivar = :'@variant_name'
match do |experiment|
require_experiment(experiment, 'segment')
experiment.context(context)
experiment.instance_variable_set(ivar, nil)
experiment.run_callbacks(:segmentation_check)
@actual = experiment.instance_variable_get(ivar)
@expected ? @actual.to_s == @expected.to_s : @actual.present?
end
chain :into do |expected|
raise ArgumentError, 'variant name must be provided' if expected.blank?
@expected = expected.to_s
end
failure_message do
%(expected #{context} to be segmented#{message_details})
end
failure_message_when_negated do
%(expected #{context} not to be segmented#{message_details})
end
def message_details
message = ''
message += %( into variant\n expected variant: #{@expected}) if @expected
message += %(\n actual variant: #{@actual}) if @actual
message
end
end
matcher :track do |event, *event_args|
match do |experiment|
expect_tracking_on(experiment, false, event, *event_args)
end
match_when_negated do |experiment|
expect_tracking_on(experiment, true, event, *event_args)
end
chain :for do |expected_variant|
raise ArgumentError, 'variant name must be provided' if expected.blank?
@expected_variant = expected_variant.to_s
end
chain(:with_context) { |expected_context| @expected_context = expected_context }
chain(:on_next_instance) { @on_next_instance = true }
def expect_tracking_on(experiment, negated, event, *event_args)
klass = experiment.instance_of?(Class) ? experiment : experiment.class
unless klass <= Gitlab::Experiment
raise(
ArgumentError,
"track matcher is limited to experiment instances and classes"
)
end
expectations = proc do |e|
@experiment = e
allow(e).to receive(:track).and_call_original
if negated
expect(e).not_to receive(:track).with(*[event, *event_args])
else
if @expected_variant
expect(@experiment.variant.name).to eq(@expected_variant), failure_message(:variant, event)
end
if @expected_context
expect(@experiment.context.value).to include(@expected_context), failure_message(:context, event)
end
expect(e).to receive(:track).with(*[event, *event_args]).and_call_original
end
end
if experiment.instance_of?(Class) || @on_next_instance
wrapped_experiment(experiment, nil, true) { |e| expectations.call(e) }
else
expectations.call(experiment)
end
end
def failure_message(failure_type, event)
case failure_type
when :variant
<<~MESSAGE.strip
expected #{@experiment.inspect} to have tracked #{event.inspect} for variant
expected variant: #{@expected_variant}
actual variant: #{@experiment.variant.name}
MESSAGE
when :context
<<~MESSAGE.strip
expected #{@experiment.inspect} to have tracked #{event.inspect} with context
expected context: #{@expected_context}
actual context: #{@experiment.context.value}
MESSAGE
end
end
end
end
end
end
RSpec.configure do |config|
config.include Gitlab::Experiment::RSpecHelpers
config.include Gitlab::Experiment::Dsl
config.before(:each, :experiment) do
RequestStore.clear!
if defined?(Gitlab::Experiment::TestBehaviors::TrackedStructure)
Gitlab::Experiment::TestBehaviors::TrackedStructure.reset!
end
end
config.include Gitlab::Experiment::RSpecMatchers, :experiment
config.define_derived_metadata(file_path: Regexp.new('/spec/experiments/')) do |metadata|
metadata[:type] = :experiment
end
end
gitlab-experiment-0.6.5/lib/generators/ 0000755 0000041 0000041 00000000000 14147762242 020122 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/gitlab/ 0000755 0000041 0000041 00000000000 14147762242 021364 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/ 0000755 0000041 0000041 00000000000 14147762242 023544 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/experiment_generator.rb 0000644 0000041 0000041 00000001475 14147762242 030326 0 ustar www-data www-data # frozen_string_literal: true
require 'rails/generators'
module Gitlab
module Generators
class ExperimentGenerator < Rails::Generators::NamedBase
source_root File.expand_path('templates/', __dir__)
check_class_collision suffix: 'Experiment'
argument :variants,
type: :array,
default: %w[control candidate],
banner: 'variant variant'
def create_experiment
template 'experiment.rb', File.join('app/experiments', class_path, "#{file_name}_experiment.rb")
end
hook_for :test_framework
private
def file_name
@_file_name ||= remove_possible_suffix(super)
end
def remove_possible_suffix(name)
name.sub(/_?exp[ei]riment$/i, "") # be somewhat forgiving with spelling
end
end
end
end
gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/templates/ 0000755 0000041 0000041 00000000000 14147762242 025542 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/templates/experiment.rb.tt 0000644 0000041 0000041 00000000550 14147762242 030675 0 ustar www-data www-data # frozen_string_literal: true
<% if namespaced? -%>
require_dependency "<%= namespaced_path %>/application_experiment"
<% end -%>
<% module_namespacing do -%>
class <%= class_name %>Experiment < ApplicationExperiment
<% variants.each do |variant| -%>
def <%= variant %>_behavior
end
<%= "\n" unless variant == variants.last -%>
<% end -%>
end
<% end -%>
gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/install/ 0000755 0000041 0000041 00000000000 14147762242 025212 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/install/templates/ 0000755 0000041 0000041 00000000000 14147762242 027210 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/install/templates/initializer.rb.tt 0000644 0000041 0000041 00000010471 14147762242 032511 0 ustar www-data www-data # frozen_string_literal: true
Gitlab::Experiment.configure do |config|
# Prefix all experiment names with a given value. Use `nil` for none.
config.name_prefix = nil
# The logger is used to log various details of the experiments.
config.logger = Logger.new($stdout)
# The base class that should be instantiated for basic experiments. It should
# be a string, so we can constantize it later.
config.base_class = 'ApplicationExperiment'
# The caching layer is expected to respond to fetch, like Rails.cache for
# instance -- or anything that adheres to ActiveSupport::Cache::Store.
config.cache = nil
# The domain to use on cookies.
#
# When not set, it uses the current host. If you want to provide specific
# hosts, you use `:all`, or provide an array.
#
# Examples:
# nil, :all, or ['www.gitlab.com', '.gitlab.com']
config.cookie_domain = :all
# The default rollout strategy that works for single and multi-variants.
#
# You can provide your own rollout strategies and override them per
# experiment.
#
# Examples include:
# Rollout::Random, or Rollout::RoundRobin
config.default_rollout = Gitlab::Experiment::Rollout::Percent
# Secret seed used in generating context keys.
#
# Consider not using one that's shared with other systems, like Rails'
# SECRET_KEY_BASE. Generate a new secret and utilize that instead.
@context_key_secret = nil
# Bit length used by SHA2 in generating context keys.
#
# Using a higher bit length would require more computation time.
#
# Valid bit lengths:
# 256, 384, or 512.
@context_key_bit_length = 256
# The default base path that the middleware (or rails engine) will be
# mounted. Can be nil if you don't want anything to be mounted automatically.
#
# This enables a similar behavior to how links are instrumented in emails.
#
# Examples:
# '/-/experiment', '/redirect', nil
config.mount_at = '/experiment'
# When using the middleware, links can be instrumented and redirected
# elsewhere. This can be exploited to make a harmful url look innocuous or
# that it's a valid url on your domain. To avoid this, you can provide your
# own logic for what urls will be considered valid and redirected to.
#
# Expected to return a boolean value.
config.redirect_url_validator = lambda do |redirect_url|
true
end
# Logic this project uses to determine inclusion in a given experiment.
#
# Expected to return a boolean value.
#
# This block is executed within the scope of the experiment and so can access
# experiment methods, like `name`, `context`, and `signature`.
config.inclusion_resolver = lambda do |requested_variant|
false
end
# Tracking behavior can be implemented to link an event to an experiment.
#
# This block is executed within the scope of the experiment and so can access
# experiment methods, like `name`, `context`, and `signature`.
config.tracking_behavior = lambda do |event, args|
# An example of using a generic logger to track events:
config.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
# Using something like snowplow to track events (in gitlab):
#
# Gitlab::Tracking.event(name, event, **args.merge(
# context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new(
# 'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0', signature
# )
# ))
end
# Called at the end of every experiment run, with the result.
#
# You may want to track that you've assigned a variant to a given context,
# or push the experiment into the client or publish results elsewhere like
# into redis.
#
# This block is executed within the scope of the experiment and so can access
# experiment methods, like `name`, `context`, and `signature`.
config.publishing_behavior = lambda do |result|
# Track the event using our own configured tracking logic.
track(:assignment)
# Push the experiment knowledge into the front end. The signature contains
# the context key, and the variant that has been determined.
#
# Gon.push({ experiment: { name => signature } }, true)
# Log using our logging system, so the result (which can be large) can be
# reviewed later if we want to.
#
# Lograge::Event.log(experiment: name, result: result, signature: signature)
end
end
././@LongLink 0000644 0000000 0000000 00000000150 00000000000 011577 L ustar root root gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/install/templates/application_experiment.rb0000644 0000041 0000041 00000000124 14147762242 034275 0 ustar www-data www-data # frozen_string_literal: true
class ApplicationExperiment < Gitlab::Experiment
end
gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/install/templates/POST_INSTALL 0000644 0000041 0000041 00000000177 14147762242 031113 0 ustar www-data www-data Gitlab::Experiment has been installed. You may want to adjust the configuration
that's been provided in the Rails initializer.
gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/install/install_generator.rb 0000644 0000041 0000041 00000002264 14147762242 031257 0 ustar www-data www-data # frozen_string_literal: true
require 'rails/generators'
module Gitlab
module Generators
module Experiment
class InstallGenerator < Rails::Generators::Base
source_root File.expand_path('templates', __dir__)
desc 'Installs the Gitlab::Experiment initializer and optional ApplicationExperiment into your application.'
class_option :skip_initializer,
type: :boolean,
default: false,
desc: 'Skip the initializer with default configuration'
class_option :skip_baseclass,
type: :boolean,
default: false,
desc: 'Skip the ApplicationExperiment base class'
def create_initializer
return if options[:skip_initializer]
template 'initializer.rb', 'config/initializers/gitlab_experiment.rb'
end
def create_baseclass
return if options[:skip_baseclass]
template 'application_experiment.rb', 'app/experiments/application_experiment.rb'
end
def display_post_install
readme 'POST_INSTALL' if behavior == :invoke
end
end
end
end
end
gitlab-experiment-0.6.5/lib/generators/gitlab/experiment/USAGE 0000644 0000041 0000041 00000001407 14147762242 024335 0 ustar www-data www-data Description:
Stubs out a new experiment and its variants. Pass the experiment name,
either CamelCased or under_scored, and a list of variants as arguments.
To create an experiment within a module, specify the experiment name as a
path like 'parent_module/experiment_name'.
This generates an experiment class in app/experiments and invokes feature
flag, and test framework generators.
Example:
`rails generate gitlab:experiment NullHypothesis control candidate alt_variant`
NullHypothesis experiment with default variants.
Experiment: app/experiments/null_hypothesis_experiment.rb
Feature Flag: config/feature_flags/experiment/null_hypothesis.yaml
Test: test/experiments/null_hypothesis_experiment_test.rb
gitlab-experiment-0.6.5/lib/generators/test_unit/ 0000755 0000041 0000041 00000000000 14147762242 022140 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/test_unit/experiment/ 0000755 0000041 0000041 00000000000 14147762242 024320 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/test_unit/experiment/experiment_generator.rb 0000644 0000041 0000041 00000000724 14147762242 031076 0 ustar www-data www-data # frozen_string_literal: true
require 'rails/generators/test_unit'
module TestUnit # :nodoc:
module Generators # :nodoc:
class ExperimentGenerator < TestUnit::Generators::Base # :nodoc:
source_root File.expand_path('templates/', __dir__)
check_class_collision suffix: 'Test'
def create_test_file
template 'experiment_test.rb', File.join('test/experiments', class_path, "#{file_name}_experiment_test.rb")
end
end
end
end
gitlab-experiment-0.6.5/lib/generators/test_unit/experiment/templates/ 0000755 0000041 0000041 00000000000 14147762242 026316 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/test_unit/experiment/templates/experiment_test.rb.tt 0000644 0000041 0000041 00000000324 14147762242 032507 0 ustar www-data www-data # frozen_string_literal: true
require 'test_helper'
<% module_namespacing do -%>
class <%= class_name %>ExperimentTest < ActiveSupport::TestCase
# test "the truth" do
# assert true
# end
end
<% end -%>
gitlab-experiment-0.6.5/lib/generators/rspec/ 0000755 0000041 0000041 00000000000 14147762242 021236 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/rspec/experiment/ 0000755 0000041 0000041 00000000000 14147762242 023416 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/rspec/experiment/experiment_generator.rb 0000644 0000041 0000041 00000000600 14147762242 030165 0 ustar www-data www-data # frozen_string_literal: true
require 'generators/rspec'
module Rspec
module Generators
class ExperimentGenerator < Rspec::Generators::Base
source_root File.expand_path('templates/', __dir__)
def create_experiment_spec
template 'experiment_spec.rb', File.join('spec/experiments', class_path, "#{file_name}_experiment_spec.rb")
end
end
end
end
gitlab-experiment-0.6.5/lib/generators/rspec/experiment/templates/ 0000755 0000041 0000041 00000000000 14147762242 025414 5 ustar www-data www-data gitlab-experiment-0.6.5/lib/generators/rspec/experiment/templates/experiment_spec.rb.tt 0000644 0000041 0000041 00000000312 14147762242 031555 0 ustar www-data www-data # frozen_string_literal: true
require 'rails_helper'
<% module_namespacing do -%>
RSpec.describe <%= class_name %>Experiment do
pending "add some examples to (or delete) #{__FILE__}"
end
<% end -%>
gitlab-experiment-0.6.5/LICENSE.txt 0000644 0000041 0000041 00000002374 14147762242 017034 0 ustar www-data www-data Copyright (c) 2020-2022 GitLab B.V.
With regard to the GitLab Software:
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
For all third party components incorporated into the GitLab Software, those
components are licensed under the original license provided by the owner of the
applicable component.