gitlab-experiment-0.9.1/ 0000755 0000041 0000041 00000000000 14535544660 015205 5 ustar www-data www-data gitlab-experiment-0.9.1/README.md 0000644 0000041 0000041 00000104011 14535544660 016461 0 ustar www-data www-data GitLab Experiment
=================
This README represents the current main branch and may not be applicable to the release you're using in your project. Please refer to the correct release branch if you'd like to review documentation relevant for that release.
Here at GitLab we run experiments as A/B/n tests and review the data the experiment generates. From that data, we determine the best performing code path and promote it as the new default code path, or revert back to the original code path. You can read our [Experiment Guide](https://docs.gitlab.com/ee/development/experiment_guide/) documentation if you're curious about how we use this gem internally at GitLab.
This library provides a clean and elegant DSL (domain specific language) to define, run, and track your experiments.
When we discuss the behavior of this gem, we'll use terms like experiment, context, control, candidate, and variant. It's worth defining these terms so they're more understood.
- `experiment` is any deviation of code paths we want to run sometimes and not others.
- `context` is used to identify a consistent experience we'll provide in an experiment.
- `control` is the default, or "original" code path.
- `candidate` defines that there's one experimental code path.
- `variant(s)` is used when more than one experimental code path exists.
- `behaviors` is used to reference all possible code paths of an experiment.
Candidate and variant are the same concept, but simplify how we speak about experimental paths -- both are widely referred to as the "experiment group". If you use "control and candidate," the assumption is an A/B test, and if you say "variants" the assumption is a multiple variant experiment.
Behaviors is a general term for all code paths -- if that's control and candidate, or control and variants. It's all of them.
[[_TOC_]]
## Installation
Add the gem to your Gemfile and then `bundle install`.
```ruby
gem 'gitlab-experiment'
```
If you're using Rails, you can install the initializer which provides basic configuration, documentation, and the base experiment class that all your experiments can inherit from.
```shell
$ rails generate gitlab:experiment:install
```
## Implementing an experiment
For the sake of having a simple example let's define an experiment around a button color, even if it's not how we'd likely implement a real experiment it makes for a clean example.
Currently our button is blue, but we want to test out a red variant. Our hypothesis is that a red button will make it more visible but also might appear like a warning button -- this is why we're testing our hypothesis.
So let's name our experiment `pill_color`, and use the generator to get some files:
```shell
$ rails generate gitlab:experiment pill_color
```
This generator will give us an `app/experiments/pill_color_experiment.rb` file, which we can use to define our experiment class and register our default variant behaviors. The generator also provides a bunch of useful comments in this file unless you `--skip-comments`, so feel free to play around with that.
Let's fill out our control and candidate behaviors with some default values to start. For our example it'll just be some made up class names where we've defined our colors.
```ruby
class PillColorExperiment < ApplicationExperiment
control { 'blue' } # register and define our default control value
candidate { 'red' } # register and define our experimental candidate value
end
```
Now that we've defined our default behaviors, we can utilize our experiment elsewhere by calling the `experiment` method. When you run (or publish) an experiment you'll need to provide a name and a context. The name is how our `PillColorExperiment` class is resolved and will also show up in event data downstream. The context is a hash that's used to determine the variant assigned and should be consistent between calls.
For our experiment we're going to make it "sticky" to the current user -- and if there isn't a current user, we want to assign and use a cookie value instead. This happens automatically if you use the [`actor` keyword](#cookies-and-the-actor-keyword) in your context. We use that in a lot of our examples, but it's by no means how everything should be done.
```haml
%button{ class: experiment(:pill_color, actor: current_user).run } Click Me!
```
Now when our view is rendered, the class attribute of the button will be pulled from the experiment and will be sticky to the user that's viewing it.
You can also provide behavior overrides when you run the experiment. Let's use a view helper for this example because it'll be cleaner than putting our override blocks into the view.
```ruby
def pill_color_experiment
experiment(:pill_color, actor: current_user) do |e|
e.candidate { 'purple' } # override the candidate default of 'red'
end
end
```
Now we can run the experiment using that helper method. Instead of a red button, users who are assigned the candidate will be given a purple button. Users in the control group will still see the blue button. Using experiments in this way permits a default experiment to be defined while also allowing the experiment to be customized in the places where its run, using the scope, variables and helper methods that are available to us where we want to run the experiment.
```haml
%button{ class: pill_color_experiment.run } Click Me!
```
Understanding how an experiment can change user behavior or engagement is important in evaluating its performance. To this end, you'll probably want to track events that are important elsewhere in code. By using the same context you can provide a consistent experience and have the ability to anonymously track events to that experience.
```ruby
experiment(:pill_color, actor: current_user).track(:clicked)
```
## Advanced experimentation
Now let's create a more complex experiment with multiple variants. This time we'll start with a grey button, and have red and blue variants.
We'll also do more advanced segmentation when the experiment is run, exclude certain cases, and register callbacks that will be executed after our experiment is run. Here are some examples of what we can start to do in our experiment classes:
```ruby
class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
# Register our behaviors.
control # register our control, which will by default call #control_behavior
variant(:red) # register the red variant that will call #red_behavior
variant(:blue) # register the blue variant that will call #blue_behavior
# Exclude any users that are named "Richard".
exclude :users_named_richard
# Segment any account older than 2 weeks into the red variant without asking
# the rollout strategy to assign a variant.
segment :old_account?, variant: :red
# After the experiment has been run, we want to log some performance metrics.
after_run { log_performance_metrics }
private
# Define the default control behavior, which can be overridden on runs.
def control_behavior
'grey'
end
# Define the default red behavior, which can be overridden on runs.
def red_behavior
'red'
end
# Define the default blue behavior, which can be overridden on runs.
def blue_behavior
'blue'
end
# Define our special exclusion logic.
def users_named_richard
context.try(:actor)&.first_name == 'Richard' # use try for nil actors
end
# Define our segmentation logic.
def old_account?
context.try(:actor) && context.actor.created_at < 2.weeks.ago
end
# Let's assume that we've tracked this and want to push it into some system.
def log_performance_metrics
# ...hypothetical implementation
end
end
```
You can play around with our new `PillColorExperiment` using a console or irb session. In our example we've used an actual `User` model with a first name and timestamps. Feel free to use something more appropriate for your project in your exploration, or just not pass in `actor` at all.
```ruby
include Gitlab::Experiment::Dsl
# The class will be looked up based on the experiment name provided.
ex = experiment(:pill_color, actor: User.first) # => #
# Run the experiment -- returning the result.
ex.run # => "grey" (the value defined in our control)
# Track an event on the experiment we've defined, using the logic we've defined
# in configuration.
ex.track(:clicked) # => true
# Publish the experiment without running it, using the logic we've defined in
# configuration.
ex.publish # => {:variant=>"control", :experiment=>"pill_color", :key=>"45f595...", :excluded=>false}
```
You can also specify the variant manually...
Generally, defining segmentation rules is a better way to approach routing into specific variants, but it's possible to explicitly specify the variant when running an experiment.
Caching: It's important to understand what this might do to your data during rollout, so use this with careful consideration. Any time a specific variant is assigned manually, or through segmentation (including `:control`) it will be cached for that context. That means that if you manually assign `:control`, that context will never be moved out of the control unless you do it programmatically elsewhere.
```ruby
include Gitlab::Experiment::Dsl
# Assign the candidate manually.
ex = experiment(:pill_color, :red, actor: User.first) # => #
# Run the experiment -- returning the result.
ex.run # => "red"
# If caching is enabled this will remain sticky between calls.
experiment(:pill_color, actor: User.first).run # => "red"
```
### Exclusion rules
Exclusion rules let us determine if a context should even be considered as something to include in an experiment. If we're excluding something, it means that we don't want to run the experiment in that case. This can be useful if you only want to run experiments on new users for instance.
```ruby
class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
# ...registered behaviors
exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
end
```
In the previous example, we'll exclude all users named `'Richard'` as well as any account older than 2 weeks old. Not only will they be immediately given the control behavior, but no events will be tracked in these cases either.
Exclusion rules are executed in the order they're defined. The first exclusion rule to produce a truthy result will halt execution of further exclusion checks.
Note: Although tracking calls will be ignored on all exclusions, you may want to check exclusion yourself in expensive custom logic by calling the `should_track?` or `excluded?` methods.
Note: When using exclusion rules it's important to understand that the control assignment is cached, which improves future experiment run performance but can be a gotcha around caching.
Note: Exclusion rules aren't the best way to determine if an experiment is enabled. There's an `enabled?` method that can be overridden to have a high-level way of determining if an experiment should be running and tracking at all. This `enabled?` check should be as efficient as possible because it's the first early opt out path an experiment can implement. This can be seen in [How it works](#how-it-works).
### Segmentation rules
Segmentation, or assigning certain variants in certain cases, is important to running experiments. This can be useful if you want to push a given population into a specific variant because you've already determined that variant is successful for that population.
```ruby
class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
# ...registered behaviors
segment(variant: :red) { context.actor.first_name == 'Richard' }
segment :old_account?, variant: :blue
end
```
In the previous example, any user named `'Richard'` would always receive the experience defined in the red variant. As well, any account older than 2 weeks would get the alternate experience defined in the blue variant.
Segmentation rules are executed in the order they're defined. The first segmentation rule to produce a truthy result is the one which gets used to assign the variant. The remaining segmentation rules are skipped. This means that for our example, any user named `'Richard'` regardless of account age, will always be provided the experience as defined in the red variant.
### Run callbacks
Callbacks can be registered for when you want to execute logic before, after, or around when an experiment is run. These callbacks won't be called unless the experiment is actually run, meaning that exclusion rules take precedence.
```ruby
class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
# ...registered behaviors
after_run :log_performance_metrics, -> { publish_to_database }
end
```
In the previous example, we're going to call the `log_performance_method`, and do a hypothetical publish to the database. If you want to do an `around_run`, you just need to call the block:
```ruby
class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
# ...registered behaviors
around_run do |experiment, block|
puts "- before #{experiment.name} run"
block.call
puts "- after #{experiment.name} run"
end
end
```
### Rollout strategies
While a default rollout strategy can be defined in configuration, it's useful to be able to override this per experiment if needed. You can do this by specifying a specific `default_rollout` override in your experiment class.
```ruby
class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
# ...registered behaviors
default_rollout :random # randomly assign one of the registered behaviors
end
```
Obviously random assignment might not be the best rollout strategy for you, but you can define your own rollout strategies, or use one of the ones provided in the gem. You can read more about configuring the default rollout and how to write your own rollout strategies in the [configuration documentation](#rollout-strategies-1) for it.
## How it works
The way experiments work is best described using the following decision tree diagram. When an experiment is run, the following logic is executed to resolve what experience should be provided, given how the experiment is defined, and using the context passed to the experiment call.
```mermaid
graph TD
GP[General Pool/Population] --> Running?[Rollout Enabled?]
Running? -->|Yes| Cached?[Cached? / Pre-segmented?]
Running? -->|No| Excluded[Control / No Tracking]
Cached? -->|No| Excluded?
Cached? -->|Yes| Cached[Cached Value]
Excluded? -->|Yes / Cached| Excluded
Excluded? -->|No| Segmented?
Segmented? -->|Yes / Cached| VariantA
Segmented? -->|No| Rollout[Rollout Resolve]
Rollout --> Control
Rollout -->|Cached| VariantA
Rollout -->|Cached| VariantB
Rollout -->|Cached| VariantN
classDef included fill:#380d75,color:#ffffff,stroke:none
classDef excluded fill:#fca121,stroke:none
classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
classDef default fill:#fff,stroke:#6e49cb
class VariantA,VariantB,VariantN included
class Control,Excluded excluded
class Cached cached
```
## Technical details
This library is intended to be powerful and easy to use, which can lead to some complex underpinnings in the implementation. Some of those implementation details are important to understand at a technical level when considering how you want to design your experiments.
### Including the DSL
By default, `Gitlab::Experiment` injects itself into the controller, view, and mailer layers. This exposes the `experiment` method application wide in those layers. Some experiments may extend outside of those layers however, so you may want to include it elsewhere. For instance in an irb session or the rails console, or in all your service objects, background jobs, or similar.
```ruby
class ApplicationJob < ActiveJob::Base
include Gitlab::Experiment::Dsl # include the `experiment` method for all jobs
end
```
### Experiment stickiness
Internally, experiments have what's referred to as the context "key" that represents the unique and anonymous id of a given context. This allows us to assign the same variant between different calls to the experiment, is used in caching and can be used in event data downstream. This context "key" is how an experiment remains "sticky" to a given context, and is an important aspect to understand.
You can specify what the experiment should be sticky to by providing the `:sticky_to` option. By default this will be the entire context, but this can be overridden manually if needed.
In a fabricated example, we might want to provide the user and a project to an experiment, but we want it to remain sticky to the project that the user is viewing and not the user viewing that project. This is a very powerful concept that you're encouraged to play around with to understand what it means, and how you can run complex experiments that aren't user centric.
```ruby
experiment(:example, actor: current_user, project: project, sticky_to: project)
```
### Experiment signature
The best way to understand the details of an experiment is through its signature. An example signature can be retrieved by calling the `signature` method, and looks like the following:
```ruby
experiment(:example).signature # => {:variant=>"control", :experiment=>"example", :key=>"4d7aee..."}
```
An experiment signature is useful when tracking events and when using experiments on the client layer. The signature can also contain the optional `migration_keys`, and `excluded` properties.
### Return value
By default the return value of calling `experiment` is a `Gitlab::Experiment` instance, or whatever class the experiment is resolved to, which likely inherits from `Gitlab::Experiment`. In simple cases you may want only the results of running the experiment though. You can call `run` within the block to get the return value of the assigned variant.
```ruby
# Normally an experiment instance.
experiment(:example) do |e|
e.control { 'A' }
e.candidate { 'B' }
end # => #
# But calling `run` causes the return value to be the result.
experiment(:example) do |e|
e.control { 'A' }
e.candidate { 'B' }
e.run
end # => 'A'
```
### Context migrations
There are times when we need to change context while an experiment is running. We make this possible by passing the migration data to the experiment.
Take for instance, that you might be using `version: 1` in your context currently. To migrate this to `version: 2`, provide the portion of the context you wish to change using a `migrated_with` option.
In providing the context migration data, we can resolve an experience and its events all the way back. This can also help in keeping our cache relevant.
```ruby
# First implementation.
experiment(:example, actor: current_user, version: 1)
# Migrate just the `:version` portion.
experiment(:example, actor: current_user, version: 2, migrated_with: { version: 1 })
```
You can add or remove context by providing a `migrated_from` option. This approach expects a full context replacement -- i.e. what it was before you added or removed the new context key.
If you wanted to introduce a `version` to your context, provide the full previous context.
```ruby
# First implementation.
experiment(:example, actor: current_user)
# Migrate the full context of `{ actor: current_user }` to `{ actor: current_user, version: 1 }`.
experiment(:example, actor: current_user, version: 1, migrated_from: { actor: current_user })
```
When you migrate context, this information is included in the signature of the experiment. This can be used downstream in event handling and reporting to resolve a series of events back to a single experience, while also keeping everything anonymous.
An example of our experiment signature when we migrate would include the `migration_keys` property:
```ruby
ex = experiment(:example, version: 1)
ex.signature # => {:key=>"20d69a...", ...}
ex = experiment(:example, version: 2, migrated_from: { version: 1 })
ex.signature # => {:key=>"9e9d93...", :migration_keys=>["20d69a..."], ...}
```
### Cookies and the actor keyword
We use cookies to auto migrate an unknown value into a known value, often in the case of the current user. The implementation of this uses the same concept outlined above with context migrations, but will happen automatically for you if you use the `actor` keyword.
When you use the `actor: current_user` pattern in your context, the nil case is handled by setting a special cookie for the experiment and then deleting the cookie, and migrating the context key to the one generated from the user when they've signed in.
This cookie is a temporary, randomized uuid and isn't associated with a user. When we can finally provide an actor, the context is auto migrated from the cookie to that actor.
```ruby
# The actor key is not present, so no cookie is set.
experiment(:example, project: project)
# The actor key is present but nil, so the cookie is set and used.
experiment(:example, actor: nil, project: project)
# The actor key is present and isn't nil, so the cookie value (if found) is
# migrated forward and the cookie is deleted.
experiment(:example, actor: current_user, project: project)
```
Note: The cookie is deleted when resolved, but can be assigned again if the `actor` is ever nil again. A good example of this scenario would be on a sign in page. When a potential user arrives, they would never be known, so a cookie would be set for them, and then resolved/removed as soon as they signed in. This process would repeat each time they arrived while not being signed in and can complicate reporting unless it's handled well in the data layers.
Note: To read and write cookies, we provide the `request` from within the controller and views. The cookie migration will happen automatically if the experiment is within those layers. You'll need to provide the `request` as an option to the experiment if it's outside of the controller and views.
```ruby
experiment(:example, actor: current_user, request: request)
```
Note: For edge cases, you can pass the cookie through by assigning it yourself -- e.g. `actor: request.cookie_jar.signed['example_id']`. The cookie name is the full experiment name (including any configured prefix) with `_id` appended -- e.g. `pill_color_id` for the `PillColorExperiment`.
### Client layer
Experiments that have been run (or published) during the request lifecycle can be pushed into to the client layer by injecting the published experiments into javascript in a layout or view using something like:
```haml
= javascript_tag(nonce: content_security_policy_nonce) do
window.experiments = #{raw ApplicationExperiment.published_experiments.to_json};
```
The `window.experiments` object can then be used in your client implementation to determine experimental behavior at that layer as well. For instance, we can now access the `window.experiments.pill_color` object to get the variant that was assigned, if the context was excluded, and to use the context key in our client side events.
## Configuration
The gem is meant to be configured when installed and is ambiguous about how it should track events, and what to do when publishing experiments. Some of the more important aspects are left up to you to implement the logic that's right for your project.
Simple documentation can be found in the provided [initializer](lib/generators/gitlab/experiment/install/templates/initializer.rb.tt).
Read on for comprehensive documentation on some of the more complex configuration options.
### Caching
Caching can be enabled in configuration and is implemented towards the `Rails.cache` / `ActiveSupport::Cache::Store` interface. When you enable caching, any variant resolution will be cached when something other than nil. Migrating the cache through context migrations is handled automatically, and this helps ensure an experiment experience remains consistent.
It's important to understand that using caching can drastically change or negate your specific rollout strategy logic.
```ruby
Gitlab::Experiment.configure do |config|
config.cache = Rails.cache
end
```
The gem includes the following cache stores, which are documented in the implementation:
- [`RedisHashStore`](lib/gitlab/experiment/cache/redis_hash_store.rb): Useful if using redis
### Rollout strategies
There are some basic rollout strategies that come with the gem, and you can use these directly, or you can use them to help build your own custom rollout strategies. Each is documented more thoroughly in its implementation file.
- [`Base`](lib/gitlab/experiment/rollout.rb): Useful for building custom rollout strategies, not super useful by itself
- [`Percent`](lib/gitlab/experiment/rollout/percent.rb): A comprehensive percent based strategy, it's configured as the default
- [`Random`](lib/gitlab/experiment/rollout/random.rb): Random assignment can be useful on some experimentation
- [`RoundRobin`](lib/gitlab/experiment/rollout/round_robin.rb): Cycles through assignment using the cache to keep track of what was last assigned
The included rollout strategies are great, but you might want to write your own for your own project, which might already have nice tooling for toggling feature flags.
It's an important aspect of this library to be flexible with the approach you choose when determining if an experiment is enabled, and what variant should be assigned.
To that end, let's go ahead through an example of how to write up a custom rollout strategy that uses the [Flipper](https://github.com/jnunemaker/flipper) gem to manage rollout. For simpliticy in our example, we're going to start by inheriting from `Gitlab::Experiment::Rollout::Percent` because it's already useful.
```ruby
# We put it in this module namespace so we can get easy resolution when
# using `default_rollout :flipper` in our usage later.
module Gitlab::Experiment::Rollout
class Flipper < Percent
def enabled?
::Flipper.enabled?(experiment.name, self)
end
def flipper_id
"Experiment;#{id}"
end
end
end
```
So, Flipper needs something that responds to `flipper_id`, and since our experiment "id" (which is also our context key) is unique and consistent, we're going to give that to Flipper to manage things like percentage of actors etc. You might want to consider something more complex here if you're using things that can be flipper actors in your experiment context.
Anyway, now you can use your custom `Flipper` rollout strategy by instantiating it in configuration:
```ruby
Gitlab::Experiment.configure do |config|
config.default_rollout = Gitlab::Experiment::Rollout::Flipper.new
end
```
Or if you don't want to make that change globally, you can use it in specific experiment classes:
```ruby
class PillColorExperiment < Gitlab::Experiment # OR ApplicationExperiment
# ...registered behaviors
default_rollout :flipper,
distribution: { control: 26, red: 37, blue: 37 } # optionally specify distribution
end
```
Now, enabling or disabling the Flipper feature flag will control if the experiment is enabled or not. If the experiment is enabled, as determined by our custom rollout strategy, the standard resolutuon logic will be executed, and a variant (or control) will be assigned.
```ruby
experiment(:pill_color).enabled? # => false
experiment(:pill_color).assigned.name # => "control"
# Now we can enable the feature flag to enable the experiment.
Flipper.enable(:pill_color) # => true
experiment(:pill_color).enabled? # => true
experiment(:pill_color).assigned.name # => "red"
```
### Middleware
There are times when you'll need to do link tracking in email templates, or markdown content -- or other places you won't be able to implement tracking. For these cases a middleware layer that can redirect to a given URL while also tracking that the URL was visited has been provided.
In Rails this middleware is mounted automatically, with a base path of what's been configured for `mount_at`. If this path is nil, the middleware won't be mounted at all.
```ruby
Gitlab::Experiment.configure do |config|
config.mount_at = '/experiment'
# Only redirect on permitted domains.
config.redirect_url_validator = ->(url) { (url = URI.parse(url)) && url.host == 'gitlab.com' }
end
```
Once configured to be mounted, the experiment tracking redirect URLs can be generated using the Rails route helpers.
```ruby
ex = experiment(:example)
# Generating the path/url using the path and url helper.
experiment_redirect_path(ex, url: 'https//gitlab.com/docs') # => "/experiment/example:20d69a...?https//gitlab.com/docs"
experiment_redirect_url(ex, url: 'https//gitlab.com/docs') # => "https://gitlab.com/experiment/example:20d69a...?https//gitlab.com/docs"
# Manually generating a url is a bit less clean, but is possible.
"#{Gitlab::Experiment::Configuration.mount_at}/#{ex.to_param}?https//docs.gitlab.com/"
```
## Testing (rspec support)
This gem comes with some rspec helpers and custom matchers. To get the experiment specific rspec support, require the rspec support file:
```ruby
require 'gitlab/experiment/rspec'
```
Any file in `spec/experiments` path will automatically get the experiment specific support, but it can also be included in other specs by adding the `:experiment` label:
```ruby
describe MyExampleController do
context "with my experiment", :experiment do
# experiment helpers and matchers will be available here.
end
end
```
### Stub helpers
You can stub experiment variant resolution using the `stub_experiments` helper. Pass a hash of experiment names and the variant each should resolve to.
```ruby
it "stubs experiments to resolve to a specific variant" do
stub_experiments(pill_color: :red)
experiment(:pill_color) do |e|
expect(e).to be_enabled
expect(e.assigned.name).to eq('red')
end
end
```
In special cases you can use a boolean `true` instead of a variant name. This allows the rollout strategy to resolve the variant however it wants to, but is otherwise just making sure the experiment is considered enabled.
```ruby
it "stubs experiments while allowing the rollout strategy to assign the variant" do
stub_experiments(pill_color: true) # only stubs enabled?
experiment(:pill_color) do |e|
expect(e).to be_enabled
# expect(e.assigned.name).to eq([whatever the rollout strategy assigns])
end
end
```
### Registered behaviors matcher
It's useful to test our registered behaviors, as well as their return values when we implement anything complex in them. The `register_behavior` matcher is useful for this.
```ruby
it "tests our registered behaviors" do
expect(experiment(:pill_color)).to register_behavior(:control)
.with('grey') # with a default return value of "grey"
expect(experiment(:pill_color)).to register_behavior(:red)
expect(experiment(:pill_color)).to register_behavior(:blue)
end
```
### Exclusion and segmentation matchers
You can also easily test your experiment classes using the `exclude`, `segment` metchers.
```ruby
let(:excluded) { double(first_name: 'Richard', created_at: Time.current) }
let(:segmented) { double(first_name: 'Jeremy', created_at: 3.weeks.ago) }
it "tests the exclusion rules" do
expect(experiment(:pill_color)).to exclude(actor: excluded)
expect(experiment(:pill_color)).not_to exclude(actor: segmented)
end
it "tests the segmentation rules" do
expect(experiment(:pill_color)).to segment(actor: segmented)
.into(:red) # into a specific variant
expect(experiment(:pill_color)).not_to segment(actor: excluded)
end
```
### Tracking matcher
Tracking events is a major aspect of experimentation, and because of this we try to provide a flexible way to ensure your tracking calls are covered.
```ruby
before do
stub_experiments(pill_color: true) # stub the experiment so tracking is permitted
end
it "tests that we track an event on a specific instance" do
expect(subject = experiment(:pill_color)).to track(:clicked)
subject.track(:clicked)
end
```
You can use the `on_next_instance` chain method to specify that the tracking call could happen on the next instance of the experiment. This can be useful if you're calling `experiment(:example).track` downstream and don't have access to that instance. Here's a full example of the methods that can be chained onto the `track` matcher:
```ruby
it "tests that we track an event with specific details" do
expect(experiment(:pill_color)).to track(:clicked, value: 1, property: '_property_')
.on_next_instance # any time in the future
.with_context(foo: :bar) # with the expected context
.for(:red) # and assigned the correct variant
experiment(:pill_color, :red, foo: :bar).track(:clicked, value: 1, property: '_property_')
end
```
## Tracking, anonymity and GDPR
We generally try not to track things like user identifying values in our experimentation. What we can and do track is the "experiment experience" (a.k.a. the context key).
We generate this key from the context passed to the experiment. This allows creating funnels without exposing any user information.
This library attempts to be non-user-centric, in that a context can contain things like a user or a project.
If you only include a user, that user would get the same experience across every project they view. If you only include the project, every user who views that project would get the same experience.
Each of these approaches could be desirable given the objectives of your experiment.
## Development
After cloning the repo, run `bundle install` to install dependencies.
Run `bundle exec rake` to run the tests. You can also run `bundle exec pry` for an interactive prompt that will allow you to experiment.
## Contributing
Bug reports and merge requests are welcome on GitLab at https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
## Release process
Please refer to the [Release Process](docs/release_process.md).
## License
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
## Code of conduct
Everyone interacting in the `Gitlab::Experiment` project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](CODE_OF_CONDUCT.md).
***Make code not war***
gitlab-experiment-0.9.1/gitlab-experiment.gemspec 0000644 0000041 0000041 00000012173 14535544660 022176 0 ustar www-data www-data #########################################################
# This file has been automatically generated by gem2tgz #
#########################################################
# -*- encoding: utf-8 -*-
# stub: gitlab-experiment 0.9.1 ruby lib
Gem::Specification.new do |s|
s.name = "gitlab-experiment".freeze
s.version = "0.9.1"
s.required_rubygems_version = Gem::Requirement.new(">= 0".freeze) if s.respond_to? :required_rubygems_version=
s.require_paths = ["lib".freeze]
s.authors = ["GitLab".freeze]
s.date = "2023-10-25"
s.email = ["gitlab_rubygems@gitlab.com".freeze]
s.files = ["LICENSE.txt".freeze, "README.md".freeze, "lib/generators/gitlab".freeze, "lib/generators/gitlab/experiment".freeze, "lib/generators/gitlab/experiment/USAGE".freeze, "lib/generators/gitlab/experiment/experiment_generator.rb".freeze, "lib/generators/gitlab/experiment/install".freeze, "lib/generators/gitlab/experiment/install/install_generator.rb".freeze, "lib/generators/gitlab/experiment/install/templates".freeze, "lib/generators/gitlab/experiment/install/templates/POST_INSTALL".freeze, "lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt".freeze, "lib/generators/gitlab/experiment/install/templates/initializer.rb.tt".freeze, "lib/generators/gitlab/experiment/templates".freeze, "lib/generators/gitlab/experiment/templates/experiment.rb.tt".freeze, "lib/generators/rspec".freeze, "lib/generators/rspec/experiment".freeze, "lib/generators/rspec/experiment/experiment_generator.rb".freeze, "lib/generators/rspec/experiment/templates".freeze, "lib/generators/rspec/experiment/templates/experiment_spec.rb.tt".freeze, "lib/generators/test_unit".freeze, "lib/generators/test_unit/experiment".freeze, "lib/generators/test_unit/experiment/experiment_generator.rb".freeze, "lib/generators/test_unit/experiment/templates".freeze, "lib/generators/test_unit/experiment/templates/experiment_test.rb.tt".freeze, "lib/gitlab/experiment".freeze, "lib/gitlab/experiment.rb".freeze, "lib/gitlab/experiment/base_interface.rb".freeze, "lib/gitlab/experiment/cache".freeze, "lib/gitlab/experiment/cache.rb".freeze, "lib/gitlab/experiment/cache/redis_hash_store.rb".freeze, "lib/gitlab/experiment/callbacks.rb".freeze, "lib/gitlab/experiment/configuration.rb".freeze, "lib/gitlab/experiment/context.rb".freeze, "lib/gitlab/experiment/cookies.rb".freeze, "lib/gitlab/experiment/dsl.rb".freeze, "lib/gitlab/experiment/engine.rb".freeze, "lib/gitlab/experiment/errors.rb".freeze, "lib/gitlab/experiment/middleware.rb".freeze, "lib/gitlab/experiment/nestable.rb".freeze, "lib/gitlab/experiment/rollout".freeze, "lib/gitlab/experiment/rollout.rb".freeze, "lib/gitlab/experiment/rollout/percent.rb".freeze, "lib/gitlab/experiment/rollout/random.rb".freeze, "lib/gitlab/experiment/rollout/round_robin.rb".freeze, "lib/gitlab/experiment/rspec.rb".freeze, "lib/gitlab/experiment/test_behaviors".freeze, "lib/gitlab/experiment/test_behaviors/trackable.rb".freeze, "lib/gitlab/experiment/variant.rb".freeze, "lib/gitlab/experiment/version.rb".freeze]
s.homepage = "https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment".freeze
s.licenses = ["MIT".freeze]
s.required_ruby_version = Gem::Requirement.new(">= 2.6".freeze)
s.rubygems_version = "3.3.15".freeze
s.summary = "GitLab experimentation library.".freeze
if s.respond_to? :specification_version then
s.specification_version = 4
end
if s.respond_to? :add_runtime_dependency then
s.add_runtime_dependency(%q.freeze, [">= 3.0"])
s.add_development_dependency(%q.freeze, ["~> 0.26.2"])
s.add_development_dependency(%q.freeze, ["~> 0.9.4"])
s.add_development_dependency(%q.freeze, ["~> 4.1.0"])
s.add_development_dependency(%q.freeze, ["~> 10.1.0"])
s.add_development_dependency(%q.freeze, ["~> 1.4.7"])
s.add_development_dependency(%q.freeze, ["~> 3.10.1"])
s.add_runtime_dependency(%q.freeze, [">= 1.0"])
s.add_development_dependency(%q.freeze, ["~> 1.0.0"])
s.add_development_dependency(%q.freeze, ["~> 6.0.3"])
s.add_development_dependency(%q.freeze, ["~> 2.20.2"])
s.add_development_dependency(%q.freeze, ["~> 2.22.0"])
s.add_development_dependency(%q.freeze, ["~> 2.1.0"])
else
s.add_dependency(%q.freeze, [">= 3.0"])
s.add_dependency(%q.freeze, ["~> 0.26.2"])
s.add_dependency(%q.freeze, ["~> 0.9.4"])
s.add_dependency(%q.freeze, ["~> 4.1.0"])
s.add_dependency(%q.freeze, ["~> 10.1.0"])
s.add_dependency(%q.freeze, ["~> 1.4.7"])
s.add_dependency(%q.freeze, ["~> 3.10.1"])
s.add_dependency(%q.freeze, [">= 1.0"])
s.add_dependency(%q.freeze, ["~> 1.0.0"])
s.add_dependency(%q.freeze, ["~> 6.0.3"])
s.add_dependency(%q.freeze, ["~> 2.20.2"])
s.add_dependency(%q.freeze, ["~> 2.22.0"])
s.add_dependency(%q.freeze, ["~> 2.1.0"])
end
end
gitlab-experiment-0.9.1/lib/ 0000755 0000041 0000041 00000000000 14535544660 015753 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/gitlab/ 0000755 0000041 0000041 00000000000 14535544660 017215 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/gitlab/experiment.rb 0000644 0000041 0000041 00000012437 14535544660 021731 0 ustar www-data www-data # frozen_string_literal: true
require 'request_store'
require 'active_support'
require 'active_support/core_ext/module/delegation'
require 'active_support/core_ext/object/blank'
require 'active_support/core_ext/string/inflections'
require 'gitlab/experiment/errors'
require 'gitlab/experiment/base_interface'
require 'gitlab/experiment/cache'
require 'gitlab/experiment/callbacks'
require 'gitlab/experiment/rollout'
require 'gitlab/experiment/configuration'
require 'gitlab/experiment/cookies'
require 'gitlab/experiment/context'
require 'gitlab/experiment/dsl'
require 'gitlab/experiment/middleware'
require 'gitlab/experiment/nestable'
require 'gitlab/experiment/variant'
require 'gitlab/experiment/version'
require 'gitlab/experiment/engine' if defined?(Rails::Engine)
module Gitlab
class Experiment
include BaseInterface
include Cache
include Callbacks
include Nestable
class << self
# Class level behavior registration methods.
def control(*filter_list, **options, &block)
variant(:control, *filter_list, **options, &block)
end
def candidate(*filter_list, **options, &block)
variant(:candidate, *filter_list, **options, &block)
end
def variant(variant, *filter_list, **options, &block)
build_behavior_callback(filter_list, variant, **options, &block)
end
# Class level callback registration methods.
def exclude(*filter_list, **options, &block)
build_exclude_callback(filter_list.unshift(block), **options)
end
def segment(*filter_list, variant:, **options, &block)
build_segment_callback(filter_list.unshift(block), variant, **options)
end
def before_run(*filter_list, **options, &block)
build_run_callback(filter_list.unshift(:before, block), **options)
end
def around_run(*filter_list, **options, &block)
build_run_callback(filter_list.unshift(:around, block), **options)
end
def after_run(*filter_list, **options, &block)
build_run_callback(filter_list.unshift(:after, block), **options)
end
# Class level definition methods.
def default_rollout(rollout = nil, options = {})
return @_rollout ||= Configuration.default_rollout if rollout.blank?
@_rollout = Rollout.resolve(rollout, options)
end
# Class level accessor methods.
def published_experiments
RequestStore.store[:published_gitlab_experiments] || {}
end
end
def name
[Configuration.name_prefix, @_name].compact.join('_')
end
def control(&block)
variant(:control, &block)
end
def candidate(&block)
variant(:candidate, &block)
end
def variant(name, &block)
raise ArgumentError, 'name required' if name.blank?
raise ArgumentError, 'block required' unless block.present?
behaviors[name] = block
end
def context(value = nil)
return @_context if value.blank?
@_context.value(value)
@_context
end
def assigned(value = nil)
@_assigned_variant_name = cache_variant(value) if value.present?
return Variant.new(name: @_assigned_variant_name || :unresolved) if @_assigned_variant_name || @_resolving_variant
if enabled?
@_resolving_variant = true
@_assigned_variant_name = cached_variant_resolver(@_assigned_variant_name)
end
run_callbacks(segmentation_callback_chain) do
@_assigned_variant_name ||= :control
Variant.new(name: @_assigned_variant_name)
end
ensure
@_resolving_variant = false
end
def rollout(rollout = nil, options = {})
return @_rollout ||= self.class.default_rollout(nil, options).for(self) if rollout.blank?
@_rollout = Rollout.resolve(rollout, options).for(self)
end
def exclude!
@_excluded = true
end
def run(variant_name = nil)
return @_result if context.frozen?
@_result = run_callbacks(run_callback_chain) { super(assigned(variant_name).name) }
end
def publish(result = nil)
instance_exec(result, &Configuration.publishing_behavior)
(RequestStore.store[:published_gitlab_experiments] ||= {})[name] = signature.merge(excluded: excluded?)
end
def track(action, **event_args)
return unless should_track?
instance_exec(action, tracking_context(event_args).try(:compact) || {}, &Configuration.tracking_behavior)
end
def enabled?
rollout.enabled?
end
def excluded?
return @_excluded if defined?(@_excluded)
@_excluded = !run_callbacks(exclusion_callback_chain) { :not_excluded }
end
def should_track?
enabled? && context.trackable? && !excluded?
end
def signature
{ variant: assigned.name.to_s, experiment: name }.merge(context.signature)
end
def behaviors
@_behaviors ||= registered_behavior_callbacks
end
protected
def identify(object)
(object.respond_to?(:to_global_id) ? object.to_global_id : object).to_s
end
def cached_variant_resolver(provided_variant)
return :control if excluded?
result = cache_variant(provided_variant) { resolve_variant_name }
result.to_sym if result.present?
end
def resolve_variant_name
rollout.resolve
end
def tracking_context(event_args)
{}.merge(event_args)
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/ 0000755 0000041 0000041 00000000000 14535544660 021375 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/gitlab/experiment/rollout/ 0000755 0000041 0000041 00000000000 14535544660 023075 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/gitlab/experiment/rollout/random.rb 0000644 0000041 0000041 00000001662 14535544660 024707 0 ustar www-data www-data # frozen_string_literal: true
# The random rollout strategy will randomly assign a variant when the context is determined to be within the experiment
# group.
#
# If caching is enabled this is a predicable and consistent assignment that will eventually assign a variant (since
# control isn't cached) but if caching isn't enabled, assignment will be random each time.
#
# Example configuration usage:
#
# config.default_rollout = Gitlab::Experiment::Rollout::Random.new
#
# Example class usage:
#
# class PillColorExperiment < ApplicationExperiment
# control { }
# variant(:red) { }
# variant(:blue) { }
#
# # Randomize between all behaviors, with a mostly even distribution).
# default_rollout :random
# end
#
module Gitlab
class Experiment
module Rollout
class Random < Base
protected
def execute_assignment
behavior_names.sample # pick a random variant
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/rollout/round_robin.rb 0000644 0000041 0000041 00000002101 14535544660 025734 0 ustar www-data www-data # frozen_string_literal: true
# The round robin strategy will assign the next variant in the list, looping back to the first variant after all
# variants have been assigned. This is useful for very small sample sizes where very even distribution can be required.
#
# Requires a cache to be configured.
#
# Keeps track of the number of assignments into the experiment group, and uses this to rotate "round robin" style
# through the variants that are defined.
#
# Example configuration usage:
#
# config.default_rollout = Gitlab::Experiment::Rollout::RoundRobin.new
#
# Example class usage:
#
# class PillColorExperiment < ApplicationExperiment
# control { }
# variant(:red) { }
# variant(:blue) { }
#
# # Rotate evenly between all behaviors.
# default_rollout :round_robin
# end
#
module Gitlab
class Experiment
module Rollout
class RoundRobin < Base
KEY_NAME = :last_round_robin_variant
protected
def execute_assignment
behavior_names[(cache.attr_inc(KEY_NAME) - 1) % behavior_names.size]
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/rollout/percent.rb 0000644 0000041 0000041 00000004705 14535544660 025070 0 ustar www-data www-data # frozen_string_literal: true
require "zlib"
# The percent rollout strategy is the most comprehensive included with Gitlab::Experiment. It allows specifying the
# percentages per variant using an array, a hash, or will default to even distribution when no rules are provided.
#
# A given experiment id (context key) will always be given the same variant assignment.
#
# Example configuration usage:
#
# config.default_rollout = Gitlab::Experiment::Rollout::Percent.new
#
# Example class usage:
#
# class PillColorExperiment < ApplicationExperiment
# control { }
# variant(:red) { }
# variant(:blue) { }
#
# # Even distribution between all behaviors.
# default_rollout :percent
#
# # With specific distribution percentages.
# default_rollout :percent, distribution: { control: 25, red: 30, blue: 45 }
# end
#
module Gitlab
class Experiment
module Rollout
class Percent < Base
protected
def validate!
case distribution_rules
when nil then nil
when Array
validate_distribution_rules(distribution_rules)
when Hash
validate_distribution_rules(distribution_rules.values)
else
raise InvalidRolloutRules, 'unknown distribution options type'
end
end
def execute_assignment
crc = normalized_id
total = 0
case distribution_rules
when Array # run through the rules until finding an acceptable one
behavior_names[distribution_rules.find_index { |percent| crc % 100 <= total += percent }]
when Hash # run through the variant names until finding an acceptable one
distribution_rules.find { |_, percent| crc % 100 <= total += percent }.first
else # assume even distribution on no rules
behavior_names.empty? ? nil : behavior_names[crc % behavior_names.length]
end
end
private
def normalized_id
Zlib.crc32(id, nil)
end
def distribution_rules
options[:distribution]
end
def validate_distribution_rules(distributions)
if distributions.length != behavior_names.length
raise InvalidRolloutRules, "the distribution rules don't match the number of behaviors defined"
end
return if distributions.sum == 100
raise InvalidRolloutRules, 'the distribution percentages should add up to 100'
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/version.rb 0000644 0000041 0000041 00000000140 14535544660 023402 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
VERSION = '0.9.1'
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/cache.rb 0000644 0000041 0000041 00000003603 14535544660 022767 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Cache
autoload :RedisHashStore, 'gitlab/experiment/cache/redis_hash_store.rb'
class Interface
attr_reader :store, :key
def initialize(experiment, store)
@experiment = experiment
@store = store
@key = experiment.cache_key
end
def read
store.read(key)
end
def write(value = nil)
store.write(key, value || @experiment.assigned.name)
end
def delete
store.delete(key)
end
def attr_get(name)
store.read(@experiment.cache_key(name, suffix: :attrs))
end
def attr_set(name, value)
store.write(@experiment.cache_key(name, suffix: :attrs), value)
end
def attr_inc(name, amount = 1)
store.increment(@experiment.cache_key(name, suffix: :attrs), amount)
end
end
def cache
@cache ||= Interface.new(self, Configuration.cache)
end
def cache_variant(specified = nil, &block)
return (specified.presence || yield) unless cache.store
result = migrated_cache_fetch(cache.store, &block)
return result unless specified.present?
cache.write(specified) if result.to_s != specified.to_s
specified
end
def cache_key(key = nil, suffix: nil)
"#{[name, suffix].compact.join('_')}:#{key || context.signature[:key]}"
end
private
def migrated_cache_fetch(store, &block)
migrations = context.signature[:migration_keys]&.map { |key| cache_key(key) } || []
migrations.find do |old_key|
value = store.read(old_key)
next unless value
store.write(cache_key, value)
store.delete(old_key)
break value
end || store.fetch(cache_key, &block)
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/nestable.rb 0000644 0000041 0000041 00000001750 14535544660 023522 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Nestable
extend ActiveSupport::Concern
included do
set_callback :run, :around, :manage_nested_stack
end
def nest_experiment(nested_experiment)
instance_exec(nested_experiment, &Configuration.nested_behavior)
end
private
def manage_nested_stack
Stack.push(self)
yield
ensure
Stack.pop
end
class Stack
include Singleton
delegate :pop, :length, :size, :[], to: :stack
class << self
delegate :pop, :push, :length, :size, :[], to: :instance
end
def initialize
@thread_key = "#{self.class};#{object_id}".to_sym
end
def push(instance)
stack.last&.nest_experiment(instance)
stack.push(instance)
end
private
def stack
Thread.current[@thread_key] ||= []
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/errors.rb 0000644 0000041 0000041 00000001525 14535544660 023241 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
Error = Class.new(StandardError)
InvalidRolloutRules = Class.new(Error)
UnregisteredExperiment = Class.new(Error)
ExistingBehaviorError = Class.new(Error)
BehaviorMissingError = Class.new(Error)
class NestingError < Error
def initialize(experiment:, nested_experiment:)
messages = []
experiments = [nested_experiment, experiment]
callers = caller_locations
callers.select.with_index do |caller, index|
next if caller.label != 'experiment'
messages << " #{experiments[messages.length].name} initiated by #{callers[index + 1]}"
end
messages << ["unable to nest #{nested_experiment.name} within #{experiment.name}:"]
super(messages.reverse.join("\n"))
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/context.rb 0000644 0000041 0000041 00000004402 14535544660 023406 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
class Context
include Cookies
DNT_REGEXP = /^(true|t|yes|y|1|on)$/i.freeze
attr_reader :request
def initialize(experiment, **initial_value)
@experiment = experiment
@value = {}
@migrations = { merged: [], unmerged: [] }
value(initial_value)
end
def reinitialize(request)
@signature = nil # clear memoization
@request = request if request.respond_to?(:headers) && request.respond_to?(:cookie_jar)
end
def value(value = nil)
return @value if value.nil?
value = value.dup # dup so we don't mutate
reinitialize(value.delete(:request))
key(value.delete(:sticky_to))
@value.merge!(process_migrations(value))
end
def key(key = nil)
return @key || @experiment.key_for(value) if key.nil?
@key = @experiment.key_for(key)
end
def trackable?
!(@request && @request.headers['DNT'].to_s.match?(DNT_REGEXP))
end
def freeze
signature # finalize before freezing
super
end
def signature
@signature ||= { key: key, migration_keys: migration_keys }.compact
end
def method_missing(method_name, *)
@value.include?(method_name.to_sym) ? @value[method_name.to_sym] : super
end
def respond_to_missing?(method_name, *)
@value.include?(method_name.to_sym) ? true : super
end
private
def process_migrations(value)
add_unmerged_migration(value.delete(:migrated_from))
add_merged_migration(value.delete(:migrated_with))
migrate_cookie(value, @experiment.instance_exec(@experiment, &Configuration.cookie_name))
end
def add_unmerged_migration(value = {})
@migrations[:unmerged] << value if value.is_a?(Hash)
end
def add_merged_migration(value = {})
@migrations[:merged] << value if value.is_a?(Hash)
end
def migration_keys
return nil if @migrations[:unmerged].empty? && @migrations[:merged].empty?
@migrations[:unmerged].map { |m| @experiment.key_for(m) } +
@migrations[:merged].map { |m| @experiment.key_for(@value.merge(m)) }
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/configuration.rb 0000644 0000041 0000041 00000020040 14535544660 024565 0 ustar www-data www-data # frozen_string_literal: true
require 'singleton'
require 'logger'
require 'digest'
module Gitlab
class Experiment
class Configuration
include Singleton
# Prefix all experiment names with a given string value.
# Use `nil` for no prefix.
@name_prefix = nil
# The logger can be used to log various details of the experiments.
@logger = Logger.new($stdout)
# The base class that should be instantiated for basic experiments.
# It should be a string, so we can constantize it later.
@base_class = 'Gitlab::Experiment'
# Require experiments to be defined in a class, with variants registered.
# This will disallow any anonymous experiments that are run inline
# without previously defining a class.
@strict_registration = false
# The caching layer is expected to match the Rails.cache interface.
# If no cache is provided some rollout strategies may behave differently.
# Use `nil` for no caching.
@cache = nil
# The domain to use on cookies.
#
# When not set, it uses the current host. If you want to provide specific
# hosts, you use `:all`, or provide an array.
#
# Examples:
# nil, :all, or ['www.gitlab.com', '.gitlab.com']
@cookie_domain = :all
# The cookie name for an experiment.
@cookie_name = lambda do |experiment|
"#{experiment.name}_id"
end
# The default rollout strategy.
#
# The recommended default rollout strategy when not using caching would
# be `Gitlab::Experiment::Rollout::Percent` as that will consistently
# assign the same variant with or without caching.
#
# Gitlab::Experiment::Rollout::Base can be inherited to implement your
# own rollout strategies.
#
# Each experiment can specify its own rollout strategy:
#
# class ExampleExperiment < ApplicationExperiment
# default_rollout :random # :percent, :round_robin, or MyCustomRollout
# end
#
# Included rollout strategies:
# :percent, (recommended), :round_robin, or :random
@default_rollout = Rollout.resolve(:percent)
# Secret seed used in generating context keys.
#
# You'll typically want to use an environment variable or secret value
# for this.
#
# Consider not using one that's shared with other systems, like Rails'
# SECRET_KEY_BASE for instance. Generate a new secret and utilize that
# instead.
@context_key_secret = nil
# Bit length used by SHA2 in generating context keys.
#
# Using a higher bit length would require more computation time.
#
# Valid bit lengths:
# 256, 384, or 512
@context_key_bit_length = 256
# The default base path that the middleware (or rails engine) will be
# mounted. The middleware enables an instrumentation url, that's similar
# to links that can be instrumented in email campaigns.
#
# Use `nil` if you don't want to mount the middleware.
#
# Examples:
# '/-/experiment', '/redirect', nil
@mount_at = nil
# When using the middleware, links can be instrumented and redirected
# elsewhere. This can be exploited to make a harmful url look innocuous
# or that it's a valid url on your domain. To avoid this, you can provide
# your own logic for what urls will be considered valid and redirected
# to.
#
# Expected to return a boolean value.
@redirect_url_validator = lambda do |_redirect_url|
true
end
# Tracking behavior can be implemented to link an event to an experiment.
#
# This block is executed within the scope of the experiment and so can
# access experiment methods, like `name`, `context`, and `signature`.
@tracking_behavior = lambda do |event, args|
# An example of using a generic logger to track events:
Configuration.logger.info("#{self.class.name}[#{name}] #{event}: #{args.merge(signature: signature)}")
# Using something like snowplow to track events (in gitlab):
#
# Gitlab::Tracking.event(name, event, **args.merge(
# context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new(
# 'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0', signature
# )
# ))
end
# Logic designed to respond when a given experiment is nested within
# another experiment. This can be useful to identify overlaps and when a
# code path leads to an experiment being nested within another.
#
# Reporting complexity can arise when one experiment changes rollout, and
# a downstream experiment is impacted by that.
#
# The base_class or a custom experiment can provide a `nest_experiment`
# method that implements its own logic that may allow certain experiments
# to be nested within it.
#
# This block is executed within the scope of the experiment and so can
# access experiment methods, like `name`, `context`, and `signature`.
#
# The default exception will include the where the experiment calls were
# initiated on, so for instance:
#
# Gitlab::Experiment::NestingError: unable to nest level2 within level1:
# level1 initiated by file_name.rb:2
# level2 initiated by file_name.rb:3
@nested_behavior = lambda do |nested_experiment|
raise NestingError.new(experiment: self, nested_experiment: nested_experiment)
end
# Called at the end of every experiment run, with the result.
#
# You may want to track that you've assigned a variant to a given
# context, or push the experiment into the client or publish results
# elsewhere like into redis.
#
# This block is executed within the scope of the experiment and so can
# access experiment methods, like `name`, `context`, and `signature`.
@publishing_behavior = lambda do |_result|
# Track the event using our own configured tracking logic.
track(:assignment)
# Log using our logging system, so the result (which can be large) can
# be reviewed later if we want to.
#
# Lograge::Event.log(experiment: name, result: result, signature: signature)
# Experiments that have been run during the request lifecycle can be
# pushed to the client layer by injecting the published experiments
# into javascript in a layout or view using something like:
#
# = javascript_tag(nonce: content_security_policy_nonce) do
# window.experiments = #{raw Gitlab::Experiment.published_experiments.to_json};
end
class << self
attr_accessor(
:name_prefix,
:logger,
:base_class,
:strict_registration,
:cache,
:cookie_domain,
:cookie_name,
:context_key_secret,
:context_key_bit_length,
:mount_at,
:default_rollout,
:redirect_url_validator,
:tracking_behavior,
:nested_behavior,
:publishing_behavior
)
# Attribute method overrides.
def default_rollout=(args) # rubocop:disable Lint/DuplicateMethods
@default_rollout = Rollout.resolve(*args)
end
# Internal warning helpers.
def deprecated(*args, version:, stack: 0)
deprecator = deprecator(version)
args << args.pop.to_s.gsub('{{release}}', "#{deprecator.gem_name} #{deprecator.deprecation_horizon}")
args << caller_locations(4 + stack)
if args.length == 2
deprecator.warn(*args)
else
args[0] = "`#{args[0]}`"
deprecator.deprecation_warning(*args)
end
end
private
def deprecator(version = VERSION)
version = Gem::Version.new(version).bump.to_s
@__dep_versions ||= {}
@__dep_versions[version] ||= ActiveSupport::Deprecation.new(version, 'Gitlab::Experiment')
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/engine.rb 0000644 0000041 0000041 00000003004 14535544660 023164 0 ustar www-data www-data # frozen_string_literal: true
require 'active_model'
module Gitlab
class Experiment
include ActiveModel::Model
# Used for generating routes. We've included the method and `ActiveModel::Model` here because these things don't
# make sense outside of Rails environments.
def self.model_name
ActiveModel::Name.new(self, Gitlab)
end
class Engine < ::Rails::Engine
isolate_namespace Experiment
initializer('gitlab_experiment.include_dsl') { include_dsl }
initializer('gitlab_experiment.mount_engine') { |app| mount_engine(app, Configuration.mount_at) }
private
def include_dsl
Dsl.include_in(ActionController::API, with_helper: false) if defined?(ActionController)
Dsl.include_in(ActionController::Base, with_helper: true) if defined?(ActionController)
Dsl.include_in(ActionMailer::Base, with_helper: true) if defined?(ActionMailer)
end
def mount_engine(app, mount_at)
return if mount_at.blank?
engine = routes do
default_url_options app.routes.default_url_options.clone.without(:script_name)
resources :experiments, path: '/', only: :show
end
app.config.middleware.use(Middleware, mount_at)
app.routes.append do
mount Engine, at: mount_at, as: :experiment_engine
direct(:experiment_redirect) do |ex, options|
url = options[:url]
"#{engine.url_helpers.experiment_url(ex)}?#{url}"
end
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/dsl.rb 0000644 0000041 0000041 00000001331 14535544660 022502 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Dsl
def self.include_in(klass, with_helper: false)
klass.include(self).tap { |base| base.helper_method(:experiment) if with_helper }
end
def experiment(name, variant_name = nil, **context, &block)
raise ArgumentError, 'name is required' if name.nil?
context[:request] ||= request if respond_to?(:request)
base = Configuration.base_class.constantize
klass = base.constantize(name) || base
instance = klass.new(name, variant_name, **context, &block)
return instance unless block
instance.context.frozen? ? instance.run : instance.tap(&:run)
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/test_behaviors/ 0000755 0000041 0000041 00000000000 14535544660 024416 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/gitlab/experiment/test_behaviors/trackable.rb 0000644 0000041 0000041 00000002721 14535544660 026675 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module TestBehaviors
module Trackable
private
def manage_nested_stack
TrackedStructure.push(self)
super
ensure
TrackedStructure.pop
end
end
class TrackedStructure
include Singleton
# dependency tracking
@flat = {}
@stack = []
# structure tracking
@tree = { name: nil, count: 0, children: {} }
@node = @tree
class << self
def reset!
# dependency tracking
@flat = {}
@stack = []
# structure tracking
@tree = { name: nil, count: 0, children: {} }
@node = @tree
end
def hierarchy
@tree[:children]
end
def dependencies
@flat
end
def push(instance)
# dependency tracking
@flat[instance.name] = ((@flat[instance.name] || []) + @stack.map(&:name)).uniq
@stack.push(instance)
# structure tracking
@last = @node
@node = @node[:children][instance.name] ||= { name: instance.name, count: 0, children: {} }
@node[:count] += 1
end
def pop
# dependency tracking
@stack.pop
# structure tracking
@node = @last
end
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/variant.rb 0000644 0000041 0000041 00000000340 14535544660 023363 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
Variant = Struct.new(:name, :payload, keyword_init: true) do
def group
name == 'control' ? :control : :experiment
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/rollout.rb 0000644 0000041 0000041 00000003344 14535544660 023426 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Rollout
autoload :Percent, 'gitlab/experiment/rollout/percent.rb'
autoload :Random, 'gitlab/experiment/rollout/random.rb'
autoload :RoundRobin, 'gitlab/experiment/rollout/round_robin.rb'
def self.resolve(klass, options = {})
options ||= {}
case klass
when String
Strategy.new(klass.classify.constantize, options)
when Symbol
Strategy.new("#{name}::#{klass.to_s.classify}".constantize, options)
when Class
Strategy.new(klass, options)
else
raise ArgumentError, "unable to resolve rollout from #{klass.inspect}"
end
end
class Base
attr_reader :experiment, :options
delegate :cache, :id, to: :experiment
def initialize(experiment, options = {})
raise ArgumentError, 'you must provide an experiment instance' unless experiment.class <= Gitlab::Experiment
@experiment = experiment
@options = options
end
def enabled?
true
end
def resolve
validate! # allow the rollout strategy to validate itself
assignment = execute_assignment
assignment == :control ? nil : assignment # avoid caching control by returning nil
end
private
def validate!
# base is always valid
end
def execute_assignment
behavior_names.first
end
def behavior_names
experiment.behaviors.keys
end
end
Strategy = Struct.new(:klass, :options) do
def for(experiment)
klass.new(experiment, options)
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/cookies.rb 0000644 0000041 0000041 00000002157 14535544660 023363 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Cookies
private
def migrate_cookie(hash, cookie_name)
return hash if cookie_jar.nil?
resolver = [hash, :actor, cookie_name, cookie_jar.signed[cookie_name]]
resolve_cookie(*resolver) || generate_cookie(*resolver)
end
def cookie_jar
@request&.cookie_jar
end
def resolve_cookie(hash, key, cookie_name, cookie)
return if cookie.to_s.empty? && hash[key].nil?
return hash if cookie.to_s.empty?
return hash.merge(key => cookie) if hash[key].nil?
add_unmerged_migration(key => cookie)
cookie_jar.delete(cookie_name, domain: domain)
hash
end
def generate_cookie(hash, key, cookie_name, cookie)
return hash unless hash.key?(key)
cookie ||= SecureRandom.uuid
cookie_jar.permanent.signed[cookie_name] = {
value: cookie, secure: true, domain: domain, httponly: true
}
hash.merge(key => cookie)
end
def domain
Configuration.cookie_domain
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/cache/ 0000755 0000041 0000041 00000000000 14535544660 022440 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/gitlab/experiment/cache/redis_hash_store.rb 0000644 0000041 0000041 00000004255 14535544660 026320 0 ustar www-data www-data # frozen_string_literal: true
# This cache strategy is an implementation on top of the redis hash data type, that also adheres to the
# ActiveSupport::Cache::Store interface. It's a good example of how to build a custom caching strategy for
# Gitlab::Experiment, and is intended to be a long lived cache -- until the experiment is cleaned up.
#
# The data structure:
# key: experiment.name
# fields: context key => variant name
#
# Example configuration usage:
#
# config.cache = Gitlab::Experiment::Cache::RedisHashStore.new(
# pool: ->(&block) { block.call(Redis.current) }
# )
#
module Gitlab
class Experiment
module Cache
class RedisHashStore < ActiveSupport::Cache::Store
# Clears the entire cache for a given experiment. Be careful with this since it would reset all resolved
# variants for the entire experiment.
def clear(key:)
key = hkey(key)[0] # extract only the first part of the key
pool do |redis|
case redis.type(key)
when 'hash', 'none'
redis.del(key) # delete the primary experiment key
redis.del("#{key}_attrs") # delete the experiment attributes key
else raise ArgumentError, 'invalid call to clear a non-hash cache key'
end
end
end
def increment(key, amount = 1)
pool { |redis| redis.hincrby(*hkey(key), amount) }
end
private
def pool(&block)
raise ArgumentError, 'missing block' unless block.present?
@options[:pool].call(&block)
end
def hkey(key)
key.to_s.split(':') # this assumes the default strategy in gitlab-experiment
end
def read_entry(key, **_options)
value = pool { |redis| redis.hget(*hkey(key)) }
value.nil? ? nil : ActiveSupport::Cache::Entry.new(value)
end
def write_entry(key, entry, **_options)
return false if entry.value.blank? # don't cache any empty values
pool { |redis| redis.hset(*hkey(key), entry.value) }
end
def delete_entry(key, **_options)
pool { |redis| redis.hdel(*hkey(key)) }
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/base_interface.rb 0000644 0000041 0000041 00000006445 14535544660 024665 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module BaseInterface
extend ActiveSupport::Concern
class_methods do
def configure
yield Configuration
end
def experiment_name(name = nil, suffix: true, suffix_word: 'experiment')
name = (name.presence || self.name).to_s.underscore.sub(%r{(?[_/]|)#{suffix_word}$}, '')
name = "#{name}#{Regexp.last_match(:char) || '_'}#{suffix_word}"
suffix ? name : name.sub(/_#{suffix_word}$/, '')
end
def base?
self == Gitlab::Experiment || name == Configuration.base_class
end
def constantize(name = nil)
return self if name.nil?
experiment_class = experiment_name(name).classify
experiment_class.safe_constantize || begin
return Configuration.base_class.constantize unless Configuration.strict_registration
raise UnregisteredExperiment, <<~ERR
No experiment registered for `#{name}`. Please register the experiment by defining a class:
class #{experiment_class} < #{Configuration.base_class}
control
candidate { 'candidate' }
end
ERR
end
end
def from_param(id)
%r{/?(?.*):(?.*)$} =~ id
name = CGI.unescape(name) if name
constantize(name).new(name).tap { |e| e.context.key(key) }
end
end
def initialize(name = nil, variant_name = nil, **context)
raise ArgumentError, 'name is required' if name.blank? && self.class.base?
@_name = self.class.experiment_name(name, suffix: false)
@_context = Context.new(self, **context)
@_assigned_variant_name = cache_variant(variant_name) { nil } if variant_name.present?
yield self if block_given?
end
def inspect
"#<#{self.class.name || 'AnonymousClass'}:#{format('0x%016X', __id__)} name=#{name} context=#{context.value}>"
end
def run(variant_name)
behaviors.freeze
context.freeze
block = behaviors[variant_name]
raise BehaviorMissingError, "the `#{variant_name}` variant hasn't been registered" if block.nil?
result = block.call
publish(result) if enabled?
result
end
def id
"#{name}:#{context.key}"
end
alias_method :to_param, :id
def process_redirect_url(url)
return unless Configuration.redirect_url_validator&.call(url)
track('visited', url: url)
url # return the url, which allows for mutation
end
def key_for(source, seed = name)
return source if source.is_a?(String)
source = source.keys + source.values if source.is_a?(Hash)
ingredients = Array(source).map { |v| identify(v) }
ingredients.unshift(seed).unshift(Configuration.context_key_secret)
Digest::SHA2.new(Configuration.context_key_bit_length).hexdigest(ingredients.join('|')) # rubocop:disable Fips/OpenSSL
end
# @deprecated
def variant_names
Configuration.deprecated(
:variant_names,
'instead use `behavior.names`, which includes :control',
version: '0.8.0'
)
behaviors.keys - [:control]
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/callbacks.rb 0000644 0000041 0000041 00000010601 14535544660 023637 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module Callbacks
extend ActiveSupport::Concern
include ActiveSupport::Callbacks
included do
# Callbacks are listed in order of when they're executed when running an experiment.
# Exclusion check chain:
#
# The :exclusion_check chain is executed when determining if the context should be excluded from the experiment.
#
# If any callback returns true, further chain execution is terminated, the context will be considered excluded,
# and the control behavior will be provided.
define_callbacks(:exclusion_check, skip_after_callbacks_if_terminated: true)
# Segmentation chain:
#
# The :segmentation chain is executed when no variant has been explicitly provided, the experiment is enabled,
# and the context hasn't been excluded.
#
# If the :segmentation callback chain doesn't need to be executed, the :segmentation_skipped chain will be
# executed as the fallback.
#
# If any callback explicitly sets a variant, further chain execution is terminated.
define_callbacks(:segmentation)
define_callbacks(:segmentation_skipped)
# Run chain:
#
# The :run chain is executed when the experiment is enabled, and the context hasn't been excluded.
#
# If the :run callback chain doesn't need to be executed, the :run_skipped chain will be executed as the
# fallback.
define_callbacks(:run)
define_callbacks(:run_skipped)
end
class_methods do
def registered_behavior_callbacks
@_registered_behavior_callbacks ||= {}
end
private
def build_behavior_callback(filters, variant, **options, &block)
if registered_behavior_callbacks[variant]
raise ExistingBehaviorError, "a behavior for the `#{variant}` variant has already been registered"
end
callback_behavior = "#{variant}_behavior".to_sym
# Register a the behavior so we can define the block later.
registered_behavior_callbacks[variant] = callback_behavior
# Add our block or default behavior method.
filters.push(block) if block.present?
filters.unshift(callback_behavior) if filters.empty?
# Define and build the callback that will set our result.
define_callbacks(callback_behavior)
build_callback(callback_behavior, *filters, **options) do |target, callback|
target.instance_variable_set(:@_behavior_callback_result, callback.call(target, nil))
end
end
def build_exclude_callback(filters, **options)
build_callback(:exclusion_check, *filters, **options) do |target, callback|
throw(:abort) if target.instance_variable_get(:@_excluded) || callback.call(target, nil) == true
end
end
def build_segment_callback(filters, variant, **options)
build_callback(:segmentation, *filters, **options) do |target, callback|
if target.instance_variable_get(:@_assigned_variant_name).nil? && callback.call(target, nil)
target.assigned(variant)
end
end
end
def build_run_callback(filters, **options)
set_callback(:run, *filters.compact, **options)
end
def build_callback(chain, *filters, **options)
filters = filters.compact.map do |filter|
result_lambda = ActiveSupport::Callbacks::CallTemplate.build(filter, self).make_lambda
->(target) { yield(target, result_lambda) }
end
raise ArgumentError, 'no filters provided' if filters.empty?
set_callback(chain, *filters, **options)
end
end
private
def exclusion_callback_chain
:exclusion_check
end
def segmentation_callback_chain
return :segmentation if @_assigned_variant_name.nil? && enabled? && !excluded?
:segmentation_skipped
end
def run_callback_chain
return :run if enabled? && !excluded?
:run_skipped
end
def registered_behavior_callbacks
self.class.registered_behavior_callbacks.transform_values do |callback_behavior|
-> { run_callbacks(callback_behavior) { @_behavior_callback_result } }
end
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/middleware.rb 0000644 0000041 0000041 00000001336 14535544660 024042 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
class Middleware
def self.redirect(id, url)
raise Error, 'no url to redirect to' if url.blank?
experiment = Gitlab::Experiment.from_param(id)
[303, { 'Location' => experiment.process_redirect_url(url) || raise(Error, 'not redirecting') }, []]
end
def initialize(app, base_path)
@app = app
@matcher = %r{^#{base_path}/(?.+)}
end
def call(env)
return @app.call(env) if env['REQUEST_METHOD'] != 'GET' || (match = @matcher.match(env['PATH_INFO'])).nil?
Middleware.redirect(match[:id], env['QUERY_STRING'])
rescue Error
@app.call(env)
end
end
end
end
gitlab-experiment-0.9.1/lib/gitlab/experiment/rspec.rb 0000644 0000041 0000041 00000027325 14535544660 023047 0 ustar www-data www-data # frozen_string_literal: true
module Gitlab
class Experiment
module TestBehaviors
autoload :Trackable, 'gitlab/experiment/test_behaviors/trackable.rb'
end
WrappedExperiment = Struct.new(:klass, :experiment_name, :variant_name, :expectation_chain, :blocks)
module RSpecMocks
@__gitlab_experiment_receivers = {}
def self.track_gitlab_experiment_receiver(method, receiver)
# Leverage the `>=` method on Gitlab::Experiment to determine if the receiver is an experiment, not the other
# way round -- `receiver.<=` could be mocked and we want to be extra careful.
(@__gitlab_experiment_receivers[method] ||= []) << receiver if Gitlab::Experiment >= receiver
rescue StandardError # again, let's just be extra careful
false
end
def self.bind_gitlab_experiment_receiver(method)
method.unbind.bind(@__gitlab_experiment_receivers[method].pop)
end
module MethodDouble
def proxy_method_invoked(receiver, *args, &block)
RSpecMocks.track_gitlab_experiment_receiver(original_method, receiver)
super
end
ruby2_keywords :proxy_method_invoked if respond_to?(:ruby2_keywords, true)
end
end
module RSpecHelpers
def stub_experiments(experiments)
experiments.each do |experiment|
wrapped_experiment(experiment, remock: true) do |instance, wrapped|
# Stub internal methods that will make it behave as we've instructed.
allow(instance).to receive(:enabled?) { wrapped.variant_name != false }
# Stub the variant resolution logic to handle true/false, and named variants.
allow(instance).to receive(:resolve_variant_name).and_wrap_original { |method|
# Call the original method if we specified simply `true`.
wrapped.variant_name == true ? method.call : wrapped.variant_name
}
end
end
wrapped_experiments
end
def wrapped_experiment(experiment, remock: false, &block)
klass, experiment_name, variant_name = *extract_experiment_details(experiment)
wrapped_experiment = wrapped_experiments[experiment_name] =
(!remock && wrapped_experiments[experiment_name]) ||
WrappedExperiment.new(klass, experiment_name, variant_name, wrapped_experiment_chain_for(klass), [])
wrapped_experiment.blocks << block if block
wrapped_experiment
end
private
def wrapped_experiments
@__wrapped_experiments ||= defined?(HashWithIndifferentAccess) ? HashWithIndifferentAccess.new : {}
end
def wrapped_experiment_chain_for(klass)
@__wrapped_experiment_chains ||= {}
@__wrapped_experiment_chains[klass.name || klass.object_id] ||= begin
allow(klass).to receive(:new).and_wrap_original do |method, *args, **kwargs, &original_block|
RSpecMocks.bind_gitlab_experiment_receiver(method).call(*args, **kwargs).tap do |instance|
wrapped = @__wrapped_experiments[instance.instance_variable_get(:@_name)]
wrapped&.blocks&.each { |b| b.call(instance, wrapped) }
original_block&.call(instance)
end
end
end
end
def extract_experiment_details(experiment)
experiment_name = nil
variant_name = nil
experiment_name = experiment if experiment.is_a?(Symbol)
experiment_name, variant_name = *experiment if experiment.is_a?(Array)
base_klass = Configuration.base_class.constantize
variant_name = experiment.assigned.name if experiment.is_a?(base_klass)
resolved_klass = experiment_klass(experiment) { base_klass.constantize(experiment_name) }
experiment_name ||= experiment.instance_variable_get(:@_name)
[resolved_klass, experiment_name.to_s, variant_name]
end
def experiment_klass(experiment, &block)
if experiment.class.name.nil? # anonymous class instance
experiment.class
elsif experiment.instance_of?(Class) # class level stubbing, eg. "MyExperiment"
experiment
elsif block
yield
end
end
end
module RSpecMatchers
extend RSpec::Matchers::DSL
def require_experiment(experiment, matcher, instances_only: true)
klass = experiment.instance_of?(Class) ? experiment : experiment.class
raise ArgumentError, "the #{matcher} matcher is limited to experiments" unless klass <= Gitlab::Experiment
if instances_only && experiment == klass
raise ArgumentError, "the #{matcher} matcher is limited to experiment instances"
end
experiment
end
matcher :register_behavior do |behavior_name|
match do |experiment|
@experiment = require_experiment(experiment, 'register_behavior')
block = @experiment.behaviors[behavior_name]
@return_expected = false unless block
if @return_expected
@actual_return = block.call
@expected_return == @actual_return
else
block
end
end
chain :with do |expected|
@return_expected = true
@expected_return = expected
end
failure_message do
add_details("expected the #{behavior_name} behavior to be registered")
end
failure_message_when_negated do
add_details("expected the #{behavior_name} behavior not to be registered")
end
def add_details(base)
details = []
if @return_expected
base = "#{base} with a return value"
details << " expected return: #{@expected_return.inspect}\n" \
" actual return: #{@actual_return.inspect}"
else
details << " behaviors: #{@experiment.behaviors.keys.inspect}"
end
details.unshift(base).join("\n")
end
end
matcher :exclude do |context|
match do |experiment|
@experiment = require_experiment(experiment, 'exclude')
@experiment.context(context)
@experiment.instance_variable_set(:@_excluded, nil)
!@experiment.run_callbacks(:exclusion_check) { :not_excluded }
end
failure_message do
"expected #{context} to be excluded"
end
failure_message_when_negated do
"expected #{context} not to be excluded"
end
end
matcher :segment do |context|
match do |experiment|
@experiment = require_experiment(experiment, 'segment')
@experiment.context(context)
@experiment.instance_variable_set(:@_assigned_variant_name, nil)
@experiment.run_callbacks(:segmentation)
@actual_variant = @experiment.instance_variable_get(:@_assigned_variant_name)
@expected_variant ? @actual_variant == @expected_variant : @actual_variant.present?
end
chain :into do |expected|
raise ArgumentError, 'variant name must be provided' if expected.blank?
@expected_variant = expected
end
failure_message do
add_details("expected #{context} to be segmented")
end
failure_message_when_negated do
add_details("expected #{context} not to be segmented")
end
def add_details(base)
details = []
if @expected_variant
base = "#{base} into variant"
details << " expected variant: #{@expected_variant.inspect}\n" \
" actual variant: #{@actual_variant.inspect}"
end
details.unshift(base).join("\n")
end
end
matcher :track do |event, *event_args|
match do |experiment|
@experiment = require_experiment(experiment, 'track', instances_only: false)
set_expectations(event, *event_args, negated: false)
end
match_when_negated do |experiment|
@experiment = require_experiment(experiment, 'track', instances_only: false)
set_expectations(event, *event_args, negated: true)
end
chain(:for) do |expected|
raise ArgumentError, 'variant name must be provided' if expected.blank?
@expected_variant = expected
end
chain(:with_context) do |expected|
raise ArgumentError, 'context name must be provided' if expected.nil?
@expected_context = expected
end
chain(:on_next_instance) { @on_next_instance = true }
def set_expectations(event, *event_args, negated:)
failure_message = failure_message_with_details(event, negated: negated)
expectations = proc do |e|
allow(e).to receive(:track).and_call_original
if negated
if @expected_variant || @expected_context
raise ArgumentError, 'cannot specify `for` or `with_context` when negating on tracking calls'
end
expect(e).not_to receive(:track).with(*[event, *event_args]), failure_message
else
expect(e.assigned.name).to(eq(@expected_variant), failure_message) if @expected_variant
expect(e.context.value).to(include(@expected_context), failure_message) if @expected_context
expect(e).to receive(:track).with(*[event, *event_args]).and_call_original, failure_message
end
end
return wrapped_experiment(@experiment, &expectations) if @on_next_instance || @experiment.instance_of?(Class)
expectations.call(@experiment)
end
def failure_message_with_details(event, negated: false)
add_details("expected #{@experiment.inspect} #{negated ? 'not to' : 'to'} have tracked #{event.inspect}")
end
def add_details(base)
details = []
if @expected_variant
base = "#{base} for variant"
details << " expected variant: #{@expected_variant.inspect}\n" \
" actual variant: #{@experiment.assigned.name.inspect})"
end
if @expected_context
base = "#{base} with context"
details << " expected context: #{@expected_context.inspect}\n" \
" actual context: #{@experiment.context.value.inspect})"
end
details.unshift(base).join("\n")
end
end
end
end
end
RSpec.configure do |config|
config.include Gitlab::Experiment::RSpecHelpers
config.include Gitlab::Experiment::Dsl
config.before(:each) do |example|
if example.metadata[:experiment] == true || example.metadata[:type] == :experiment
RequestStore.clear!
if defined?(Gitlab::Experiment::TestBehaviors::TrackedStructure)
Gitlab::Experiment::TestBehaviors::TrackedStructure.reset!
end
end
end
config.include Gitlab::Experiment::RSpecMatchers, :experiment
config.include Gitlab::Experiment::RSpecMatchers, type: :experiment
config.define_derived_metadata(file_path: Regexp.new('spec/experiments/')) do |metadata|
metadata[:type] ||= :experiment
end
# We need to monkeypatch rspec-mocks because there's an issue around stubbing class methods that impacts us here.
#
# You can find out what the outcome is of the issues I've opened on rspec-mocks, and maybe some day this won't be
# needed.
#
# https://github.com/rspec/rspec-mocks/issues/1452
# https://github.com/rspec/rspec-mocks/issues/1451 (closed)
#
# The other way I've considered patching this is inside gitlab-experiment itself, by adding an Anonymous class and
# instantiating that instead of the configured base_class, and then it's less common but still possible to run into
# the issue.
require 'rspec/mocks/method_double'
RSpec::Mocks::MethodDouble.prepend(Gitlab::Experiment::RSpecMocks::MethodDouble)
end
gitlab-experiment-0.9.1/lib/generators/ 0000755 0000041 0000041 00000000000 14535544660 020124 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/gitlab/ 0000755 0000041 0000041 00000000000 14535544660 021366 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/ 0000755 0000041 0000041 00000000000 14535544660 023546 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/experiment_generator.rb 0000644 0000041 0000041 00000001607 14535544660 030325 0 ustar www-data www-data # frozen_string_literal: true
require 'rails/generators'
module Gitlab
module Generators
class ExperimentGenerator < Rails::Generators::NamedBase
source_root File.expand_path('templates/', __dir__)
check_class_collision suffix: 'Experiment'
argument :variants, type: :array, default: %w[control candidate], banner: 'variant variant'
class_option :skip_comments, type: :boolean, default: false, desc: 'Omit helpful comments from generated files'
def create_experiment
template 'experiment.rb', File.join('app/experiments', class_path, "#{file_name}_experiment.rb")
end
hook_for :test_framework
private
def file_name
@_file_name ||= remove_possible_suffix(super)
end
def remove_possible_suffix(name)
name.sub(/_?exp[ei]riment$/i, "") # be somewhat forgiving with spelling
end
end
end
end
gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/templates/ 0000755 0000041 0000041 00000000000 14535544660 025544 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/templates/experiment.rb.tt 0000644 0000041 0000041 00000005661 14535544660 030707 0 ustar www-data www-data # frozen_string_literal: true
<% if namespaced? -%>
require_dependency "<%= namespaced_path %>/application_experiment"
<% end -%>
<% module_namespacing do -%>
class <%= class_name %>Experiment < ApplicationExperiment
# Describe your experiment:
#
# The variant behaviors defined here will be called whenever the experiment
# is run unless overrides are provided.
<% variants.each do |variant| -%>
<% if %w[control candidate].include?(variant) -%>
<%= variant %> { }
<% else -%>
variant(:<%= variant %>) { }
<% end -%>
<% end -%>
<% unless options[:skip_comments] -%>
# You can register a `control`, `candidate`, or by naming variants directly.
# All of these can be registered using blocks, or by specifying a method.
#
# Here's some ways you might want to register your control logic:
#
#control { 'class level control' } # yield this block
#control :custom_control # call a private method
#control # call the private `control_behavior` method
#
# You can register candidate logic in the same way:
#
#candidate { 'class level candidate' } # yield this block
#candidate :custom_candidate # call a private method
#candidate # call the private `candidate_behavior` method
#
# For named variants it's the same, but a variant name must be provided:
#
#variant(:example) { 'class level example variant' }
#variant(:example) :example_variant
#variant(:example) # call the private `example_behavior` method
#
# Advanced customization:
#
# Some additional tools are provided to exclude and segment contexts. To
# exclude a given context, you can provide rules. For example, we could
# exclude all old accounts and all users with a specific first name.
#
#exclude :old_account?, ->{ context.user.first_name == 'Richard' }
#
# Segmentation allows for logic to be used to determine which variant a
# context will be assigned. Let's say you want to put all old accounts into a
# specific variant, and all users with a specific first name in another:
#
#segment :old_account?, variant: :variant_two
#segment(variant: :variant_one) { context.actor.first_name == 'Richard' }
#
# Utilizing your experiment:
#
# Once you've defined your experiment, you can run it elsewhere. You'll want
# to specify a context (you can read more about context here), and overrides
# for any or all of the variants you've registered in your experiment above.
#
# Here's an example of running the experiment that's sticky to current_user,
# with an override for our class level candidate logic:
#
# experiment(:<%= file_name %>, user: current_user) do |e|
# e.candidate { 'override <%= class_name %>Experiment behavior' }
# end
#
# If you want to publish the experiment to the client without running any
# code paths on the server, you can simply call publish instead of passing an
# experimental block:
#
# experiment(:<%= file_name %>, project: project).publish
#
<% end -%>
end
<% end -%>
gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/install/ 0000755 0000041 0000041 00000000000 14535544660 025214 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/install/templates/ 0000755 0000041 0000041 00000000000 14535544660 027212 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/install/templates/initializer.rb.tt 0000644 0000041 0000041 00000013764 14535544660 032523 0 ustar www-data www-data # frozen_string_literal: true
Gitlab::Experiment.configure do |config|
# Prefix all experiment names with a given string value.
# Use `nil` for no prefix.
config.name_prefix = nil
# The logger can be used to log various details of the experiments.
config.logger = Logger.new($stdout)
# The base class that should be instantiated for basic experiments.
# It should be a string, so we can constantize it later.
config.base_class = 'ApplicationExperiment'
# Require experiments to be defined in a class, with variants registered.
# This will disallow any anonymous experiments that are run inline
# without previously defining a class.
config.strict_registration = false
# The caching layer is expected to match the Rails.cache interface.
# If no cache is provided some rollout strategies may behave differently.
# Use `nil` for no caching.
config.cache = nil
# The domain to use on cookies.
#
# When not set, it uses the current host. If you want to provide specific
# hosts, you use `:all`, or provide an array.
#
# Examples:
# nil, :all, or ['www.gitlab.com', '.gitlab.com']
config.cookie_domain = :all
# The default rollout strategy.
#
# The recommended default rollout strategy when not using caching would
# be `Gitlab::Experiment::Rollout::Percent` as that will consistently
# assign the same variant with or without caching.
#
# Gitlab::Experiment::Rollout::Base can be inherited to implement your
# own rollout strategies.
#
# Each experiment can specify its own rollout strategy:
#
# class ExampleExperiment < ApplicationExperiment
# default_rollout :random # :percent, :round_robin, or MyCustomRollout
# end
#
# Included rollout strategies:
# :percent (recommended), :round_robin, or :random
config.default_rollout = :percent
# Secret seed used in generating context keys.
#
# You'll typically want to use an environment variable or secret value
# for this.
#
# Consider not using one that's shared with other systems, like Rails'
# SECRET_KEY_BASE for instance. Generate a new secret and utilize that
# instead.
config.context_key_secret = nil
# Bit length used by SHA2 in generating context keys.
#
# Using a higher bit length would require more computation time.
#
# Valid bit lengths:
# 256, 384, or 512
config.context_key_bit_length = 256
# The default base path that the middleware (or rails engine) will be
# mounted. The middleware enables an instrumentation url, that's similar
# to links that can be instrumented in email campaigns.
#
# Use `nil` if you don't want to mount the middleware.
#
# Examples:
# '/-/experiment', '/redirect', nil
config.mount_at = '/experiment'
# When using the middleware, links can be instrumented and redirected
# elsewhere. This can be exploited to make a harmful url look innocuous
# or that it's a valid url on your domain. To avoid this, you can provide
# your own logic for what urls will be considered valid and redirected
# to.
#
# Expected to return a boolean value.
config.redirect_url_validator = lambda do |_redirect_url|
true
end
# Tracking behavior can be implemented to link an event to an experiment.
#
# This block is executed within the scope of the experiment and so can
# access experiment methods, like `name`, `context`, and `signature`.
config.tracking_behavior = lambda do |event, args|
# An example of using a generic logger to track events:
config.logger.info "Gitlab::Experiment[#{name}] #{event}: #{args.merge(signature: signature)}"
# Using something like snowplow to track events (in gitlab):
#
# Gitlab::Tracking.event(name, event, **args.merge(
# context: (args[:context] || []) << SnowplowTracker::SelfDescribingJson.new(
# 'iglu:com.gitlab/gitlab_experiment/jsonschema/0-2-0', signature
# )
# ))
end
# Logic designed to respond when a given experiment is nested within
# another experiment. This can be useful to identify overlaps and when a
# code path leads to an experiment being nested within another.
#
# Reporting complexity can arise when one experiment changes rollout, and
# a downstream experiment is impacted by that.
#
# The base_class or a custom experiment can provide a `nest_experiment`
# method that implements its own logic that may allow certain experiments
# to be nested within it.
#
# This block is executed within the scope of the experiment and so can
# access experiment methods, like `name`, `context`, and `signature`.
#
# The default exception will include the where the experiment calls were
# initiated on, so for instance:
#
# Gitlab::Experiment::NestingError: unable to nest level2 within level1:
# level1 initiated by file_name.rb:2
# level2 initiated by file_name.rb:3
config.nested_behavior = lambda do |nested_experiment|
raise Gitlab::Experiment::NestingError.new(experiment: self, nested_experiment: nested_experiment)
end
# Called at the end of every experiment run, with the result.
#
# You may want to track that you've assigned a variant to a given
# context, or push the experiment into the client or publish results
# elsewhere like into redis.
#
# This block is executed within the scope of the experiment and so can
# access experiment methods, like `name`, `context`, and `signature`.
config.publishing_behavior = lambda do |result|
# Track the event using our own configured tracking logic.
track(:assignment)
# Log using our logging system, so the result (which can be large) can
# be reviewed later if we want to.
#
# Lograge::Event.log(experiment: name, result: result, signature: signature)
# Experiments that have been run during the request lifecycle can be
# pushed to the client layer by injecting the published experiments
# into javascript in a layout or view using something like:
#
# = javascript_tag(nonce: content_security_policy_nonce) do
# window.experiments = #{raw Gitlab::Experiment.published_experiments.to_json};
end
end
././@LongLink 0000644 0000000 0000000 00000000150 00000000000 011577 L ustar root root gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/install/templates/application_experiment.rb.tt gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/install/templates/application_experiment.rb0000644 0000041 0000041 00000000124 14535544660 034277 0 ustar www-data www-data # frozen_string_literal: true
class ApplicationExperiment < Gitlab::Experiment
end
gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/install/templates/POST_INSTALL 0000644 0000041 0000041 00000000177 14535544660 031115 0 ustar www-data www-data Gitlab::Experiment has been installed. You may want to adjust the configuration
that's been provided in the Rails initializer.
gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/install/install_generator.rb 0000644 0000041 0000041 00000002100 14535544660 031246 0 ustar www-data www-data # frozen_string_literal: true
require 'rails/generators'
module Gitlab
module Generators
module Experiment
class InstallGenerator < Rails::Generators::Base
source_root File.expand_path('templates', __dir__)
desc 'Installs the Gitlab::Experiment initializer and optional ApplicationExperiment into your application.'
class_option :skip_initializer,
type: :boolean, default: false, desc: 'Skip the initializer with default configuration'
class_option :skip_baseclass, type: :boolean, default: false, desc: 'Skip the ApplicationExperiment base class'
def create_initializer
return if options[:skip_initializer]
template 'initializer.rb', 'config/initializers/gitlab_experiment.rb'
end
def create_baseclass
return if options[:skip_baseclass]
template 'application_experiment.rb', 'app/experiments/application_experiment.rb'
end
def display_post_install
readme 'POST_INSTALL' if behavior == :invoke
end
end
end
end
end
gitlab-experiment-0.9.1/lib/generators/gitlab/experiment/USAGE 0000644 0000041 0000041 00000001407 14535544660 024337 0 ustar www-data www-data Description:
Stubs out a new experiment and its variants. Pass the experiment name,
either CamelCased or under_scored, and a list of variants as arguments.
To create an experiment within a module, specify the experiment name as a
path like 'parent_module/experiment_name'.
This generates an experiment class in app/experiments and invokes feature
flag, and test framework generators.
Example:
`rails generate gitlab:experiment NullHypothesis control candidate alt_variant`
NullHypothesis experiment with default variants.
Experiment: app/experiments/null_hypothesis_experiment.rb
Feature Flag: config/feature_flags/experiment/null_hypothesis.yaml
Test: test/experiments/null_hypothesis_experiment_test.rb
gitlab-experiment-0.9.1/lib/generators/test_unit/ 0000755 0000041 0000041 00000000000 14535544660 022142 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/test_unit/experiment/ 0000755 0000041 0000041 00000000000 14535544660 024322 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/test_unit/experiment/experiment_generator.rb 0000644 0000041 0000041 00000000724 14535544660 031100 0 ustar www-data www-data # frozen_string_literal: true
require 'rails/generators/test_unit'
module TestUnit # :nodoc:
module Generators # :nodoc:
class ExperimentGenerator < TestUnit::Generators::Base # :nodoc:
source_root File.expand_path('templates/', __dir__)
check_class_collision suffix: 'Test'
def create_test_file
template 'experiment_test.rb', File.join('test/experiments', class_path, "#{file_name}_experiment_test.rb")
end
end
end
end
gitlab-experiment-0.9.1/lib/generators/test_unit/experiment/templates/ 0000755 0000041 0000041 00000000000 14535544660 026320 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/test_unit/experiment/templates/experiment_test.rb.tt 0000644 0000041 0000041 00000000324 14535544660 032511 0 ustar www-data www-data # frozen_string_literal: true
require 'test_helper'
<% module_namespacing do -%>
class <%= class_name %>ExperimentTest < ActiveSupport::TestCase
# test "the truth" do
# assert true
# end
end
<% end -%>
gitlab-experiment-0.9.1/lib/generators/rspec/ 0000755 0000041 0000041 00000000000 14535544660 021240 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/rspec/experiment/ 0000755 0000041 0000041 00000000000 14535544660 023420 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/rspec/experiment/experiment_generator.rb 0000644 0000041 0000041 00000000600 14535544660 030167 0 ustar www-data www-data # frozen_string_literal: true
require 'generators/rspec'
module Rspec
module Generators
class ExperimentGenerator < Rspec::Generators::Base
source_root File.expand_path('templates/', __dir__)
def create_experiment_spec
template 'experiment_spec.rb', File.join('spec/experiments', class_path, "#{file_name}_experiment_spec.rb")
end
end
end
end
gitlab-experiment-0.9.1/lib/generators/rspec/experiment/templates/ 0000755 0000041 0000041 00000000000 14535544660 025416 5 ustar www-data www-data gitlab-experiment-0.9.1/lib/generators/rspec/experiment/templates/experiment_spec.rb.tt 0000644 0000041 0000041 00000000312 14535544660 031557 0 ustar www-data www-data # frozen_string_literal: true
require 'rails_helper'
<% module_namespacing do -%>
RSpec.describe <%= class_name %>Experiment do
pending "add some examples to (or delete) #{__FILE__}"
end
<% end -%>
gitlab-experiment-0.9.1/LICENSE.txt 0000644 0000041 0000041 00000002374 14535544660 017036 0 ustar www-data www-data Copyright (c) 2020-2022 GitLab B.V.
With regard to the GitLab Software:
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
For all third party components incorporated into the GitLab Software, those
components are licensed under the original license provided by the owner of the
applicable component.