Data-driven: A/B Testing / MVT testing Platform

[hhstore]

[hhstore]

灰度发布和 A/B test:

• https://chai2010.cn/advanced-go-programming-book/ch5-web/ch5-09-gated-launch.html

通过分批次部署实现灰度发布

通过业务规则进行灰度发布

数据驱动业务:

MVT testing(Multivariate testing):

• 多变量测试
• Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations. Websites and mobile apps are made of combinations of changeable elements.

A/B 测试平台:

• 是针对业务数据的 A/B 测试, 而不是开发/发布/测试.

• 基于 data-driven A/B testing 的方式 驱动业务开发.

• 关键词:

feature toggle framework

Netflix 实践:

• https://github.com/Netflix
• We're continuously improving the product through data-driven A/B testing that enables us to experiment with novel concepts and understand the value of every feature we ship.
• https://netflix.github.io/

AB测试驱动业务:

• TestinData.AI：如何用AB测试驱动产品增长？
• 为什么国内网站很少做 A/B 测试？
• 大众点评大规模并行AB测试框架Gemini
• [A/B]那些年，我们追过的AB Testing （五）Experiments @ Airbnb

[hhstore]

项目调研:

flagr:

• https://github.com/checkr/flagr
• go 语言开发.
• 特性基本完备.

Wasabi:

• https://github.com/intuit/wasabi
• java

Petri:

• https://github.com/wix/petri/
• http://wix.github.io/petri/
• Petri is Wix’s holistic experiment system, it covers the lifecycle of product experimenting including A/B tests and feature toggles.
• 企业级

mendel

• https://github.com/yahoo/mendel
• yahoo 项目, 已废弃, 不过可以参考一下.

proctor:

• https://github.com/indeedeng/proctor

abxtracted:

• https://github.com/Abxtracted/abxtracted
• java 开发的.
• https://github.com/Abxtracted/go-abxtracted
• go client

feature:

• https://github.com/etsy/feature
• PHP

其他:

• https://github.com/rwbutler/FeatureFlags

运维相关的 A/B 测试:

• https://github.com/ContainerSolutions/k8s-deployment-strategies

[hhstore]

ref:

• How A/B Testing Can Drive Business Decisions

• HOW TO TAKE A DATA-DRIVEN APPROACH TO A/B TESTING

• Which are the best open source A/B testing framework to use in your projects?

[hhstore]

intuit/wasabi:

• 项目文档: https://intuit.github.io/wasabi/v1/guide/index.html

• 架构:

• https://medium.com/blueprint-by-intuit/the-architecture-behind-wasabi-an-open-source-a-b-testing-platform-b52430d3fd80

与业务产品(生产环境)集成:

• https://intuit.github.io/wasabi/v1/guide/index.html#integrating-ab-testing-in-your-product

A/B 测试规则录入:

• https://intuit.github.io/wasabi/v1/guide/index.html#how-do-i-form-the-rules
• 支持的比较策略: https://intuit.github.io/wasabi/v1/guide/index.html#what-type-of-comparisons-can-i-have-in-my-rules

[hhstore]

附件文档:

Feature Flag Best Practices.pdf

[hhstore]

checkr/flagr:

• Flagr 是一款基于 Go 的开源服务。

• 它提供功能标记、AB测试、动态配置功能。

• 它提供 flag 管理和 flag 评估的 REST API。

• 项目文档: https://checkr.github.io/flagr/#/home

• https://engineering.checkr.com/introducing-flagr-a-robust-high-performance-service-for-feature-flagging-and-a-b-testing-f037c219b7d5

• 官方博客介绍, 附带链接, 可查看.

docker 部署配置:

• https://www.cnblogs.com/rongfengliang/p/10773993.html
• 部署方式: 支持多种 db

官网给的示例场景:

• https://checkr.github.io/flagr/#/flagr_overview?id=flagr-running-example
• 产品设计了一个按钮, 3种颜色, 不清楚用户喜好.
• 就分别在3个不同的州, 不同比例的用户样本, 做测试, 评估用户喜好.