{"id":233,"date":"2015-08-07T08:52:28","date_gmt":"2015-08-07T15:52:28","guid":{"rendered":"https:\/\/setheliot.com\/blog\/?p=233"},"modified":"2024-02-26T09:40:19","modified_gmt":"2024-02-26T17:40:19","slug":"amazon-and-testing-in-productions-some-good-some-bad","status":"publish","type":"post","link":"https:\/\/setheliot.com\/blog\/2015\/08\/07\/amazon-and-testing-in-productions-some-good-some-bad\/","title":{"rendered":"Amazon and Testing in Productions: Some good, some bad"},"content":{"rendered":"

Amazon has a well deserved reputation of being data-driven in its decision making.\u00a0 TiP is a vital part of this, but may not have always been approached as a legitimate methodology instead of an ad hoc approach.\u00a0 An example of the latter can be seen by anyone on the production Amazon.com site who searches for {test ASIN}. where ASIN” is the Amazon Standard Identification Number assigned to all items for sale on the Amazon site.\u00a0 Such a search will turn up the following Amazon items for sale”:<\/p>\n

\"\"<\/a><\/p>\n

\"\"<\/a><\/p>\n

This is TiP done poorly as it diminishes the perceived quality of the website, and exposes customers to risk — a $99,999 charge (or even $200 one) for a bogus item would not be a customer satisfying experience.<\/p>\n

Another TiP slip” occurred prior to the launch of Amazon Unbox (now Amazon Instant Video<\/a>).\u00a0 Amazon attempted to use Exposure Control to limit access to the yet un-launched site, however and enterprising hacker” found the information anyway and made it public<\/a>.<\/p>\n

However Amazon’s TiP successes should outweigh these missteps.\u00a0\u00a0 Greg Linden talks about the A\/B experiment he ran<\/a> to show that making recommendations based on the contents of your shopping cart was a good thing (where good thing equals more sales for Amazon).\u00a0 A key take-away was that prior to the experiment an SVP thought this was a bad idea, but as Greg says:<\/p>\n

I heard the SVP was angry when he discovered I was pushing out a test. But, even for top executives, it was hard to block a test. Measurement is good. The only good argument against testing would be that the negative impact might be so severe that Amazon couldn’t afford it, a difficult claim to make. The test rolled out.<\/p>\n

The results were clear. Not only did it win, but the feature won by such a wide margin that not having it live was costing Amazon a noticeable chunk of change. With new urgency, shopping cart recommendations launched.<\/p><\/blockquote>\n

Another success involved the move of Amazon’s ordering pipeline (where purchase transactions are handled) to a new platform (along with the rest of the site).\u00a0 A simple” migration, the developers did not expect much trouble, however testers’ wisdom prevailed and a series of online experiments used TiP to uncover revenue impacting problems before the launch [Testing with Real Users<\/a>, slide 56].<\/p>\n","protected":false},"excerpt":{"rendered":"

Amazon has a well deserved reputation of being data-driven in its decision making.\u00a0 TiP is a vital part of this, but may not have always been approached as a legitimate methodology instead of an ad hoc approach.\u00a0 An example of the latter can be seen by anyone on the production Amazon.com site who searches for […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-233","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/posts\/233","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/comments?post=233"}],"version-history":[{"count":0,"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/posts\/233\/revisions"}],"wp:attachment":[{"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/media?parent=233"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/categories?post=233"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/setheliot.com\/blog\/wp-json\/wp\/v2\/tags?post=233"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}