🏃 Know when to walk away, know when to run

When should you bail on an experiment you're running on your website? Plus: the car brand that's banning the color gray [MutinyLogo-01](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6mw3lLzNV1-WJV7CgL9sW2fNnzT4nPLyCW6KVqS58hMDf1W2Jgcc97JT_w-N8HwXm_Vhg6wW5c5g772JY8N0VWQwNy8bV8BFN3tRt9c2Ft7NMmwxFlN1_7MVspr103X99JhW6TYmW98rnTv0V2ywqW5ppznqW8BXjJj4rypg_W4qBNHq2bWnqHW3gK-QF6mc2T9W5_YqRR8t9SVnW2gThqX6Tmgy1N23RkBWK_Pz5W7M-8347nbjnv3c__1) Hi there, I absolutely love this quote, and feel it's especially relevant to marketers: "I have not failed. I've just found 10,000 ways that won't work." – Thomas Edison A big part of marketing is getting something live, noticing how the market reacts, and then iterating to make it better based on what worked well. That's why following a regular cadence of launching new experiments is a critical step in improving everything about your marketing: from messaging to conversion rates. Every test, even those that result in a negative lift, will help to inform your next move, build compounding value, and refine your growth engine. But how do you know when an experiment should be iterated on, or when it should be abandoned completely? In this email I'll share the 3 primary signs to look out for when deciding whether you should [iterate on a website experiment](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7CgVjqW18TN4y6ycYFNV9Yyz-3g_NpRVwJwXc4r1sDFW4Sb1XM7C0WgyVDx4hP6zxJ-LW36sFmd42stSJW1Th4CC4TGRMQW3K4Dst8GxRc2W2P6N__4MmWrDW2vJfww4nCfyRW3DKP4r7Qd3sfW4ZyPH82vK4pWW2xw0sM65R679Vmpmjg7klv9PW2f9Mjt7K2hhYW6dltkt3-qDBlW5vhMVp4GPNFkW6Q0s8c6XLplmW30s9z41H4md2W7t3dJr1tpJ55W3_7M5G44DvC6W99G7xW14ZV2rW7QXvHX8XGnh1W22n_--84xQg432rN1), or bail completely and try again. In this email - A simple framework for when to give up on an experiment - A car maker is banning the color gray? [When To Iterate On A Website Experiment,](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7CgVcfN51kkYwwFRnlW2wdKsQ3WCnFzW3jtdGt17p1sbN6HJzJH4DM56W3H08rn54mr7gW5FZQjW8FZClNW32SMMB8D2_lgW2J2sXZ54VBk5VJW9fF8BW_sxW5XS5r496ZRf7W939SCj2TrkGYW5vxKQv2_PK9SW7szw0W8Qlt84W5y33qB5BDqszN1ZG5KFNRHNDW6ZNY2z2R7T9NW2n4PQ77-VVKvW4VLQ7T2VtXrpW8nq_rV6JvGKmW8qPCjt55ryx6W7Jmvyf3l_kBSW9g4qwP86st5FW77YKD18rxBNGW2ny0t94t69493qn71) [And When To Move On](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7CgClyW17xR7n959dBXW5QfCpz7ZJb0gW73NZX_8DcTdhW6X-YTB3S4vQmW46hmnr8HQSpxW85M34W8T5MZBN6lzLYk-hT3YW56H8Xl992frxN3jKcZv1ZXG9W5ddTDM6NH__KW7bHcqG30KPbTW3sMz4K25SZF3W1ZYLgP9bS1ZLW5tZ0NG7J1y_MW1pfDYT24f-rsW2X595J27sNMZW1mRnJ574mHsTW453xPS5S5L2yW6BSgR45ymlDmW7bBbNw30s_-VW39f4SD8-2PHTV5gBGF8gR7hSW2mFZZV4-MF8DW4hlRzW8lmgD13nKS1) Here's a handy dandy diagram to get us started with. As we go through the different cases, I'll highlight the part of the diagram worth paying attention to in purple. [Marketing-experiment-iteration copy](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7Cg-93W16q-bN4tTFcGW7tLy546fVTRBW76nL8T5bjx1fN3Jkw7Gd-XhYW18BqPg5mzH0SW7KRnpj1vMwGdW61q7Mc2BMb6FN4nK3gvH2krfW8yHGvf2tTb_QW5B7Pq82PCmb5W3SZTXg1p0HL4N4K6CkXjZYjtW6g3KzN8GmjMXN6Z0YqHk_TmPW2lCvf73kHplbW4mNZW13p3m6DW8YhB5j10v04FW2P3qrn3w3W-yW5wXg423xst3QW4lwPpf49YrscW205gdM5KjsmfW7sBwZY32hSpsMV4N9vb2mz_W2sK7jg2dtmBt2YZ1) Case 1: Negative lift + medium or high traffic volume [Marketing-experiment-iteration copy-1](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7CgG89W7Y5P1K213y7nW5QZFDx4f9PMqW5ybJYl7dpK5NN56Wj-1ttzvzW68HsZx4l374JW7Q__sW5BXPKXW1XR0SS8BTFPQW5LrY2l2LxvdmW7bqG0D7LjTpfW7Z99B47h0xFBW58fnys5KL5NlN7qhm6MWFfM3W1km4955KZ21jW8DlbP85Qt1NkW4kqLMK99L54bW8SFPgN2KnnY2W8Ht7ZQ2lq_tGW1m5Q6M6rDQskW1jPxW58SbMQDW2BXLyH6XcKsQW1g3svs89-58SW7zS0Q66G0m1FW1mWG043Y7b6_W1NcdxW7jCM2M3pm61) Webpages with a lot of traffic are the best testing grounds because you get directional results very quickly. That means you do not need to wait for your experiment to reach a statistically significant result if you're seeing a negative trend. If you see a consistent negative trend after a few days with enough visitor traffic, you should consider iterating on your experiment. Consider changing the copy to be more direct. If one day the results are positive, and the next day they're negative, there is no need to iterate just yet. Give your experience some time to normalize before making conclusions. Case 2: Flat result + high traffic volume 113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7CgVgxW2q18Kq5xDRVLW6M8hrp1dCXcDW8QXLqq1H9whKW2_X8Dd1ht98-VxzkkC3_B8LvW7RxS162fSHDNW2y5gpH1C69xlW7vqL2k73MB27W84z-Hq3q5nQHW5cB2bc2GlSqdW85LYKK7DnPdgW1BSQkY6n936NW47F9Bx18lw5LW1sKm175Zb-d2VzHN_x5tJMcQW5619Hj6T82h1W1FQpSN5FBXznW2QMswH7S0kwjW96kKbQ5_XXhKW5mgltn2PDczkN6-KQB4p3lywW5wTHn-3xl2ncW5pQZ1_1Ytk_mW5SGqSJ5Gbn4Y3fF31 Unlike [A/B testing](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6n53lLBmV1-WJV7CgCmMW1z2ZYb3_HZm8W2Z9vbM2bxPsMW6mvN5d1BNf-rW7_cs3X1DL0VQW8Tp0gf6tNJhpN4HghW-W2LprW36GNnG7kj7rDW8Ds5LX4Z2qBFMQHlRdZsM6jW4fGSVv7Hxx97VKf9mX7yRPHMW2yMNL73mFKBTN6pVSPw3qfCnN75KRYjj5QJgW4MXzsT8sjgPKW2YqJ4l1bsxDhW3WF77Y8kW_T3W961gBn6LPLwJW4vtYp52jmXyWVqQ_XL6y5rsLW58p0972xydxTW11G30q7dLbT73fHc1), personalization should generate a large lift (normally at least 20% compared to 4% for A/B testing). If your lift is smaller (between -20% to 20%) and the experiment has not reached statistical significance, you should iterate on the experience to create a larger impact. Take what you've learned from the experiment and run another experiment that's a bit bolder. Make sure your experiment has enough visitors to determine the result is flat. Look for at least 300 visitors in each experiment before making this conclusion. Case 3: Low visitor traffic 113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7CgPn2W8L-p0V3CjnR3W910vc86rxfyGW517-wH58pqJXW5QSSHx5tMfp6W7YfJ0W8K50rpN3tVb7QZwJJ2W31Jkr_1fv8lqW2L6Z4813vvzCV-fs841d96JZVtMZtt6RyxtlW4sXgD_10MkcGW6hqCGW7wH4XjN2bGmHzfH40RN16vtblM9BQyW7nsbd8871VklVMtJVm6ztwWvW5yY_gJ3cyW0RW5GxTm7742_lWVqlC0Q8Z571kW8MZvns97_2TlN33HMLjYZ7lbW7l_G3L1Jxf-nVJbfcp1lWrvyW3rshzs7Mm4Z83dNd1 If your experiments are being seen by fewer than 100 visitors, it will be difficult for you to measure performance differences. In these cases you should try to [expand your segment size](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6n53lLBmV1-WJV7CgGtCW3nRwCN9cJCRTW5VjsVW3QtDq3N3YQBKt1zWc5W71PcrJ3CkCb1W3MKn4Y11Wx-NW1GBFnm4x4Q_-W8mf90F33WxvJW58WycB61LKLyVGxq734bJJrSW8QmTJ_98lqLnW2mWyzH7RRRmxW5BsgSW6wqt9XW4LGyj656PLVGVBDvjt3MDznTW2vRHFV7mbS0ZW21t2gx2-7gKZMyFpdnzw6zkW5F8Cc_4lqpJ8W8KnH081hNzkTW4HnYhQ2W8mLCW1DtWX83P0gk-W5Dgkqx6j1_9V3d0y1) to reach a larger audience. The bigger the audience, the faster you'll see results. It’s in the cases when you’re getting good results and the experiment is getting a lot of traffic that you want to hold-out until it reaches stat sig. This will ensure you’re getting the most conversions from your test and on the right path to growing revenue through your website. We’ve put together [examples of how you can iterate your tests](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7CgRTgW3r3wTR6GXvDDW3jqc4W2knjmVW2yDg0F8lBn1CVwCCYn68btV8W6jZDXm1kJtT0W4vSfL81PYB-jW3HmLCn3WJVNFW9jTN2k36knR3W9k5LqJ1RKLJSW1_Z22-8J7Q6TW61D4Kv8x_g_yW6fmD6j2v_H_sW161s-28WXNY6W1z3Wq37VY9s7W8fdxB02HCjxzW8N6c-H8cs8phW2gqbB93bZhMgW2SvBCp4JW25xW3rlzym4wDfL4W8-PCN43tbKZ6N6-y29zCh7ktW7llTps83PsM5W86xs2t2K4NPXW7j9MH91svRh5355p1) in each of these cases so you can get a new experiment launched and collect data ASAP. [Examples of how to iterate your experiments](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6np3lLBGV1-WJV7Cg-HnW84F-gm4428ZfW6Dv4qg1QjsslV--8rk8yMQ12W1jfFpM6pnzHRW6bfS5f37NRRPW8p3W-v93jWxWW2tC1j232Nr2cW1nKXnw6DYJzVF2GzV2frCs6W1bGb9J3FYMQYW6gLBg15DFzX4W7fCjwH24wRycW8s2QKC8vNYJmW5VW2VM3WKfH1W8lrmwC1RgnnNVd0BWf5VbFXLW1GkkjV94L-cMW97_8KQ1KssxGW4hYBdn5L6Tf7W4gwWc284qGXXW5vHrt767thQdW5HxbCV39dgZ9W3r9-y26KmwhMW6-mPDp7ThP9l3pth1) [Stuff We're Loving](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6mQ3lLB3V1-WJV7CgP0qW8PqzlT8NBl8rW8kKDlV3jfHPqW2HyrQq4bT9hDW7cRRN_6jlppxW5yJJJF8934pbN3St5y-n7TR_W4hwTgT4R82HGN1DM13dbmqgBW4yQ8tL96lPbSW354xyb2xPYT3N2Bp8mnmqtkRW23xr3P99dxmsVb2Sd93TZJVTW1MNvRy1mLpGLW6DJQfH8_rZT_W1fhGBg7NXscdN4-h_8cgcPX_W99v7lW13k1YkW7C5Vj753pDztVVH67w6PFSJq3gNN1) 🚗 FIAT zigging where others are zagging I think this campaign from the automaker FIAT is brilliant. They're no longer making gray vehicles. Yup, you read that right – only bright pastels and eye-popping shades for future FIAT owners. Check out the most Italian launch video ever, featuring their CEO: [Screenshot 2023-07-06 at 4.29.20 PM](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6mQ3lLB3V1-WJV7CgJKtVRXDqL2SSWdzW45kWMq7fpczkW3ZTw8M4fLCz1W5g1fPs2Sfw2nW2_lBRm8pPZRKW3VH3CP85Y0gqVGFrQw76XHflW2D2bDN3Cnd_QW6q67Dq69zyP2W72zn4k2HxP6KW8gNtjD1txJjZVN5_wr9k_KRlW4Tw1Vt6kKP5CW33NWzs98Fyr5W4GfxZs4gs3mHN1jRfGkXCRn-W50X7YB6h7CL-W4wWCgf44_VxcN1KcfBVcsWXNW3TyqXW4QFFW131nb1) 🎆 Hope you enjoyed your weekend (more than these people)! I hope that your holidays are going better than these people. [Screenshot 2023-07-06 at 4.35.20 PM](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6n53lLBmV1-WJV7CgXx0W7KkpRV3MzhjNW53nkkk5WvDYdW7gXLR87zgkWlW2zrnyG7tJMMlW5rCpSG2NbH8_W2_b45l2Fcw8MW55-VBy2Tzqp9W3q2h2k8rJNNTMFbvZRfSycvN2KTr__S4FMNVwwxtX6-HXWqN87r7XS_6SFQVxkkmf8qZjthW3dVt591rRgqKW5CmZzK8_TKG2W7xfK7B7Bhh-hW8WtNg680bjJ9N3pkw3wdhwF_W21Zp4C1-FdQbW1q42ZW1_tpT5W3bBXRT8B8PrwW4H0Tdy76cx3_3kHH1) Until next time, happy growing. [Stewart Hillhouse](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6n53lLBmV1-WJV7Cg-vvW1N3spY1JybdTW5_BDTm7BL3sYW4wTjV176kjDpN1WtWB-Zszk8W78Gr4c1rvqxBW6mgz9S25FgBXW97Zt5w3Z__tlW7xzkQL2zLvSTVjjNGv7tN_kGW226yPG6Ts4k9W2nBdPv6yV3BZW1_NNGs20NQS0W5ZZFnP4Vh-J1W34WFgp3Rw2Q1VvSDR0858ldxW4_rKH22MQVJHW1mLxTc5D4pCHW7HPp7r6SRxdqVynn_y9hf4gkN69cGxvlz42MVBhnNl60Bd75W911T1D1cm7r639Lv1) Head of Content at Mutiny What does Mutiny do again? Mutiny is a conversion platform that allows marketers to convert more of their top-of-funnel traffic into customers. Mutiny is trusted by Notion, Brex, Snowflake, 6sense, and hundreds more of the fastest growing companies in the world to power their conversion. Want a walkthrough of how Mutiny works? [Take an interactive demo](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6n_3lLCfV1-WJV7CgRl5VpxMWF99C4DSW19PnKY3GFV-8W1p7J6s4SXscHN35MF-xmw49yW40shBQ8BfdwJW7H_fV-2BwpN2W6wh9HL6Pn8fxW7KwdHC4tlSb7W7S8WWW4YH71sW4NnV-p52FXr1W7YdQvd7Gq-SxW6F8QyD6SjdRvW7JhBzS4pLHy4W6XdMT12rWMVlW8zVw-B65TXzvW97WjJ66LsXQxVHRFvJ3pJYWlW2tFbD36Kd3MNW754QQB5JJRy0W39BNYN6z0WW4F4lF9w6RXxdN3q3hmf50p9rW2PTGTv1B-clHW2JRhCm1rmsdFMBfSsfxJxL1W2kkTLf3nMbbfW5SkB0D76WwyZW1lq7dh4Kpvr23dRD1). If you made it this far, you'll fit right in Join [875+ other revenue-focused marketers](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6mw3lLzNV1-WJV7CgBngW12pVnw86KrWfW1NdNvL2r-4SyW6nRhdp7fpy7fW5lYlVc6cxb7dW8lbHHC1wSTpqW1C2Cqw6yTz19W3D5-yr5S9VFhW7Sn0nP1KZYLlW3dKj4k4sq9DpW2lTDK-88StHWV8dbsf2GKtlGW76lm5p1K9g_MW42bDNh15jwctVPZGlp6J_n1yW2KydCS4z7QbVW5-N2hh4ZLKc6W8mbKkH5l7V72Vvv5R03WHLxd3nMT1) who use M2 to up-level their career and solve their hardest challenges. Members get access to private roundtables, exclusive in-person events, and a community Slack channel. [Get access to the good stuff – apply to M2 today](113/cMsZQ04/MWb-HmL45cPW6nzMWt77vP-zW1RGk9h50J7PCN63y6mw3lLzNV1-WJV7CgZp2W8rv2Md1jllzgN6DTZ6xCf4zlW757GWY32T-L6VxG5Zs386t2cW24SGtr6-gjPJW5rV_782MPC7BN4m5DZP_ylm2VcQX7K3n_lKcW333sMy2-FLK1W7fV_nx8sFQW_W8D_Z354w-TwPW8wvdnv36wGZpW4nb7r22Nkm3DW3QXqgR1-MjGFW5Ydk5g16CX6YW5f1Qsc9dJhmPW3vhvkr4s3YS2VYp-Br4jgRX-3gzY1). Mutiny HQ, Mission District, San Francisco, California 94110 [Unsubscribe](