Show Notes
In this episode of Startups For The Rest Of Us, Rob and Mike talk about the pitfalls of commonly recommended marketing tactics. This topic was inspired by a tweet from Scott Watermasysk from KickoffLabs. Some of the tactics discussed include split testing, affiliate programs, content marketing, and more.
Items mentioned in this episode:
Welcome to Startups For The Rest Of Us. The podcast that helps developers, designers, and entrepreneurs be awesome at building, launching, and growing software products, whether you build your first product or you’re just thinking about it. I’m Mike.
Rob: And I’m Rob.
Mike: And we’re here to share our experiences to help you with the same mistakes we’ve made. What’s going on, Mr. Rob?
Rob: We received a voicemail from someone, I believe he was in Brisbane, Australia, but it was definitely an Australian accent and I could barely make out the audio. The audio quality was so bad that we don’t even know his name or the question he asked. If that is you, please call us back or better yet, just record an MP3 on your local machine and just attach it or send us a Dropbox link to questions@startupsfortherestofus.com. Because it sounded like a good question but couldn’t really make out what he’s saying during that. How about you? What are you up to?
Mike: I’m in the middle of testing a new road map process with one of my customers using Teamwork. The basic idea is that I set up a project in Teamwork, mainly because this customer’s been with me for a while and it’s not a really complicated stuff that they’re asking for what they need, but they need assurances or like timeline and implementation and everything else.
I’ve got a road map that I’ve been using in the background using Trello for a while, but I just haven’t kept up with it. I’ve never really gotten into Trello like most people have. What I did was I created a project in Teamwork and then I invited them to look at it. I took their […], threw them in there. Basically, I’m working through them and it gives me the opportunity to comment on those things directly and allow them to comment back on them as well. It creates this really nice feedback loop between those things.
I haven’t added any other customers into it yet, but I think that that might be something that I’ll look into the future. But I’ve looked at other road mapping tools in the past and I’ve never really quite found one that suited me.
Rob: Is this for communicating your road map to customers?
Mike: If a customer asks for something, how do I communicate it to the rest of the customers, like, “This is what is going to be going on and this is what the priority is.” It’s a little bit different than like a bug tracker, obviously, if there’s a bug or somebody needs support or something like that, it’s different, but I’m the only one who can add stuff in so that it’s visible, but anybody can comment on it, at least so it seems like.
Rob: I’m trying to think there’s a tool that we use at Drip and of course, I can’t remember the name of it and we always used it to announce what we had built. It’s like, “This is what we released in September.” It’s like a change log tool but it was user-friendly. It was not super techie. I’m trying to think if you could use that pretty easily to suggest like, “Here’s what we’re working on in order.” I think there was actually a little feature where you could just add a bulleted list, but it was super stripped down. It was a very simple tool. It was $10 a month or something. It’s probably someone’s side project, although it’s pretty well-designed. Is that all you need is just a list of things? Why not just have, maybe a public Google doc or even just a static page on your website that’s like /roadmap? It’s just five things in bold list like, “Here are the next major five efforts we’re doing.”
Mike: There’s two things. One is we’ve been going back and forth through email and I said, “Look, I understand you got all these different things that you want to see, but I need a list of them. Just going back and forth one at a time is really not working. Send me the whole list.” They sent me the entire list and broke it out into three different sets like, “These are things we need very, very soon, these are things that can wait a little while, and these are things we’d like to eventually see down the road.” I looked at them and they all mapped pretty well like my internal road map anyway, which I have in my bug tracker, but I’m not going to pay for other people outside of business to have an account there just because it’s going to get expensive.
I was trying to figure out how do I work back and forth with them because I had a bunch of questions on some of them and there were comments on other things. It was a question of, “How do I work with them to flesh out what each of these things is?” because if I got an email that’s, let’s say, two pages or something like that, like bullets that they email to me, it’s hard to go back and forth about one line item and five line items in an email. I really needed something that was going to break them out so that I can comment on things individually and separate them a little bit more. Does that makes sense?
Rob: It does, yeah. It’s not just for outbound communication, it’s kind of like you’re almost collaborating with a client. It’s not that they’re paying you for this and not that you’re going to build it exactly as you’re expecting, but they really are. It’s much more customer development process than it is, “Hey, we’ve got suggestions in here is what’s we’re going to build.”
Mike: Yeah, I’d say customer development is probably a better way to phrase it than I did. It is a lot of back-and-forth to figure out like, “Is this going to work for you or what exactly did you mean by that? How are you going to use this?” for example. Because some of the questions are like, “I hear what you’re saying and it makes sense but where you going to use this for? I want to know, is there a better way to do this or is there something else that we could do either short-term or manually that would get them there faster?”
Rob: Right. Cool, it sounds interesting. I think of it as less of a road map process and more of, I guess, it is impacting your road map, but it’s more like a customer collaboration or a customer development process that you’re doing using Teamwork. That makes sense?
Mike: Yeah, it is.
Rob: Cool.
Mike: But I went on Google and searched for road map software a while back and I came across all these other things, like there’s ProductBoard, aha.io, ProductPlan, ProdPad, and things like that. It’s just none of them really seem to fit what I was looking for in terms of this back-and-forth communication with either individual customers or with small groups of them. I really think I just want to identify a couple of more customers to maybe work on this with, but it’s really a matter of, does it make sense in that particular context for that customer?
Rob: That makes sense. Have you seen MailChimp’s new branding?
Mike: I got an email about it, but I haven’t looked at it yet.
Rob: It’s a trip. I think I saw a news article. I think I was on SparkToro Trending. SparkToro is Rand Fishkin’s new startup and there’s a trending page and they talked–there was an article written about MailChimp’s new branding. Obviously, I think of MailChimp as the ESP. There are other email service providers but they are the biggest one. They are the ones that send a billion emails a day. Like the title tag on their homepage, which is what will appear on Google now, is marketing platform for small business.
I don’t know if it’s a pivot as much of just a rebrand. But like their homepage used to say, “We help you sell more stuff at MailChimp,” or something like that, it’s very much commerce-focused, but their headline is now, “Your business was born for this. Become the brand you want to be with smarter marketing built for big things.” It’s very, very interesting. It’s a gutsy move. The colors are different. I actually like the design of the site. It’s definitely jarring.
Remember when the Drip rebrand happened and there was the magenta and the other colors, I feel this is similar but different, but the headline and the value prop on it feels not nearly as focused as what I’ve always thought of as MailChimp. The fact that the sub-headline is “Become the brand you want to be with smarter marketing built for big things,” that doesn’t resonate with me as a customer and never would have at any point in my entrepreneurial career. I’m concerned. These guys are smart, let’s not fault MailChimp, but I feel like maybe they’ve made a misstep here.
Mike: I don’t know. I think it’s easy to think that if a company changes their brand name, that, “Oh, this company isn’t for me any more. This tool isn’t for me,” but at the same time, the tool itself hasn’t change. It’s really just a marketing message. I forget who said this, but the line is something along the lines of, “What got you here won’t get your there.” Like you said, they’re sending a billion emails a day but that doesn’t necessarily acquire them more customers faster, they need a marketing message that resonates with a larger group of people than they currently have or they’re currently after because they probably saturated the market. The ESP market is really highly saturated. There’s tons of players there so they need to differentiate themselves an go after people who are either just not using something or are using something else and want to find something that is going to fit them better.
Rob: Yeah. That makes sense and that’s thing. When you’re not inside an privy to the conversations that they’re having and who they’re competing against when they’re going for big deals, you don’t know. That’s where I’m like, “Who am I to say?” but this is my opinion on the outside. Again, with the Drip stuff when there was that rebrand, it was right before I was leaving the company. It was happening and rolling out. There were so many naysayers and haters on social media and all the stuff, but they were privy to the information that is actually going on inside the company.
I guess what I’m saying is, it could be so easy for me to sit here and just rail on this MailChimp rebrand because it’s “different,” but I trust that MailChimp is pretty damn smart and pretty good at what they do, and that they took the information. It’s either going to work or they’ll change. If that headline doesn’t resonate with people, they’ll change it. I think that’s the thing. When you make these big gutsy moves and you’re trying to land and expand or pivot into a larger space or whatever it is that they’re doing, apply to more people, as you said, you have to do risky stuff at some point.
What’s interesting is in the early days of a startup, you take a lot of risks but no one’s there to notice and no one cares. But when you have 1000, 10,000, 50,000 customers and you take these risks, a lot of people get up in arms over it, but really it’s the same risk that you’ve been taking all along.
Mike: I think for large companies like that, it doesn’t matter nearly as much as we might sit here in armchair quarterback it, you know Monday morning quarterback. Just because their existing customers are still going to be paying them money, it’s about them acquiring new customers, not appealing to the ones that they already have. That’s the way I see it.
Rob: That’s the thing. As long as you don’t do such a pivot that you completely turn your existing customers off and have a bunch of churn and stuff, man, that would be really hard to do, especially with ESPs where there is some lock-in in the space.
Mike: Yeah, it is hard to move from one to another. I’ve done that before. It sucks.
Rob: Yup. So what are we talking about today?
Mike: Today, we’re going to be talking about a couple of pitfalls of some commonly recommended marketing tactics. This topic was inspired by a tweet from Scott over at KickoffLabs and we’ll link that into the show notes. But he said, “One of the worst startup diseases is AB testing. Such a colossal waste of time when you don’t have significant traffic,” and it got me to thinking, what other things are there that are commonly recommended marketing tactics that I see and hear about or you’ve see and hear about, that are either colossal waste of time or there’s pitfalls or things that you don’t necessarily know about until you get into it, and only to realize that you’re walking into a hornet’s nest and there’s more trouble than it’s worth or it’s just not going to work for you or it’s something that you can do short term, but it’s not going to be sustainable for you.
Rob: Cool. Let’s dig in. I was pretty excited when you talked about doing this topic today because I think there are a lot of, I won’t say misinformation but it’s more like myths or people making things look easier than it actually is in order to sell things, in order to sell their software or sell their infoproducts.
Mike: I think that’s the big piece of it is that they think it look easier or sound easier than it actually is. For split testing, we’ll start off there because of the tweet from Scott but you need substantial traffic and most early entrepreneurs don’t have it. It’s not just that you need substantial traffic, you need substantial traffic and a short enough time window where your test is going to be able to give you at least some sort of significant results.
The other thing that I think is a little bit misleading is about whether or not the split testing that you actually could do, if you were to get 1% results or an increase every month for 12 months, would you actually be that much better off? I have seen some odd anecdotes where people will say, “Hey, we did all these split test or we split tested something against itself and one was significantly different than the other.” It does make me wonder a little bit as to whether or not with the resources that we as small business owners have available, can we actually even effectively execute on that stuff in a way that is going to move the needle for our business?
Rob: If you’re thinking about the early days of your startup and you’re trying to build something people want, you’re in customer development, you have a website, a little bit of traffic, split testing is the furthest thing from your mind or should be the furthest thing from your mind. You need to be talking to customers in those early days, to figure out what is that I can build that people are willing to pay for, that’s different enough from the competition that I can communicate that and people will sign up.
Typically, you’re going to split test your home page or your pricing page or your sign-up funnel, and I would not even consider it before I had, let’s say 5000 or 10,000 uniques a month. Really, at that point, it depends on what revenue is and if I have the time to test the headline here or there. But if I were to do split testing at that point, let’s say I had 10,000 uniques a month coming to their home page, I would probably just set up one or two alternate headlines and just let it run and see what happens. But it wouldn’t be some massive effort where I would be redesigning the entire page to spending a bunch of time to try to get some major difference in conversions, because there are other levers you should be pulling at that point. When you have 5000 or 10,000 uniques, you should be worrying about more traffic, or you should be worrying about, frankly, converting more of the existing trials you have to pay the customer. It’s going to tend to move the needle more than spending a lot of time on split testing.
Mike: The next marketing tactic is using affiliate programs. I see a lot of startups from Product Hunt, BetaList, and a couple of other places that startup it up where they’re pitching this affiliate program and saying, “Hey you can manage your affiliate program through our SaaS and you’ll be able to get more customers for your business.” But the reality is that the attribution itself is fine within those pieces of software, but finding the affiliates who are able to bring in enough leads to make it worth your time is actually pretty hard.
I’ve done this a couple of times. It seems to me the software itself is reasonably straightforward in most cases to implement, but finding the people who are actually going to bring leads that are qualified to you is a lot more challenging. It’s just, you have to do a lot of education to those people, you have to make it worth their time, and you have to be finding affiliates who have a substantial traffic source already or existing list where you can point them back to your website, give them an affiliate code or something like that, and give them an offer that’s actually going to get them to convert. If you don’t have all of those things, then it’s just not going to work. It doesn’t matter how many people you sign up for your affiliate program. You need the qualified leads to be coming in.
Rob: Yeah. Finding affiliates is way harder than most people think unless this is a primary strategy for you and you have a network. The people who I’ve seen make it work—let’s talk about Clay Collins of LeadPages—it was built mostly on the affiliate model. He did that because he knew all the people in the internet marketing space. He had the clout to get them to do webinars with him. It wasn’t just, “Hey, send me some traffic.” It was, “Let’s do a webinar,” and then he pitch an annual deal and it was 50% off, there was time pressure and there were bonuses, I should say, that you got if you sold there. That’s how he grew it that fast.
ConvertKit did the same thing. It was the same playbook. If you’re going to do that, then do that and go all in on it. I recall, at one point, Clay was doing 20 webinars a month or something in the early days, 15 or 20. It was crazy. He’s just a machine. In that case, you’re an exception. You’re not going to start from a cold network where you don’t know other people who have big audiences and make affiliate programs work from the start. It’s going to be a ton of work. In addition, the process side I just talked about, I’ve only seen it work in one niche and it’s in this aspirational business niche. It’s in the people who want to be bloggers or be info marketers and they’re being sold info on how to start your own biz from home.
You look at LeadPages and ConvertKit and they both serve that same path Flynn-ish niche. You can’t just go to enterprise software and think you’re going to do this big affiliate model, sell to Fortune 500 companies or Fortune 1000 companies using that. Or even freelancers would be possible because there are people with those audiences, but it wouldn’t work to the same extent that it does because the audience is just aren’t as big and they aren’t as prone to buy. I’d say there is one or two other spaces that I know of that are similar to that, that aspirational thing where you can sell them the idea or the promise of, “Hey, you’re going to have a landing page provider or here’s an ESP and here’s how to start your blog and make money.” But beyond those, it is really hard to get an affiliate program profitable one off the ground.
I had affiliate program with HitTail and Drip, and they made money, they were profitable, but they were not major driving factors. Even when we did joint ventures and we did joint venture webinars, we’d do JV mailings, it made money and it was fine, but these were not the major drivers of growth that I’ve seen in most of my startups. Again, not saying it can’t work but it’s definitely different than it appears.
Mike: Yeah, it’s definitely a lot harder than it appears. You have mentioned enterprise software sales where affiliate models wouldn’t really work and I agree with that, but there’s a slightly different model for those enterprise deals where it’s basically a reseller arrangement. You don’t have direct contact with the enterprise companies, you resell to other companies, you sign them on as resellers which is a slightly different take on an affiliate program, but it’s not substantially that much different. It’s the same basic idea–somebody else is bring in the lead in. The difference with a reseller is that the reseller is basically managing the customer relationship versus with the affiliate, they’re bringing them in and then they’re hands-off at that point.
The next one is content marketing. I think that the content marketing has been all the rage for the past several years and I don’t see it necessarily ending anytime soon, but I think the bottom line for content marketing is that it is time and resource intensive to general content on a repeated basis. If you’re trying to blog once a week or put out a few articles each week or each month, that’s fine, but it’s not just the generation of those which is time and resource intensive, it’s also the marketing and distribution of those. If you’re trying to post it to Quora and various startup list and out your email newsletter and social media, that gets to be time and resource intensive, especially if you’re trying to cross-promote between different channels and schedule everything–it just gets complicated.
There’s also very long lead time to getting results and getting measurable results from them. It could be anywhere from three to six months, it could be as much as a year or 18 months. Regardless if that, it’s a lot of time and effort to get that engine running. But once it’s running, you’ll do really well with it but it just takes a long to get there and there’s probably better places for you to spend your time if you’re early on.
Rob: Yeah, this one is tough because content marketing can and does work. You just got to remember content marketing is more about SEO and it is a long-term play. That’s why when we get the questions from someone like, “Hey, I’m at $2000 MRR. Should I start my content marketing?” It depends. Probably not, but it really depends what niche are you in? Are there distribution networks? Not even networks but like growthhackers.com and YCombinator are distribution avenues for you to get the content out and are there people daily reading stuff like this? Are you going to be able to drive it? Are you going to build your list? And then long term, are you targeting SEO terms? Organic terms that are going to bring traffic? That’s how the play has really been successful for these larger companies.
Of the three we’ve talked about so far, I think content marketing is the best and most viable, but your critiques are absolutely correct in that it often takes a long time to get results and it is very resource intensive to generate content at the quality, and these days at the length that’s required to make it dent because there’s so much noise, man. I have not been on the social news sites like inbound.org used to be one but it shut down. growthhackers.com is still there, Hacker News, even SparkToro Trending, I had not been on any of those. I just don’t go on them regularly.
I’ve looked at a few of them this morning and the volume of content is crazy. It’s so much more and so much of it is highly targeted and you can tell it’s targeted to try to just get clicks to it. It’s a startup that is trying to get people to come through from Hacker News or from Product Time or from whatever to generate traffic to then funnel into the leads. We’ve all been there. I totally know that playbook. I was doing it back in 2012-2013 or even before that from my blog in 2010, but there is just a lot of noise out there. To rise above that, it’s a lot of time and money to create content that is good enough to warrant people’s attention.
Mike: The fourth item on our list is social media marketing and by this, I don’t mean paid ads like if you’re going on Twitter or Facebook or Instagram, whatever you like. You can pay for advertising, but that’s not what I’m talking about. I’m talking about building an audience and then trying to market content to them. I think the reason that this can be a substantial pitfall for people is that it seems like it should be easy but it’s time-intensive to gain followers. When you post things, a lot of times only a fraction of your followers are going to see a particular piece of content.
For example, I think on Twitter, the stuff that I’ve heard is about 5% of your followers are going to see any given tweet that you put out. So even if you have 100,000 followers, only about 5000 of them are going to see that. So the strategy to overcome that is you tweet multiple times a day and use something like Buffer to put those tweets out at different times a day to try and catch people on different time zones.
At that point, I feel like you’re also oversaturated in the people who are on Twitter a lot and it could very well be a turn-off to those people, but again, it’s a matter of, “What type of product are you promoting to people and is it going to be relevant to the people who are on Twitter?” I tried Twitter advertising for AuditShark, for example, and it just absolutely did not work because the people who are in the enterprise are just not the type of people who are looking for security software on Twitter.
Rob: Yeah and that’s not to say that Twitter advertising won’t work for anyone. I also think LinkedIn could be better for you, but I tried LinkedIn many times with multiple products and never got LinkedIn advertising to pay itself back. Again, not saying it won’t work, it’s just going to take trial and error if you can get it to work at all.
Mike: On that note, James Kennedy is going to be speaking at MicroConf Europe on a LinkedIn strategy for acquiring leads, so that might be interesting to you.
Rob: Awesome. That would be a cool one. I agree, man, social media marketing is great for B2C and it can be great for prosumer stuff, aspirational entrepreneurship, or even photographers–they tend to be aspirational. But when you’re talking about real sales tools, or real email marketing tools, or real tools that you want people to buy and use for years, and you need businesses that are making decisions, comparing you to competitors and all that, social media is a nice to have. It is not something in general that’s going to drive your bottom line if you are a B2B SaaS app. I’m sure there’s one exception to this, maybe two, I’m sure there are a couple, but overall as a bootstrapper or as someone who is really just trying to block and tackle, there are so many more things that you could be doing than to tooling around on Facebook and Twitter.
Mike: I think the real challenge here is just the fact that entrepreneurs tend to be on the internet a lot, we see Twitter a lot and we see Facebook a lot, so it’s natural to assume, “Hey, I should test that out or use that as a marketing strategy.” But again, it’s time-intensive to gain the followers. The reality is that what you want and most of those cases is actually just send them over to an email list anyway so you […] email address. Assuming you can get an engine up and running that can do that for you then that’s fine, but you still need a way to make it work and it can be very time-intensive to gain those followers and it’s not obvious always how you’re going to be able to do that.
The last one we have on our list today is offline advertising. By this, you can take it a bunch of different directions. It could be billboard advertising, […] response in a conference or podcast or anything where the direct attribution is a little bit more challenging. That also include things like sending postcards or physical mailers to people. I think, is it GRC marketing that does that? They advocate that a lot for sending out the bulky mail to people just to get their attention. That works great for those situations where you have a higher price point product or it’s a service, and there’s going to be a relationship that you’re trying to establish with them or the dollar amount is high enough that it’s worth it to send those.
The biggest downside of those is that it’s extremely hard to do the attribution in most cases and then there’s also a much longer iteration cycle. Instead of looking at couple of weeks for paid ads on Facebook or Twitter, you’re looking at a month, two months, maybe three months for an iteration cycle to send out a mailer and then figure out whether or not you got results from it, track those back, then make some adjustments or tweaks, and then move on to the next group. It can get very complicated to juggle all of those things at once because even if you’re just doing it repeatedly over the course of a month or two, it’s just going to suck up all of your time and attention. You’re not really going to be able to do very much else.
Rob: Yeah, it is expensive, too. It’s a long turn-around time and the iteration cycles are just, I would say, too long for a startup. Now, once you’re down the line, you have product market fit and it you’re in a space where offline is a really good option, obviously you could experiment with it. But it’s not something I would be messing with in the early days. And as you said, attribution is rough. I did some trade publication magazine advertising for one of my products once or twice and it didn’t work. Again, it’s not saying it wouldn’t work for you, but I quickly realized how expensive it was and just how you can’t tell if it’s working.
There’s a little adage, “50% of advertising doesn’t work.” You just can’t tell which 50% and that’s when people are talking about magazine and TV and radio and that kind of stuff. Obviously, you can tell which 50% works when you’re online and we are spoiled by that, in all honesty. I think that’s a real boon to the online marketer today. Offline is not something that I think you should get into lightly unless you really know what you’re doing.
Mike: I do know people who make offline advertising work. The problem is just that the iteration cycles are anywhere from 8-12 weeks just to find out whether or not a particular mailer got through to the right people and whether they got those people into their sales funnel. It does work especially in his particular business, but again, it’s just the lead time for you to go from one iteration cycle to the next and get the information back. It’s just hard if you’re still trying to make ends meet.
I think generally speaking, when you are trying to evaluate a marketing tactic is to whether or not it’s going to work for you or decide whether it’s going to be something that you want to try, there’s a couple of things to keep in mind. The first one is, is there a complicated setup that needs to be done first? Are you going to need to go create a bunch of accounts or are you going to need to integrate a bunch of different tools together? Is there ongoing effort that needs to be done? Are you going to have to constantly be creating or doing things? Any of those things where it takes your involvement on a very repeated basis is going to eat into the feasibility of using that strategy in the long term.
For blogging, for example, or content marketing, if you have to do that every single week, it can be difficult. It’s not saying you can’t outsource it. If you have money, you can obviously substitute that in for your time, but again, that’s a resource trade-off that you’re going to need to make down the road. You can start off doing it yourself or you can outsource it to somebody else. But those are the type of things you need to think about when you’re trying to figure out, “Is this something that I’m going to try and do long-term?” or are there better places that you could be spending your time?
Rob: Right because it’s one thing to intentionally try something and know that it’s not going to scale, but to do it as an experiment. It’s another thing to try a bunch of different tactics that really you don’t have much hope of sustaining without really doubling down on them. If you shotgun it and you try five different things but all of them need a tremendous amount of resources and effort, and you only go 10% of the way with each of them, you’re throwing an article here and you set up an affiliate plan here, and you do a Twitter and Facebook post a week, it’s like you’re not doing anything well.
But if you dive into one and experiment, figure out is there any way to make this possible, you dig in for a month or six weeks or whatever it takes, and you do these sprints where you dig in, learn everything you can about it and execute on it or you hire someone to do that if you have the budget. Determine, “Is this going to work right now given my business, yes or no?” Answer that question and then move on to the next thing–that is much more of the approach that I would recommend and the approach that I’ve taken in the past. You don’t need many of those to work in order to scale your business. If you find one or two pretty substantial marketing practices and you figure out the angle and you figure out how to get in there, that can grow your business to well under seven figures. It doesn’t take 10 different marketing tactics to get there.
With that, we’ll wrap up for today. If you have a question for us, call our voicemail number at 888-801-9690 or email us at questions@startupsfortherestofus.com. Our theme music is an excerpt from We’re Outta Control by MoOt, used under creative commons. Subscribe to us on iTunes by searching for startups and visit startupsfortherestofus.com for a full transcript of each episode. Thanks for listening. We’ll see you next time.
Episode 335 | Lessons Learned from Unbounce’s 5 Split Tests Across 2.3M Exit Overlay Visitors
/
Show Notes
In this episode of Startups For The Rest Of Us, Rob and Mike talk through an article by Unbounce about split testing their exit overlays. The guys give you a full walk through and their opinions on the findings of the article.
Items mentioned in this episode:
Transcript
Rob: In this episode of Startups for the Rest of Us, Mike and I talk about lessons learned from Unbounce’s five-split test across 2.3 million exit overlay visitors.
Welcome to ‘Startups for the Rest of Us,’ the podcast that helps developers, designers, and entrepreneurs be awesome at launching software products whether you’ve built your first product or you’re just thinking about it. I’m Rob.
Mike: And I’m Mike.
Rob: And we’re here to share our experiences to help you avoid the same mistakes we’ve made. What’s the word this week, sir?
Mike: Well, we are in sunny Las Vegas as people are listening to this, and right now outside it is raining.
Rob: Is it supposed to rain on Tuesday?
Mike: No, I’m still in New England. We recorded a week ago early. Come on, man! Get with the program!
Rob: We’re time warping here. Yeah, so we record just a few days early, and I take off for Vegas in a couple days, and it’s supposed to be what? 70s and 80s? Pretty sunny there?
Mike: Yeah, I think I saw like 84 or 85, something like that last week when I looked.
Rob: It’d be nice. I’m looking forward to getting out and seeing the folks, you know, seeing the speakers, meeting the new speakers, and hanging out with people we haven’t seen for six to twelve months. It’s always exciting to kind gear up for that.
Mike: Yeah, it’ll definitely be interesting this time around just trying to keep up and pace ourselves for the two conferences. I know in years past every single year without fail, my voice starts to fail at about the second or third day. So I’m going to have to be a little careful about that.
Rob: I agree. The good thing is we’re not emceeing the Starter Edition, so we should have less talking. Although, you’re doing a talk there, so you’re really going to need to keep your voice going.
Mike: Yeah, my talk is the last slot on the last day so four or five days of talking and hopefully my voice will still stay with it.
Rob: Indeed. On my end, not too much new this week. Wanted to look at a couple recent iTunes reviews, and we’re up to 517 worldwide iTunes review. This one- I love the subject- “Best podcast ever” from John Turner, and he says “best podcast for anyone looking to start selling software or a SaaS product.” And we have Ray223 who says, “The oldest and the best. Thanks Rob and Mike for all the insights. Keep up the great work.” We have another from [Ludwig?] from Denmark, and he says, “If you’re considering have a startup, listen in here.” Really appreciate these reviews. If you haven’t left us an iTunes review, it goes a long way towards helping us to reach new audiences, to grow the show, and frankly, just keeps us motivated and keeps us from crying ourselves to sleep at night. I know Mike, you often pull up the reviews and read them to feel good about yourself, right? Oh, is it just me?
Mike: I think it’s just you, yeah.
Rob: Cool, so let’s get into the topic this week. We’re going to be talking through an article that’s on Unbounce.com’s blog. And the title of the article is “Lessons Learned from 2,345,864 exit overlay visitors.” And what they did is they ran split tests and some observational tests, which we’ll talk about in a second. They ran five of them over the course of two years, and it’s about 2.3-2.4 million visitors during that time. And so they just outlined the tests they ran, and with the results you would expect, and what actually happened. And I like how detailed this post is, and I like that some of the assumptions that we will make- when I say the split test you’ll think “I know which one is going to win.” Sometimes that’s correct, and other times it’s not. And so it’s kind of a fun walkthrough of people who are doing a lot of pretty sophisticated testing.
Mike: Do you think that that’s an exact number or just kind of a ballpark estimate of the 2,345,864?
Rob: I think they went into Google Analytics, you know, and just pulled up the number from the date of the first split test until today or whatever so I imagine it’s pretty exact.
Mike: So one of the things we wanted to mention before we kind of dive into this particular episode is the fact that there is a certain kind of minimum threshold you have to kind of be at in order for split testing like this to work. And I think Rob, you said the minimum you would even consider doing these kinds of split tests is when you get to about 30,000 uniques a month, right?
Rob: Yeah. You and I were talking before the show, and I ran some loose math, and let’s say these guys have 2.4 million unique visitors to their blog over the course of this 2 year thing, which is what they’re positing. So back of the napkin, 2.4 million is about 100,000 visitors a month, and at that rate, think about if you’re using exit overlay you’re around 1% to start with in terms of converting people to subscribers. So 1% of 100,000 is about 1,000 new subscribers a month. If you split test you could very likely get to 2% and that’s 1,000 to 2,000 and that’s a big difference. That moves the needle. As you start lowering the number of visitors you’re getting, let’s think about 50,000. That number’s going to move from about 500 to 1,000 new emails a month, and that’s still for me, still moves the needle. But as you start dropping down- 30,000? It’s like well, I’m going from 300 to 600. That’s cool. That’s about as low as I would go. If I have 10,000 and I go from 100 to 200 subscribers a month, I actually think at that point you should be focusing more time on generating traffic rather than split testing the traffic you have. So my mental math is somewhere in the 30-40,000 uniques a month if you’re going to start running experiments like this to have it actually move the needle.
Mike: So with that in mind, let’s start digging into these experiments. Let’s see what they found out.
Rob: Yeah. So they say back in 2015 they launched their first ever exit overlay. So it’s an exit intent popup that only appeared when you tried to leave the site. And the idea was they just wanted to collect more email addresses. And when they first started their conversion rate with the very first one, and it says “Conversion optimization tips you’ll actually use. Subscribe to our blog.” There’s no lead magnet, there’s no mini course, it’s just subscribe to the blog. And they were at 1.25% from the start, and we should probably take a quick break here. Some people hate exit intent popups. Some people love them for how well they convert. I think Unbounce takes a pretty good approach to it, and they actually write custom code to assure that you only see one even if they have a bunch running in different categories as people hear later. They were very specific to not hose the user experience because that’s the thing that sucks with exit overlays. If they keep popping up and you’re trying to read and you’re trying to go to another tab to do something, so this is a place where you can listen to these ideas, and you don’t have to do this with exit overlays if personally you don’t want to. This could just as well be a little Drip widget in your lower right, this could be a lead box, this could be embedded form in your website- all the stuff they do could still be applied to that, but they just happen to do it using exit intent.
Mike: Yeah, that kind of applies across the board to a lot of different marketing efforts. If it’s something that you’re not particularly enthralled with when you come across it- if it’s something that you just say “Aw, I can’t stand these things,” and you just keep closing them and they keep coming back, that really turns you off from trying them to begin with. But there’s also a difference between your experience on your website and the visitor who’s coming to your website. So you have to keep in mind what your end goal with these is and whether or not it even makes sense. Obviously you’re not the one browsing your site, but at the same time you are not necessarily representative of the person browsing your site. They may not care nearly as much as you do about it.
Rob: So we’ll kick into first experiment here in a second. In the post they say there are a few testing conditions they want to lay out. Number one: all overlays were triggered on exit, which means they were launched only when abandoning visitors were detected. Number two, they say for these first three experiments they compared sequential periods to measure results rather than true A-B tests. So they do say this is less scientific, more observational because you run them for thirty days and then run them for thirty days and compare the two time periods. They say that when comparing these sequential periods testing conditions were isolated by excluding new blog posts from showing any of the overlays. Conversion is defined as either completing the form, meaning they enter their email or later on they try something where it’s just a click, and we’ll get into that later. And lastly, it’s from January 2015 and ended November 2016, so almost two years. So that’s setting the stage. The first experiment they ran was to go from their generic signups which I talked about earlier which just said “Conversion optimization tips you’ll actually use.” and “Subscribe to the Unbounce blog” button. If you compare that with actually giving away content, the hypothesis is if we give away an e-book it’s going to generate more opt-ins. Both Mike and I would agree and most listeners that you’re going to increase your conversion rate if you do that. So they gave away an e-book called “23 Principles for Visually Designing More Persuasive Landing Pages.” And what happened, Mike?
Mike: Well, it looks to me like the conversion rate doubled from 1.27% to about 2.65%. Now the interesting thing here is you kind of mentioned this upfront- this wasn’t a true split test. The duration for the test itself was about half the time of the stuff they had running. So they ran the initial one for 170 days and then the split tests or the tests that they ran was for 96 days to offer this e-book.
Rob: The variant or the challenger, if you will.
Mike: Yep. And the conversion went from 1.27% to 2.65% so almost half the time and twice the conversion rate which kind of translates to the fact that they got about the same number of conversions in half the time. That’s a phenomenal increase to be honest.
Rob: It’s really cool. Yeah, this is good. I feel like this supports a lot of stuff that we’ve talked about over the years and what you would naturally think so that’s good. Their observation on this is offering tangible resources versus non-specific promises compositively impact conversion rates.
So experiment two gets a little more complicated. So they went from their single field- meaning just getting an email address- and you think about at this point they’re at a 2.65% conversion rate, they’re giving away an e-book, they’re asking for an email. They said that data people always spoil the party and someone internal to Unbounce wanted to get more information, not just the email. So they added three other form fields, so instead of just email they asked for a first name, do you work for an agency, and your company size, and they knew this was going to tank when they knew it was going to tank conversion rates, and it’s exactly what I thought would have intuited as well. Every form field you add from 1, 2, 3, and up, you are going to almost in all cases a decrease in conversion rates. And this once again confirms that. Take us through the results.
Mike: So the results basically went in the opposite direction. Just to point this out as Rob said at the very beginning, these were all run against different people, so it’s like somebody saw one and they ended up seeing another one. It’s really different segments of their audience that were seeing these, so again, the time periods were different that they used. And it looks like they did not want to run this one for very long just because the results were so bad, but the conversion rates went from 2.65% down to 1.25%.
Rob: D’oh! Right back to where they started. They’re still giving away that resource but just adding the three fields just tanked them.
Mike: Yeah, so it’s pretty obvious that even just looking at it you probably don’t want to fill out that form. If it’s in exiting intent and you are already inclined to leave, making it more difficult to send them information is just not going to work out in your favor.
Rob: I like the quote the post author says, “I knew this was for a good reason. Building our data warehouse is important. Still, a small part of me died that day.” And then it got worse. Basically since they added all the fields to it they had to expand the size the overlay and they just made it bigger so it could hold the form fields and then they realized that they had made it so big that the size was too large for many people’s browser windows. So the overlay only fired two out of every three visits. So not only did they have a lower conversion rate, but they cut their potential audience by a third. So then they redesigned the overlay to make it smaller and fit, but internally they decided that even though it was a lower conversion rate, they needed that data. And so now their new baseline is back to 1.25% which is kind of hard as a conversion rate optimized to make that much of a gain to double and then have to go back.
Mike: It depends on your situation because this is one of those places where you really need to make a judgment call about whether you need that data or not and whether you need it now. I think it’s this case with the types of questions they’re asking, I think they could probably get away with not asking certain things. But they went from asking just an email to first name, email, do you work for an agency, and company size. And those individual things really let someone on a marketing team target down and send a highly specific content to those people. You’re not going to send something about a freelancer to somebody who’s running an agency with thirty-five people. So it allows you to target the content you’re sending them and make sure that it is speaking their language and talking directly to their pain points because they’re going to have different pains than a freelancer. And if it’s important to have that information now, awesome. But I would also question whether you need to have it on that page. So one of the things that I’ve seen is that you can do this type of thing where you ask for a very limited amount of data and then on a next page you ask for more because they’ve already bought into the premise at that point. Let’s say they give you an email address and then it flips to another page and says, “Hey, thanks for that information. Can you tell us a little bit more info so we can more specific about the things we send you?” And at that point you can ask them, and if they don’t give it to you, you can follow up later and try and get it later or there’s different ways down the road that you can get it. But the question is do you need that information right now?
Rob: Yeah, and that’s called progressive profiling, if people are curious. There’s a really good tool that actually has a tight integration with Drip that does exactly that. You can set up all these fields and you can progressively over time- the first time you visit it’ll ask you for email, and then the next time it knows it already has your email so instead of doing nothing it’ll adjust for your first name or it’ll ask for something else. It’s pretty sophisticated stuff. That tool’s called Convert Flow, if you’re interested.
Alright, onto the the third experiment. I like this one. It starts off with the quote “It seemed like such a good idea at the time…” So the third experiment is challenging one overlay like they have now, just the one overlay giving away the e-book with the four form fields versus ten different overlays. And the ten different overlays are highly targeted or hyper relevant to each of their ten blog categories. So you can imagine on a blog you might have categories and one of them is split testing, one is copyrighting, one is email, one is paper click, one is social. So creating a separate overlay for each of those with hyper relevant headline/pitch, all that stuff. And so they did this and they said it took three hours maybe to do this, and then they have a table, and the results are kind of crazy. The email category result was .45%. So remember their baseline is currently at 1.25%. So their email is at .45%, their legion and content marketing is at .85% so there’s kind of the bottom end ones. And then their copyrighting one is at 3.22%, and their landing pages and mobile optimization is at 2.46%. So huge range now, but having that more granular data is super more interesting because now you know you have some big wins and some big losers you can just focus more on the losers rather than trying to do it across the board. They only did it for ten days. They said their conversion rate was a combined 1.36%. So it is slightly higher than the 1.25%. They said it eventually crept up to 1.42% after an additional quarter million views, but it didn’t do anything as much as they had hoped. So to take a step back, if you take a look at their copyrighting category they say “How to write killer landing page copy” versus their conversion rate optimization category was “How to Get Started with A-B Testing” and then the next one is UX and so the headline was “The Seven Deadly Sins of Landing Page Usability.” And so I don’t know that these were e-books, and they don’t say that each of these was as effective as that 23 Principles of Design e-book they originally gave away, but that was the idea here. That’s how they were trying to target to the specific category.
Mike: What I find interesting about this is that they decided to just go back to the baseline and use the average of all of these because it seems like there are certain cases where the targeted headlines work a lot better. So for example the copyrighting and the campaign strategy and the mobile optimization- those things have a much higher conversion rate. But then if you look at the other ones, like the email category or the social category, the conversion rate on those is less than 1%. So it makes sense to at the very least go back to the baseline but I kind of question why they didn’t go with those specific ones in those particular categories that were overperforming 2-3x what the baseline was.
Rob: Yeah, I agree. The average was a little weird, and I think they’re doing that just to kind of give you a general idea because now that you have this data, what would you expect is the next thing to do? It’s to keep the ones that are at 2% and above and only try to optimize the low performing categories, and that’s exactly what they did. They were going to test new offers over five categories that had low conversion rates and reasonable enough traffic volume to make it worth the test. Because some of these, now that you’re getting more granular, some might only get 1,000 views in a month so it’s not really worth diving in. What they did is they were then doing resource versus resource, and so they were basically trying a different headline to give away an e-book.
Mike: One thing that comes to mind now that you mention that that kind of leads into the next experiment they did, is if they start changing that and they start running different types of ads in some categories versus others then it changes a lot of the math around and it becomes a lot harder to test that stuff.
Rob: Oh totally. Yep. And so the result of this one, they don’t really go into much detail. They just say they saw a slight improvement in one of them, in a couple of them they saw a dropoff, but in legion and content marketing there was a dramatic uptick, and the results were statistically significant. So they’re making progress. They had five low-performing ones, and they basically knocked two of them out, so really they just have three low-performing ones.
And so for their fifth and final experiment as of this writing, they wanted to test legion overlays, which is what they’ve had the whole time asking for email and first name, etc. versus click through overlays. And so they talk about have an exit intent popup that basically has a headline and then a button that says either “get the video” or “get the e-book”- they’re still giving something away, but then that takes you to a landing page. And that landing page has a lot more text and a lot more conversion-oriented stuff. And so that’s what they tested.
Mike: So to contrast this experiment versus the other ones: the other ones were tweaking the headlines, they were asking for more or less information. And in this particular experiment what they mean by “click through” is they would make an offer to them, and the person did not have to fill anything out. They could simply click on a button that would take them to a landing page that would then ask them for information. So essentially they were providing a mechanism for them to say yes, I’m interested or not, and they could quickly answer that. And when the person clicks on the button it takes them to that landing page and then they can fill out that information so it becomes a two-step process instead of a one-step process.
Rob: Right. And this is actually something Clay Collins has talked about for years, and I heard about his podcast he used to have called The Marketing Show, and then when he came on our show a couple years ago well before the first [?] thing happened, he talked a lot about not popping something up and asking for information and they go with this juiced up opt-in where you can click a button and then the next step was a form and it just feels more- it tends to have better conversion rates and tends to make people feel more welcome and like you’re not just asking for something right up front.
So for this experiment they point out for this to be successful, the conversion rate on the overlays would need to increase enough to offset the dropoff that they expect by adding that extra landing page step. Because you figure 1.25% of people might submit one with the form fields in it, and then maybe after two or three times of having to click the button but now you need to get a lot of those folks to actually fill out the form on the landing page. So not surprisingly engagement with the overlays increased amazingly the ones with the buttons. And engagement is just clicks. And so the one with the form fields was .79%, and the ones with just the button click through was 3.62% so that’s about 4 1/2 times better. So that was just getting click throughs. In the end in their CRO category, the overlay with the button where you click through to the landing page, that essentially netted them twice the leads. So they had 100% improvement in terms of the net leads they received by having someone do a two-step opt-in instead of just popping up a form field in front of them. And in their legion category (they have this category on their blog) it was about a 50% (it was from 45-56) so little under 50% increase. And so they only did it in two categories and their next step would be applying the same format to all the other categories and measuring the results. So two-step opt-in for the win.
Mike: Yeah, so if it’s confusing, to kind of consolidate that, it is a two-step process versus a one-step process, so depending on where you calculate the conversions for that you have to take those into account. And not just the percentages but also the net result of it. So it’s not just about the people who click on that link and go to the next page, it’s about who clicks on the link and goes to the next page and submits their information. And what is the net result of that versus the original. And in both the cases here they had either a 100% or a 50% increase. Both of which are pretty significant especially considering the volume. And they took the time to say okay, this makes sense for us to kind of expand this and go out through the rest of our categories and implement it across the board.
Rob: And so that’s where they leave off. They’re kind of summarizing stuff that ended three months ago, and I’m sure they’re running their next tests as we speak. The nice part about this, I like their concrete examples. There’s so much that you read about split testing that can be vague or rule-of-thumb, and these- your mileage my vary, but a lot of the stuff they intuit really comes to pass, and I think it’s always helpful to get ideas for your own split tests. You may not have time to create ten different overlays, and unless you have 100,000 uniques a month to your blog, I wouldn’t create ten overlays. But especially some of those earlier tests were pretty big wins for not a ton of effort, for writing a new headline and giving away a resource you already have on your hard drive.
Mike: You know, Quickbooks is going to love you for saying “Intuit” all the time.
Rob: Yeah, I know. For using their intuition, yeah. I’m going to come up in their Google Alerts a bunch.
Mike: Yep, their SEO guys are going to start calling you. Interesting side note here, one of the things that Unbounce talks about is that they also have a list of what they’re going to be looking at next. The first one is a charter test which is what happens when they test their on-exit trigger. I guess there’s a 15-second time delay. And then there’s a referral test, which is what happens when they show different overlays to users from different traffic sources. And then the third one they’re looking at is new versus returning visitors. Do returning blog visitors convert better than first-time blog visitors? And I think those are all going to be interesting tests. If you’re interested in taking a look at that, go over and check out Unbounce’s website, subscribe to their blog, and I’m sure they will post out the results when they get those, and we’ll be sure to check it out.
Rob: While you’re at it, go to the lead page’s blog and subscribe too. They turn out good content like this as well. Got to plug the company man.
Mike: I get it. Well, I think that about wraps us up. If you have a question for us, you can call it into our voicemail number at 1-888-801-9690 or email it to us at questions@startupsfortherestofus.com. Our theme music is an excerpt for ‘We’re Outta Control’ by MoOt used under creative commons. Subscribe to us in iTunes by searching for “startups” and visit startupsfortherestofus.com for a full transcript of each episode.
Thanks for listening. We’ll see you next time.
