So obviously, there was a cost associated with this incentivization. I mean, you had to actually pay for these hotels at the end of the day. >> Yeah. >> So there was a cost associated with that. How did you track and evaluate that expense compared to the return that you were getting? >> Yeah, so what's great about really digital in general is that you really can see where all your traffic is, and usually in one pane of view. So we were really big on Google Analytics. There are a number of platforms that do these types of analytics, but we were using Google at the time. And so, we were able to see what traffic was coming from these particular blogs. And so, we could see just how many page views or just how many visitors we were getting from a particular source. So we were somewhat basic in that we just captured and said okay, this particular blogger spent this much money in credit, and then we saw how many people came through there. And you can actually attribute an actual customer transaction back to that original source. So that's how we calculated ROI is to say okay, how much revenue came from the original source of that particular blog? And then, compare that back to the original investment. >> At the end of the day, how many pieces of native content did you end up testing and running as campaigns? >> Yeah, we had hundreds of articles written about us in the first couple years. And we probably used about 15 or 20 articles during that same time frame. And for each one of those, you're really creating a number of different combinations on the advertising platforms themselves. And the reason for that is because a different headline is going to attract different audiences and be more effective as far as how many people that go to a CNN.com or a publisher's website and then actually click through on that advertisement to one of these bloggers that have written about you. And so, you're constantly trying to put together combinations of headlines and photography to try to get your click rate up as high as possible. For a bunch of reasons, one, you want to make sure that you're getting a bunch of different people, so people that react to different things through that funnel. But also, it's so important to get that click rate up because ultimately, your cost or your CPC, or cost-per-click, is going to be driven by how effective you are for that publisher. Those publishers only get so many impressions and they want to make as much money as possible, too. So if you think about it, the more clicks you get, the less they have to charge you. >> Yeah, the less effort they have to go through to get your ad served. Of course, the less inventory they use, the less they can charge you. >> Right. >> That's awesome. So as we've talked about, these campaigns are really iterative in that you've gotta take a lot of time to set them up, but you're constantly tweaking things like an image or the text that goes along with a preview the consumer might see before they click and engage with that blog post. How many hours a week would you be spending in Tabula and Nativo setting up these campaigns, making changes, AB testing campaigns? >> Yeah, I think that when we're kind of going full throttle, it was really a full-time job. Probably a full-time employee was required to do it. >> To do it well. >> To do it well. And not only to set up the campaigns and all that, but to run the proper analysis, to think about what are the next moves or what are the iterations that you want to make to try to make this more effective. >> Yeah, and so, when you were looking at those campaigns in the platforms themselves, what were some KPIs that you were looking at or even just metrics that you were looking at to assess the viability or the effectiveness of the campaign? >> Yeah, so I think that we looked at three very specific metrics as kind of the key metrics for how well Native was performing for us. And so, I think first is cost-per-click. I think at the very base fundamental level you're trying to figure out how do you get an interaction with a customer, even if its not to our website, but to a blog for the cheapest amount as possible, right? And I think that Native was extremely competitive on that front. Next, we were looking at what we call cost-per-user. And so, when you got to TravelPony to actually be able to view the different hotels that we offered, you had to do a free sign-up for an account. And there's a number of reasons why we did this. But this was great because we then captured an email address and were able to send them marketing about hey, we have some promotions going on. And that helped us get some additional transactions. And so, it was very, very valuable for us to get somebody converted to a user. And so, the third one, really, for us was cost-per-acquisition, our CPA. And that's really cost-per-customer acquisition. So getting somebody to actually make a purchase. At the end of the day, we were an e-commerce company and we needed revenue in the door or else we weren't going to survive. And so, that was a really, really big one for us. And when we looked at CPA for native advertising versus some other platforms, it was just hands-down the most effective for us, especially early on. I will say that we definitely looked at some other metrics like just how many visitors, how many new visitors, how many page views, and how long they were spending on the website. Which is really important in understanding what your engagement looks like. But I think that we were very much a transaction, really thinking about getting somebody into the funnel and what are your costs there, but also revenue, right? Just revenue and cost, how do we get to profitability? >> Yeah, and so what were some tweaks that you actually made on some of those campaigns to get that profitability up? Did you really see a big difference when you would manipulate the creative? >> We saw extreme differences in changing creative. And so, as you can imagine, we had probably invested several thousand dollars in stock photography trying just different types of images in there. >> And taglines. I mean, we spent hours upon hours and hours of recreating, trying to understand what element of a particular tagline was driving that cost-per-click down and the conversion rates up. And so, yeah, I would definitely be lying to you if I said that that part of it was a wizard and you just hit the play button and you're good to go. There's a lot of work that goes into managing advertisements.
