Hi again Evan here. In this video, I'm going to walk you through three labs that provide solutions, to help your engineering teams drive cost optimization. One quick way to optimize costs, is to get rid of resources that you're not using. In our first lab, you'll learn how to identify and then delete unused IP addresses automatically. In the next lab, we'll walk you through identifying unused and orphaned persistent disks, another easy way to reduce your cost. The third lab, we'll show you how to use stackdriver cloud functions and then cloud scheduler, to identify opportunities to migrate storage buckets, to less expensive storage classes. Okay, so in this lab we use cloud functions and cloud scheduler, to identify and clean up wasted cloud resources. So, in Google Cloud platform, when a static IP addresses reserved but not used, it accumulates a higher hourly charge, that if it's actually attached to a machine. In apps that heavily depend on static IP addresses, in large scale dynamic provisioning, this waste can become very significant overtime. So what are you going to do? You create a compute engine VM, with the static external IP address, and a separate unused static external IP address. Then you'll deploy a cloud function to sniff out and identify any unused IP addresses, and then you create a cloud scheduler job that's going to run every night at 2 AM to call that function and delete those IP addresses. Once again, just remember that GCP's user interface can change, so your environment may look slightly different, than what I'm going to show you in this walkthrough, let's take a look. So here we find ourselves back in another quick lab, now this one is all about using cloud functions to do magical things. In this realm of world, we're just going to be creating cloud functions to clean up resources aren't being used, in this particular case we're going to be creating a function that's going to clean up unused IP addresses. So the actual function code as you're going to see a way down there, is just a little bit of I think it's written in JavaScript, just a functional JavaScript code. The great news is you don't have to write the function yourself, Google Cloud engineers have provided a lot of these functions on their External GitHub repository, which is kind of cool. So if you wanted to just use literally the things that you're using inside of this lab right now, you can copy and paste into your own project at work as well. So highlighting the things that you're going to be doing, you first need to create a virtual machine. And like we said in the introduction you're creating a couple external IP addresses, one that you're going to use in one, and one that's going to be unused. And then you can be building that code that's actually going to go through, sniff out any that aren't in use, and then bring them down now that's only if you manually triggered it. So the second part of this lab, is actually to scheduled that cloud function to run in this particular case nightly at 2 AM I believe, and that will automatically invoke that function and then do that clean up automatically. So once you set it up once which is great, it will just run in perpetuity. So a couple of different things that I want to highlight. So the first thing you need to do is inside of the lab youll be working off of code that already exists inside of this GitHub repository. In fact this is for the last three labs, anything that has to do with cleaning up things, is going to be based on this repository here, so I'll show you just very briefly. The first lab is going to be on the unused IP addresses, the second cleanup lab is the unattached persistent disk or pd. And the third lab is going to be migrating storage to a cheaper storage class, if that bucket isn't being used that actively sounds kind of cool? So the code for the unused IP address is just this JavaScript code here. Again, you have to write it, you just have to invoke it, deploy it for your own functional code, but you can kind of see what it's doing. It says all right, you can actually view the walk for this the function is called. There are how many IP addresses that are out there for each of them, if it's not used or if it's not reserved, then you could potentially delete it. If you can't, it will say could not delete the address and then boom just 60 or so lines of JavaScript code that basically says, all right, there are statuses associated with these IP addresses. I want you to have some logic around them to basically say, okay iterate through all the IP addresses that people on my team have created throughout my project, and then remove those ones that aren't used. So this is the actual nuts and bolts of the lab, is this public JavaScript code here in the GitHub Library. Let's take a look at how you actually deploy and invoke them. So after you've cloned that project code in, what you need to do then is you need to kind of simulate production environment. You're going to create the unused IP address and they used IP address, then you're going to associate them with a particular project, and then you want to confirm that they're actually created. So actually show you just this one command right here. So this says, hey, gcloud compute, how these commands are structured by the way, is Google Cloud what service or product you want to use? This is compute engine, And then for IP addresses, it's just called the addresses and they just want to list and I only want to filter those for my particular regions. Just a flag filter that you can have. I've actually already run through this labs you can see that there is no IP address that doesn't say it's not in use because I actually already ran the function and it deleted it. But as you work with it way through your lab, you have a long list of unused IP addresses. That trim down to just the ones that are in use, so it's pretty cool. So most of the Magic. Again, since this is using the command line to deploy the cloud function. He's going to happen here, but once you actually validate that it works and it cleaned up those IP addresses that weren't in use, what you can then do at the end of the lab is basically, say, hey, I don't want to come into the command line every time invoked this function. I'll show you what their function invokes. Looks like right now. Deploy it, trigger it. Here we go. So after you deploy it and you get ready to work, the last part of the lab is actually to schedule it so it uses a gcloud scheduler. Cloud schedulers are relatively new product. This essentially just like a glorified Cron job that Google manage all the maintenance and the hardware behind the scenes for you. You can use the VM. I use the the SSH command line terminal to create it, but then I also like to go into and see where it actually is. I think something like the admin tools here, Tools we want cloud scheduler with little clock here. And one of the things that you can do is this one was the unused IP addresses. The next lab you'll be creating one for the unattached persistent disks jobs. You can instead of invoking it via the terminal, you could do the run now as well, which is kind of cool. So it goes lightning fast because just running that JavaScript code and then just killing out all the IP addresses that are used, which is great. So I'm a big fan of seeing after you've created all your work inside of the terminal, you can view all of your jobs either via the terminal or within the UI as well and then boom, it automatically runs at a frequency and again much like a Cron job. This denotes 2AM. Every night and their website utilities out there that help you convert time into a cronjob syntax right here. Don't worry about that too much. All right, so that's the first deep time that you've had into using a cloud function to do something a little bit more sophisticated than hello world. In our first cleanup use case, was removing those unused IP address. So go ahead and try that lab and then all that knowledge that you're going to be learning there will make the next two labs very easy because you going to be doing the same things operating off of the same repository. Good luck. In this lab you'll use cloud functions and cloud scheduler to identify and clean up wasted cloud resources. In this case you'll schedule your cloud function to identify and clean up unattached an orphaned persistent disks. You'll start by creating two persistent disks and then create a VM that only uses one of those disks. Then you'll deploy and test a cloud function like we did before that can identify those orphan disks and then clean them up so you're not paying for them anymore. Let's take a look. Here we are into the quick lab for cleaning up those unused, an orphan persistent disks. Again, one of my favorite things about those quick labs is as you're working your way through the lab, you'll get those points as you complete all those lab objectives automatically. Gloves is smart and knows whether or not you did the work or not. But it's also really fun to get those perfect scores all the way at the end. So as you scroll down and look at this lab, you already starting to get familiar with cloud functions, and again, those are those magical server list triggers that can look for things to happen, be triggered, and then do other things. So the other lab that you worked on just before this was cleaning up this unused IP addresses and you set that up as running as a cronjob you in the cloud schedule at 2AM. Same general kind of concept for this lab, except you don't care about IP addresses here. You care about persistent disks. Those are those hard drives are attached to the virtual machines because again, inside of Google you have the separation of compute and storage. Just because you get a virtual machine doesn't mean that you need to have that number. Virtual machine running 247/ just to keep that data alive. If you need compute power for an hour and you need persistent storage for in perpetuity, you can actually separate those, which is kind of cool. But ,say you didn't want that data just around when you had no virtual machine associated with them. You can identify those orphaned persistent disks as we mentioned in the introduction, you'll be creating two of those persistent disks. The VM is only use one of them. Going to detach that disk and then going to create some code or copy some code from the repository. That's going to be able to look through and find any of those disks that were never attached, never used. Basically, say hey why you're paying for stuff that you're not using. Then you deploy that cloud function that will remove all this persistent disks. And then lastly so you don't have to constantly wake up every morning and press that button to say remove persistent disk that would be really boring job. You're going to create a Cron job, the cloud scheduler to automatically do that for you. So again if you already did the last lab, you've seen the demography of the last lab You'll be working off of the code that's in this public Google repository, for GCF Automated Resource Resource Cleanup, GCF is just Google Cloud Function. Here we have the unattached persistent disks. Instead of JavaScript this time, it's actually written in Python, which is pretty cool. It's a little bit more involved, so basically says, all right, well, I want to find and look at and delete the unattached persistent disks. So much like you iterating through the IP addresses before in the prior lab, here you're getting the list of all the disks in iterating through them. And then hey, if the disk was never attached, and that's a metadata associated with the disks that it was never attached, there's a timestamp associated with it. In fact, it's actually just last attached timestamp, it's not present. Then you're basically going to say, all right, well, this disk was never attached to anything, was never used, I'm going to go ahead and delete that. So this code will run and handle all of that code automatically for you. You're not going to be writing Python, don't worry about it. This is just code that you can lift and shift and use on your own public applications. The main argument that you want to be considering here is deploying this code as a repeatable cloud function and then having an invoke at a regular nightly interval, say every night at 2AM as the cloud schedule will help you. So back inside of the lab, the orphan persistent disk. Let's take a look at some of the things that we can do, we'll run some of this too. So we just look through the repository, after that, you're going to actually create those persistent disks. Here is where your give just orphan disk, great, unused disk, great. You actually going to create those two disks, so I'll go ahead and just run these now. Inside of cloud shell, let's see what directory I'm in. I'm in the root directory. I need to go into the wherever the code for the unattached persistent disk is. Now I'm in there. As you saw, we were just looking at that Python function before maine.pie. By the way, if you're not familiar with Unix commands, a couple of useful ones are LS, which just lists the contents of a given working directory. CD says change directory. It's kind like double clicking on a particular thing into the directory, so it's double clicking unattached PD, and then CAT shows the contents of the file, doesn't do anything with it, but it shows the contents. So that same Python code that you saw before is now just visible on the screen here. So what do we want to do? We want to, I didn't want to copy that. We want to create some persistent disks, have some that aren't going to be used and then delete them. So we're in that directory, we're going to create some names, and this is literally what you're going to be doing inside of the lab, is working your way through copying and pasting. I'm hovering over these boxes, clicking on the clipboard to copy, creating all of them. I need to make sure that my project ID is set, so let's see. It's probably because I skipped an earlier step inside the lab, but the great news is if your project ID is not set, there is a command for that as well. So we'll set the project ID. It's updated properly. Now we'll try again. Export is just basically saying define this as a variable. Create those two disks. Now it's because I didn't run the export project ID up here. Boom, done. This is why it's super helpful to go through the lab in order. And then let's create the disks which should now work, make this a little bit larger. It's creating all these disks automatically and this is exactly what you could be doing through the UI. Here we go. We've got some disks that are ready. Let's validate these disks, just were created before we blow away the ones that are unattached. What this do we have? Whoa, we got a lot, great. We've got an orphan disk and unused disk, and I have other stuff that I really don't want to be deleted, [LAUGH], so hopefully this code works as intended. So orphan disk and unused disk, keep your eyes on those. And, of course, as you working with lab, you click on check your progress in your real lab instances as well. I've created the VM already, and then, I'll give it just a different name this time. Let's see. So here, we're going to create that virtual machine instance, and then look, we're giving it the disk name orphan disk, which, I bet you can tell exactly what we're going to do to it. So right now, we have a virtual machine that's using this disk, so the next thing in order to get orphaned, we've gotta detach it. So let's see, inspect to make sure that it was actually attached to the disk. Boom, and that's last attachment time and everything in there. Now, lets orphan it. Detach the disks marked orphan, just a command to detach it. So now it's off in the world. Let's see detach disk disk instance, my name for this demo I just have a .1. Boom, it's going to detach it. And now, a detached it and we're going to view the detached disk, it is orphaned, it is detached. Great, now the last part of this code is actually deploying that cloud function that will sniff through and look through all the disks that are out there and then detach them. It's having you inspect that Python code just kind of be familiar with it, again, you don't have to write any of that Python code yourself, but getting with a familiarity with it can't hurt. Okay, so now I've already deployed the cloud function but before recording this video, I've scheduled it. Now what I want to do, is list all this will be the magic that you're going to labs, I'll list all the disks that are there. Show you an orphan-disk in an unused-disk, and then now if I got everything set up correctly, I'm going to go into my [SOUND] Cloud Scheduler. I'm going to show you just using UI here you can use the command line as you wish. Unattached persistent disk job, boom Run now, class in a second to run it, right? And let's see if there's still there, are they gone? All right so as you see here, we're just about to run that clean up of the unattached persistent disks. We've got an orphan-disk, and then one that was just never used, but see if and hopefully that code runs gcloud compute disk, we've already run the function, it takes up to a minute for that actually to run. It will submit the function, but sometimes the code will take a little bit longer, so I've gone ahead and ran that gcloud just compute disks list shows the disks that are out there. And if you notice there are two disks that are no longer in here, the one that was unused, and the one that was orphaned. So I can say with certainty that the code works, at least when I recorded this video, so go ahead and inside of your labs. Experiment with that and then maybe create three unused ones, or couple orphaned ones, and just get familiar with how to create deploy those cloud functions. And then get the ability to invoke them manually via Cloud Scheduler or automatically via Cloud Scheduler Cron job frequency, give it a try. Gcp provides storage object lifecycle rules, that you can use to automatically move objects to different storage classes. These rules can be based on a set of attributes, such as their creation date, or their lives state. However, they can't take into account whether not the objects have been accessed. One way you might want to control your costs, is to move newer objects to nearline storage, if they haven't been accessed for a certain period of time. So in this lab, you'll create two storage buckets in generate loads of traffic against just one of them, and then you create a stackdriver monitoring dashboard to visualize that bucket's utilization or usage. After that, like we did before, you'll create a cloud function to migrate the idle bucket to a less expensive storage class. And then we can test this by using a mock stackdriver notification, to trigger that function, let's take a look. Now one of the last optimization strategies that you're going to see here, is saying all right, well I've got objects I'm storing inside of Google Cloud storage bucket, or a GCS bucket. What happens if I have them in a storage class like a regional or near line or there's a more efficient way to store those assets depending upon their usage? And how can I migrate them, move them between those storage classes, automatically? So one of the first things I want to show you is just what all the difference towards classes are, and you'll experiment with these inside of your lab. This is just the URL for Google Cloud storage in the different storage classes that are out there, and this all shows just the storage classes that are available. So generally for standard if you just create a Google Cloud storage bucket, it will be just standard storage that's accessible, and you don't need to specify any particular classroom when you're first creating it'll default the standard. But if you don't use your data that often say it's not a public bucket that gets a lot of traffic, and you want to enable some cost savings like for may be archival data. Or you want to automatically say, well, if you're not using it let's put it on something that cost a little bit less and is accessed a little bit more infrequently. So, that's when you can actually shift data that's stored in a GCS bucket for standard storage, and then re-class it, it will re-classify into something like nearline storage, or even cold line storage. If it's may be accessed like once a year or once a quarter instead of once a day for something like standard storage. So, now that you're familiar with the fact that different buckets can have different storage Classes, let's get back to the lab. So the lab here, is going to walk you through the different types of storage and then you're going to be creating different storage buckets. I've already created these buckets a little bit before, but you're going to be running through just the same repository before we're going to be migrating the storage. You're going to be creating a public bucket. You'll be uploading a text file that just says this is a test. And then you've been creating a second bucket that just doesn't have any data in it and then spoiler alert we're going to call that the idle-bucket or the bucket it's not going to do anything. So you've got those two buckets and one of the really cool things that you can do is you'll set up a Stackdriver Workspace in Monitoring dashboard. That's going to see the usage of each of those different buckets. Similar with how in a previous lab you monitor the CPU usage instead of this lab just monitoring the usage of the bucket. Again, Stackdriver is very flexible in terms of finding a resource on Google Cloud platform and monitoring how well it's used. After that one of my favorite things to do is using Apache library I think this is Apache Bench to serve traffic, fake traffic to that particular text file. So let's do that right now this is kind of fun. So hopefully this will work let's see I don't want to be unattached persistent disk. I will actually want to be in my great storage let's see ls into the migrate-storage. This is where the Python code that actually handles those storage actually lives which is cool. So let's see if we can just generate the traffic. Now the Apache Command is not found so one thing is that you have to do is you have to install Apache Bench serving library. So go ahead and install that. And then once that's available were served 10000 requests. Two are one of the Republic buckets. So as you can see here, I'm in the Google Cloud storage page. How you can get here is just the navigation menu under not compute, but storage this time I'm going to the browser. I have a couple buckets. The two that would be creating as part of this lab where the serving bucket which has the text file and you can see it's marked as public, which means anyone on the Internet can access it. And the idle bucket, which is doing nothing. It's already been reclassified to nearline storage, as opposed to something like standard original. That's because I've ran the cloud function to make sure that this demo worked before we recorded it. Okay, so now let's serve a ton of traffic. We've done that command and then boom. Benchmarking be patient. A 1000 requests like 1000 different people went to hit that text file 4000, 5000 thousand so you can see if you're on your Stackdriver Dashboard it's like spiking up through the roof. So we're going to be doing later is going to be saying all right well, this one particular file on this bucket is getting a ton of traffic, so regional storage is perfectly fine for it, but this other one is got nothing. Nothing is being accessed and there's nothing in there to be accessed. Let's move it from say regional to nearline. And that's exactly what the Python function is going to do that you're going to be creating a Cloud Function for an wrapping that inside of a Cloud Scheduler as well. So back inside the lab after you've done that artificial traffic, which is really fun. I like being disturbed like dosing yourself, right? You see the actual code that's going to be doing the migration is basically says, alright well let's update it to nearline storage instead of it's not used that much. Same thing as your previous labs you deploy that function, you'll give it an http endpoint that cloud scheduler can then invoke, and then you'll make sure that it actually gets set up with a logging feature where you can see them actually being deployed via the JSON file. And then let's see for us, I've already invoked the function and let's just confirm that it is in Nearline Storage. Boom, so is moved from a more frequently accessed storage class, likely regional or standard. And it has been reclassed into something that's a little bit cheaper because they thought is you're going to be accessing that data more infrequently, as evidenced by the fact that it wasn't given 10000 units of traffic and it re classified it automatically to Nearline. Alright, that's it for this lab. Good luck with your attempt of it and keep in mind for quick labs you can execute them more than one tim. So don't worry if the timer runs out if you didn't complete all of your activity tracking objectives, you can always click end lab and start it again for another fresh hour at the lab. Good luck.
