This module covers qualities of the data engineering solution beyond functionality and performance. It addresses reliability, policies and security. Designing for security and compliance. Security is a broad term, includes privacy, authentication, and authorization, and identity, and access management. It could include intrusion detection, attack mitigation, resilience, and recovery. So, security really appears across the technology is not in just one place. You need to be aware of the granularity of control for each service. The exercise is, imagine there are two people, one needs access and the other must not have access, what's the smallest unit or degree of control the technology supports? Can you distinguish security to an individual field, or to a row, or record, or to specific columns, or to a specific database, or entity, or just the kinds of actions that can be performed on the service? Commit a security checklist to memory. Sometimes just running down a list will rapidly identify a solution. A key concept is to assign roles to groups and use group membership to grant permissions to individuals. How will the service be monitored or reported and how often will these items be reviewed? Finally, you need to know what kinds of logs and reporting are available from each technology. A policy is set on a resource and each policy contains a set of roles and role members. Resources inherit policies from parents. So, a policy can be set on a resource, for example, a service and another policy can be set on a parent such as a project that contains that service. The final policy is the union of the parent policy and the resource policy. What happens when these two policies are in conflict? What if the policy on the resource only gives access to a single, let's say, Cloud Storage bucket and restricts access to all other buckets? However, at the project level, a rule exists that grants access to all buckets in the project, which rule wins? The more restrictive rule on the resource or the more general rule on the project? If the parent policy is less restrictive, it overrides a more restrictive resource policy, so in this case the project policy wins. Folders map well to organization structure. It's a way to isolate organizations, or users, or products while still having them share billing and corporate resources. There are many encryption options for data at rest and in storage. Default encryption at rest uses the key management system, KMS, to generate KEKs, which are key encryption keys and DEKs, the data encryption keys. When you use Cloud Dataproc, cluster and job data is stored on persistent disks associated with the compute engine, VMS., in your cluster and in a Cloud Storage bucket. This PD and bucket data is encrypted using a Google generated data encryption key, the DEK, and key encryption key, the KEK. Customer-managed encryption keys, CMEK, is a feature that allows you to create, use and revoke the key encryption key, the KEK. Google still controls the data encryption key, the DEK. Client-side encryption simply means that you encrypt the data or file before you upload it to the cloud. Key concepts, cloud armor, cloud load balancing, cloud firewall rules, service accounts, separation into front end and back end, isolation of resources using separate service accounts between services. Because of the pervasive availability of firewall rules, you don't have to install a router in the network at a particular location to get firewall protection. That means you can layer the firewalls as shown in this example. Because of pervasive support for service accounts, you can lock down connections between components. When faced with a security question on an exam or in practice, determine which of the specific technologies or services is being discussed, for example, authentication, encryption. Then determine exactly what the goals are for sufficient security. Is it deterrence? Is it meeting a standard for compliance? Is the goal to eliminate a particular risk or vulnerability? This will help you define the scope of a solution, whether on an exam or an application.
