Category: Cloud Computing

Key Facts for Google Cloud Platform Engineer

by Davin LewisCloud Computing / Google Cloud PlatformLeave a Comment on Key Facts for Google Cloud Platform Engineer

Key Facts for Google Cloud Platform Engineer – These are key facts and concepts around Google Cloud Platform Cloud Engineering and will help in a quick revision for your Google Associate Cloud Engineer Study. 

 

  1. The command – line command to create a Cloud Storage bucket is gsutil mb, where gsutil is command line for accessing and manipulating Cloud Storage from command line. mb is the specific command for creating, or making, a bucket.
  2. Adding Virtual Machines to an instance group can be triggered in an autoscaling policy by all of the following :
    • CPU Utilisation 
    • Stackdriver metrics
    • Load balancing serving capacity
  3. Datastore options in GCP for transactions and the ability to perform relational database operations using fully complinat SQL data store – Spanner & Cloud SQL
  4. Instance templates are used to create a group of identical VMs. The instance templates include the following configuration parameters or attributes of a
    1. VM
    2. Machine Type
    3. Boot Disk image
    4. Container Image
    5. Zone
    6. Labels.
  5. The most efficient way to implement an object management policy via administrators that requires ojects stored in Cloud Storage to be migrated from regional storage to enearline storage 90 days after the object is created is via lifecycle management configuration policy specifying an age of 90 days and SetStorageClass as nearline.
  6. Command to synchronize the contents of the two buckets gsutil rsync.
  7. All of the following are components of firewall rules a.) direction of traffic, b.) priority c.) action on match d.) enforcement status e.) target f.) source g.) protocol
  8. VPC’s are global resources and subnets are regional resources.
  9. Web application – deployment but does not want to manage managed servers or clusters. A good option is a PaaS – App Engine.
  10. Data warehouse needing SQL query capabilities over petabytes of data but with no manage servers or clusters, such requirements can be met by Big Query.
  11. Internet of Things space, will stream large volumes of data into GCp. The data needs to be filtered, transformed and analysed before being stored in GCP Datastore. — Cloud Dataflow.
  12. Cloud Dataflow allows for stream and batch processing of data and is well suited for ETL work.
  13. Dataproc is a managed Hadoop and Spark service thiat is used for big data analysics
  14. Buckets, directories and subdirectories are used to organise storage
  15. gcloud is the command line tool for IAM and list-grantable-roles will list roles granted to a resource, gcloud iam list-grantable-roles <resource>
  16. Cloud Endpoints is an API Service
  17. Cloud Interconnect is a network service.
  18. Compute Engine Virtual Machine is a zonal resource.
  19. Within the Zonal & Regional scope, GCP geographic scopes are network latencies generally less than 1 millisecond.
  20. To create a custom role, a user must possess iam.roles.create
  21. Project is the base-level organizing entity for creating and using GCP resources and services.
  22. Organisations, folders and projects are the components used to manage an organizational hierarchy.
  23. gcloud compute regions describe, gets a list of all CPU types available in a particular zone.
  24. Cloud Function responds to events in Cloud Storage, making them a good choice for taking actiona after a file is loaded
  25. Billing is setup at the Project level in the GCP resource hierarchy.
  26. Cloud Dataproc is the managed Spark Service
  27. Cloud Dataflow is for stream and processing
  28. Rate Quotas resets at regular intervals.
  29. There are two types of quotas in billing, Rate Quotas and Allocation Quotas.
  30. In Kubernetes Engine, a node pool is a subset of node instances within a cluster that all have the same configuration.
  31. Code for Cloud Functions can be written in Node.js and Python
  32. Preemptible virtual machines may be shutdown at any time but will always be shut down after running for 24 hours.
  33. After deciding to use Cloud Key Management Services and before you can start to create cryptographic keys you must enable KMS Api (Google Cloud Key Management Service) and setup billing.
  34. GCP Service for storing and managing Docker containers is Container Registry.
  35. You must verify the project selected is the one you want to work with, once you have opened the GCP console at console.google.com before performing task on VM’s. All operations you perform apply to resoures in the identified project.
  36. One time task you will need to complete before using the console is setting up the billing. You will be able to create the project only if the billing is enabled.
  37. A name for VM, machine type, a region and a zone are minimal set of info you will need while creating a VM.
  38. Different zones may have different machine types available.
  39. Billing of different departments for the cost of VM’s used for their applications is possible with labels and descriptions.
  40. Google Cloud Interconnect – Dedicated is used to provide a dedicated connection between customer’s data center and a Google data center
  41. Purpose of instance groups in a Kubernetes cluster is to create sets of VM’s that can be managed as a unit.
  42. A Kubernetes cluster has a single cluster master and one or more nodes to execute workloads.
  43. A pod is a single instance of a running process in a cluster
  44. To ensure applications calling Kubernetes services
  45. ReplicaSets are controllers that are responsible for maintaining the correct number of pods.
  46. Deployments are versions of application code running on a cluster. 
  47. To maintain availability even if there is a major network outage in a data center, Multizone/multiregion clusters are available in Kubernetes Engine and are used to provide resiliency to an application
  48. Starting with an existing template, filling in parameters, and generating the gcloud command is the most reliable way to deploy a Kubernetes cluster with GPUs.
  49. gcloud beta container clusters create ch07-cluster-1 –num-nodes=4 will create a cluster named ch07-cluster-1 with four nodes.
  50.  Application name, container image, and initial command can all be specified when using create deployment from cloud console when creating a deployment from cloud console. Time to Live (TTL) is not specified and not an attribute of deployments.
  51. Deployment configuration files created in Cloud Console use YAML format.
  52. When working on a Kubernetes Engine a cloud engineer may need to configure, Nodes, Pods, services, clusters and container images.
  53. After observing performance degradation, inorder to see details of a specific cluster, after opening Cloud Console, Click the cluster name to see details of a specific cluster.
  54. You can find the number of vCPUS on the cluster listing in the Total Cores column or on the Details Page in the Node Pool section in the size parameter.
  55. High level characteristics of a cluster — gcloud container clusters list
  56. gcloud container clusters get-credentials is the correct command to configure kubectl to use GCP credentials for the cluster.
  57. Clicking Edit button allows you to change, add, or remove labels from the Kubernetes cluser.
  58. When resizing, the gcloud container clusteres resize command requires the name of the cluster,size and the node pool to modify.
  59. Pods are used to implement replicas of a deployment, and it is best practice to modify deployments which are configured with a specification of the number of replicas that should always run. 
  60. In the Kubernetes Engine Navigation menu, you would select Workloads inorder to see a list of deployments.
  61. 4 actions available for deployments is Autoscale, Expose, Rolling Update and Scale.
  62. Command to list deployments is kubectl get deployments
  63. You can specify container image, cluster name and application name along with the labels, initial command and namespace. 
  64. The Deployment Details page includes services.
  65. kubetcl run command is used to start a deployment. It takes name for the deployment, image & port.
  66. Command for service not functioning as expected and needs to be removed from the cluster — kubectl delete service m1-classfied
  67. Container Registry is the service for managing images that can be used in other services like Kubernetes Engine and Compute Engine.
  68. gcloud container images list  — is to list container images in the command line. 
  69. gcloud container images describe — to get  a detailed description of each containers
  70. kubectl expose deployment — makes a service accessible.
  71. Autoscaling is the most cost-effective and least burdensome way to respond to changes in demand for a service.
  72. Version aspect of App Engine components would you use to minimize disruptions during updates to the service. Versions support migration. An app can have multiple versions, and by deploying with the --migrate parameter, you can migrate traffic to the new version.
  73. Autoscaling enables setting a maximum and minimum number of instances, the best way to ensure that you have enough instances to meet demand without spending more than you have to.
  74. Applications have one or more services. Services have one or more versions. Versions are executed on one or more instances when the application is running. Hence in the hierarchy of App Engine components, instance is the lowest-level component.
  75. gcloud app deploy, is used to deploy App engine app from command line.
  76. The app.yaml file is used to configure an App Engine application, so if python associated with the app were to be upgraded, it would be upgraded via the file.
  77.  A project can support only one App Engine app
  78. The best way to get the code out as soon as possible without exposing it to customers, would be to deploy with gcloud app deploy --no-promote
  79. App Engine applications are accessible from URLs that consist of the project name followed by appspot.com.
  80. Related to App Engine, max_concurrent_requests lets you specify the maximum number of concurrent requests before another instance is started. target_throughput_utilization functions similarly but uses a 0.05 to 0.95 scale to specify maximum throughput utilization. max_instances specifies the maximum number of instances but not the criteria for adding instances. max_pending_latency is based on the time a request waits, not the number of requests.
  81. App Engine Basic scaling only allows for idle time and maximum instances
  82. App Engine, The runtime parameter specifies the language environment to execute in. The script to execute is specified by the script parameter. The URL to access the application is based on the project name and the domain appspot.com. 
  83. Two kinds of instances present in App Engine Standard, Resident instances are used with manual scaling while dynamic instances are used with autoscaling and basic scaling.
  84. For Apps running in App Engine, using dynamic instances by specifying autoscaling or basic scaling will automatically adjust the number of instances in use based on load.
  85. For Apps running in App Engine, gcloud app services set-traffic, can allocate some users to a new version without exposing all users to it
  86. For Apps running in App Engine, –split-traffic parameter to gcloud app services set-traffic is used to specify the method to use when splitting traffic
  87. For Apps running in App Engine, –splits parameter to gcloud app services set-traffic is used to specify the percentage of traffic that should go to each instance
  88. For Apps running in App Engine,--migrate is the parameter for specifying that traffic should be moved or migrated to the newer instance,
  89. The cookie used for cookie based splitting in App Engine is called GOOGAPPUID
  90. From the App Engine console you can view the list of services and versions as well as information about the utilization of each instance.
  91. All three methods listed, IP address, HTTP cookie, and random splitting, are allowed methods for splitting traffic for App Engine Traffic
  92. New app will require several backend services, three business logic services and access to relational databases. Each service will provide a single function and it will require services to complete a business task.Service execution time is dependent on the size of input and is expected to take up to 30 minutes in some cases. App Engine is designed to support multiple tightly coupled services comprising an application.
  93. Cloud Functions, which is designed to support single-purpose functions that operate independently and in response to isolated events in the Google Cloud and complete within a specified period of time.
  94. In Cloud Functions,a timeout period that is too low would explain why the smaller files are processed in time but the largest are not.
  95. In Cloud Functions, An event, is an action that occurs in GCP, such as a file being written to Cloud Storage or a message being added to a Cloud Pub/Sub topic.
  96. In Cloud Functions, a trigger is a declaration that a certain function should execute when an event occurs. 
  97. GCP products listed do generate events that can have triggers associated with them, Cloud Storage, Cloud Pub/Sub/ Firebase, HTTP.
  98. Python, Node.js 6, Node.js 8 are supported in Cloud Functions.
  99. In Cloud Functions, an HTTP trigger can be invoked by making a request using DELETE, POST and GET.
  100. With Cloud Storage working with Cloud Functions, upload or finalise, delete, metadata update and archive are the 4 events supported.
    1. google.storage.object.finalize
    2. google.storage.object.delete
    3. google.storage.object.archive
    4. google.storage.object.metadataUpdate
  101. Following feature cannot be specified in a parameter and must be implemented in Function code, i.e. File type to apply the function to.
  102. Cloud Functions can have between 128MB and 2GB of memory allocated.
  103. By default Cloud Functions can run for up to 1 minute before timing out, you can, however, set the timeout parameter for a cloud function for periods of up to 9 minutes before timing out.
  104. Python Cloud Functions is currently in beta. The standard set of gcloud commands does not include commands for alpha or beta release features by default. You will need to explicitly install beta features using the gcloud components install beta command
  105. google.storage.object.finalize, which occurs after a file is uploaded.
  106. If you are defining a cloud function to write a record to a database when a file in Cloud Storage is archived, you need runtime, trigger-resource, trigger-event only.
  107. If you’d like to stop using a cloud function and delete it from your project, gcloud functions delete.

  108. As part of python code for a cloud function to work with Cloud Pub/Sub. Deccode function will be required in Pub/Sub cloud function for messages in Pub/Sub topics are encoded to allow binary data to be used in places where text data is expected. Messages need to be decoded to access the data in the message.

  109. Bigtable is a wide-column database that can ingest large volumes of data consistently.
  110. Once a bucket is created as either regional or multi-regional, it cannot be changed to the other.
  111. The goal is to reduce cost, so you would want to use the least costly storage option. Coldline has the lowest per-gigabyte charge at $0.07/GB/month.
  112. Memorystore is a managed Redis cache. The cache can be used to store the results of queries. Follow-on queries that reference the data stored in the cache can read it from the cache, which is much faster than reading from persistent disks. SSDs have significantly lower latency than hard disk drives and should be used for performance-sensitive applications like databases. 
  113. While versioning on a bucket, the latest version of the object is called the live version.
  114. Lifecycle configurations can change storage class from regional to nearline or coldline. Once a bucket is created as regional or multiregional, it cannot be changed to the other.
  115. Transactions and support for tabular data are important, Cloud SQL and Spanner are relational databases and are well suited for transaction-processing applications.
  116. Sample command for deployment of python cloud function call pub_sub_function_test,
    • gcloud functions deploy pub_sub_function_test –runtime python37 –trigger-topic gcp-ace-exam-test-topic
  117. There is only one type of event that is triggered in Cloud Pub/Sub, and that is when a message is published.
  118. Both MySQL and PostgreSQL are Cloud SQL options
  119. nam3 is a single super region
  120. us-central1 is a region
  121. us-west1-a is a zone
  122. Multiregional and multi-super-regional location of nam-eur-aisa1 is one the most expensive Cloud Spanner Configurations.
  123. BigQuery, Datastore, and Firebase are all fully managed services that do not require you to specify configuration information for VMs.
  124. Document data model is used by Datastore.
  125. BigQuery is a managed service designed for data warehouses and analytics. It uses standard SQL for querying and can support tens of petabytes of data.
  126. Bigtable can support tens of petabytes of data, but it does not use SQL as a query language. 
  127. Firestore is a document database that has mobile supporting features, like data synchronization.
  128. Consistency, cost, read / write patterns, transaction support and latency are features of storage which should be considered while securing additional storage.
  129. Once a bucket has its storage class set to coldline, it cannot be changed to another storage class.
  130. To use BigQuery to store data, you must have a data set to store it.
  131. With a second-generation instance in Cloud SQL, you can configure the MySQL version, connectivity, machine type, automatic backups, failover replicas, database flags, maintenance windows, and labels.
  132. Access charges are used with nearline and coldline storage
  133. Memorystore can be configured to use between 1GB and 300GB of memory.
  134. Your company has a web application that allows job seekers to upload résumé files. Some files are in Microsoft Word, some are PDFs, and others are text files. You would like to store all résumés as PDFs. And the solution for this one is, implement a Cloud Function on Cloud Storage to execute on a finalize event. The function checks the file type, and if it is not PDF, the function calls a PDF converter function and writes the PDF version to the bucket that has the original.
  135. Options for uploading code to a cloud function are as follows 
    1. Inline editor
    2. Zip upload
    3. Cloud source repository
  136. The HTTP trigger allows for the use of POST, GET, and PUT calls to invoke a cloud function.
  137. Cloud SQL is a fully managed relational database service, but database administrators still have to perform some tasks. Creating Databases is one of them.
  138. Cloud SQL is controlled using the gcloud command; the sequence of terms in gcloud commands is gcloud followed by the service, in this case SQL; followed by a resource, in this case backups, and a command or verb, in this case create.  Command is used to create a backup of a Cloud SQL database. “gcloud sql backups create”
  139. Command will run automatic backup on an instance called ace-exam-mysql. The base command is gcloud sql instances patch, which is followed by the instance name and a start time passed to the –-backup-start-time parameter. gcloud sql instances patch ace-exam-mysql –backup-start-time 03:00
  140. GQL – Go Query Language is used for Datastore
  141. Export Data from Datastore uses the following command, gcloud datastore export -namespaces='[NAMESPACE]’ gs://[BUCKET_NAME]
  142. BigQuery analyzes the query and BigQuery displays an estimate of the amount of data scanned. This is important because BigQuery charges for data scanned in queries.
  143. Command to get an estimate of the volume of data scanned by BigQuery from the command line, the correct bq command structure, which includes location and the ––dry_run option. This option calculates an estimate without actually running the query. bq ––location=[LOCATION] query –use_legacy_sql=false ––dry_run [SQL_QUERY]
  144. You are using Cloud Console and want to check on some jobs running in BigQuery. You navigate to the BigQuery part of the console. Job History is the menu item would you click to view jobs.
  145. Estimate the cost of running BigQuery query, BigQuery provides an estimate of the amount of data scanned, and the Pricing Calculator gives a cost estimate for scanning that volume of data.
  146. You have just created a Cloud Spanner instance. You have been tasked with creating a way to store data about a product catalog, the next step is to create a database within the instance. Once a database is created, tables can be created, and data can be loaded into tables. 
  147. Your software team is developing a distributed application and wants to send messages from one application to another. Once the consuming application reads a message, it should be deleted. You want your system to be robust to failure, so messages should be available for at least three days before they are discarded. It involves sending messages to the topic, and the subscription model is a good fit. Pub/Sub has a retention period to support the three-day retention period. 
  148. Pub/Sub works with topics, which receive and hold messages, and subscriptions, which make messages available to consuming applications.
  149. Command line tools for Big Table Environment can be secured using gcloud components install cbt to install the Bigtable command-line tool.
  150. cbt createtable iot-ingest-data, to create a table iot-ingest-data in Big Table
  151. Cloud Dataproc is a managed service for Spark and Hadoop. Cassandra is a big data distributed database but is not offered as a managed service by Google
  152. gcloud dataproc clusters create spark-nightly-analysis ––zone us-west2-a, command to create data proc cluster
  153. Command to rename an object stored in a bucket, gsutil mv gs://[BUCKET_NAME]/[OLD_OBJECT_NAME] gs://[BUCKET_NAME]/ [NEW_OBJECT_NAME
  154. Dataproc with Spark and its machine learning library are ideal for the use of selling more products.
  155. gsutil mb is used to create buckets in Cloud Storage
  156. gsutil cp is the command to copy files from your local device to a bucket in Cloud Storage assuming your have Cloud SDK installed.
  157. Using the cloud console you can upload files and folders, if you are migrating a large number of files from a local storage system to Cloud Storage
  158. When exporting a database from Cloud SQL, the export file format options are CSV and SQL
  159. SQL format, exports a database as a series of SQL data definition commands. These commands can be executed in another relational database without having to first create a schema
  160. gcloud sql export sql ace-exam-mysql1 gs://ace-exam-buckete1/ace-exam-mysql-export.sql \ ––database=mysql, this command will export a MySQL database called ace-exam-mysql1 to a file called ace-exam-mysql-export.sql in a bucket named ace-exam-buckete1
  161. Command is required to back up data from your Datastore database to an object storage system. Your data is stored in the default namespace, gcloud datastore export ––namespaces=“(default)” gs://ace-exam-bucket1
  162. Datastore export command process creates a metadata file with information about the data exported and a folder that has the data itself
  163. XML is not an option in BigQuery export process, CSV, AVRO,JSON are valid options.
  164. CSV, AVRO,Parquet are valid BigQuery options.
  165. bq load ––autodetect ––source_format=[FORMAT] [DATASET].[TABLE] [PATH_TO_SOURCE], BigQuery to analyse the data and make data available for analysis in BigQuery.
  166. You have set up a Cloud Spanner process to export data to Cloud Storage. You notice that each time the process runs you incur charges for another GCP service, which you think is related to the export process.Dataflow is a pipeline service for processing streaming and batch data that implements workflows used by Cloud Spanner.
  167. Exporting from Dataproc exports data about the cluster configuration. Dataproc supports Apache Spark, which has libraries for machine learning. 
  168. Correct command to create a Pub/Sub topic, gcloud pubsub topics create
  169. gcloud pubsub subscriptions create ––topic=ace-exam-topic1 ace-exam-sub1, command will create a subscription on the topic ace-exam-topic1
  170. Direct Advantages of using a message queue in distributed systems, It decouples services, so if one lags, it does not cause other services to lag.
  171. gcloud components install beta, to install beta glcoud commands
  172. BigQuery parameter to automatically detect the schedma of a file import, –autodetect.
  173. Avro supports Deflate and Snappy compression when exporting from BigQuery. CSV supports Gzip and no compression. 
  174. As a developer on a project using Bigtable for an IoT application, you will need to export data from Bigtable to make some data available for analysis with another tool. A Java program designed for importing and exporting data from Bigtable.
  175. The A record is used to map a domain name to an IPv4 address. The AAAA record is used for IPv6 addresses.
  176. DNSSEC is a secure protocol designed to prevent spoofing and cache poisoning.
  177. The TTL parameters in a DNS record specify, the time a record can be in a cache before the data should be queried again.
  178. Command to create a DNS zone in command line,  gcloud beta dns managed-zones create.
  179. –visibility=private is the parameter that can be set to private for DNS
  180. Virtual private clouds are global, so option D is correct. By default, they have subnets in all regions. Resources in any region can be accessed through the VPC.
  181. IP ranges are assigned to subnets, when CIDR ranges are defined, they are defined as per the number of subnets.
  182.  Dynamic routing is the parameter that specifies whether routes are learned regionally or globally. 
  183. gcloud compute networks create – command to create a VPC
  184. The Flow Log option of the create vpc command determines whether logs are sent to Stackdriver.
  185. Shared VPC’s can be created at the organisation or folder level of the resource hierarchy.
  186. While creating a VM that should exist in a custom subnet, one needs specify the subnet in the Networking tab of the Management, Security, Disks, Networking, Sole Tenancy section of the form.
  187. VPC peering is used for interproject communications.
  188. Target, this part of the firewall rule can reference the network tag to determine the set of instances afftected by the rule. 
  189. Direction specifies whether the rule is applied to incoming or outgoing traffic.
  190. The 0.0.0.0/0 matches all IP addresses.
  191. gcloud compute firewall-rules create, the product you are working with is compute and the resource you are creating is a firewall rule.
  192. Using gcloud to create a firewall rule, –network parameter is used to specify the subnet it should apply to.
  193. gcloud compute firewall-rules create fwr1 –allow=udp:20000-30000 –direction=ingress.The service endpoints will accept any UDP traffic and each endpoint will use a port in the range of 20000–30000.
  194. You want it to apply only if there is not another rule that would deny that traffic. 65535 is the appropriate priority because  it is the largest number allowed in the range of values for priorities. The larger the number, the lower the priority. Having the lowest priority will ensure that other rules that match will apply.
  195. The VPC create option is available in the Hybrid Connectivity section
  196. If you want to configure the GCP end of the VPN, the following section of the Create VPN form would be used, the Google Compute Engine VPN is where you specify information about the Google Cloud end of the VPN connection.
  197. Global dynamic routing is used to learn all routes on a network and if you want the router on a tunnel you are creating to learn routes from all GCP regions on the network.The autonomous system number (ASN) is a number used to identify a cloud router on a network
  198. When using gcloud to create a VPN, you need to create forwarding rules, tunnels, and gateways, so all the gcloud commands listed would be used. gcloud compute forwarding-rule, gcloud compute target-vpn-gateways, and gcloud compute vpn-tunnels
  199. When you create a cloud route, you need to assign an ASN for the BGP protocol.
  200. Incase a remote component in your network has failed, which results in a transient network error, when you submit a gsutil command, it fails because of a transient error, by default the command will retry using a truncated binary exponential back-off strategy. This strategy is as follows, gsutil will retry using a truncated binary exponential backoff strategy:
    • Wait a random period between [0..1] seconds and retry;
    • If that fails, wait a random period between [0..2] seconds and retry;
    • If that fails, wait a random period between [0..4] seconds and retry;
    • And so on, up to a configurable maximum number of retries (default = 23),with each retry period bounded by a configurable maximum period of time (default = 60 seconds).

Thus, by default, gsutil will retry 23 times over 1+2+4+8+16+32+60… seconds for about 10 minutes. You can adjust the number of retries and maximum delay of any individual retry by editing the num_retries and max_retry_delay configuration variables in the “[Boto]” section of the .boto config file. Most users shouldn’t need to change these values.For data transfers (the gsutil cp and rsync commands), gsutil provides additional retry functionality, in the form of resumable transfers. Essentially, a transfer that was interrupted because of a transient error can be restarted without starting over from scratch. For more details about this, see the “RESUMABLE TRANSFERS” section of gsutil help.

 

 

 

References

 

Tagged : / / / / / / / / / /

Interacting with Google Cloud Platform

by Davin LewisCloud Computing / Google Cloud PlatformLeave a Comment on Interacting with Google Cloud Platform

There are four ways you can interact with Google Cloud Platform, each of them are listed below :

  1. GCP Console
  2. SDK and Cloud Shell
  3. Mobile App
  4. APIs. 

 1. GCP Console

  • The GCP Console is a web-based administrative interface. If you build an application in GCP, you will interact with it and not exposed to the end users of your app.
  • It lets you view and manage all your projects and all the resources they use. 
  • It also lets you enable, disable and explore the APIs of GCP services. 
  • It gives you access to Cloud Shell. 

2. A. Cloud Shell

That’s a command-line interface to GCP that’s easily accessed from your browser. 

From Cloud Shell, you can use the tools provided by the Google Cloud Software Development kit SDK, without having to first install them somewhere.

2. B. Google Cloud SDK – Software Development Kit 

The Google Cloud SDK is a set of tools that you can use to manage your resources and your applications on GCP. 

These include the gcloud tool, which provides the main command line interface for Google Cloud Platform products and services. 

 GSUTIL

gsutil is a Python application that lets you access Cloud Storage from the command line. You can use gsutil to do a wide range of bucket and object management tasks, including:

  • Creating and deleting buckets.
  • Uploading, downloading, and deleting objects.
  • Listing buckets and objects.
  • Moving, copying, and renaming objects.
  • Editing object and bucket ACLs.

BQ

bq is a python-based, command-line tool for BigQuery.

A virtual machine with all these commands is already installed. You can also install the SDK on your own computers – your laptop, your on-premise servers of virtual machines and other clouds. The SDK is also available as a docker image, which is a really easy and clean way to work with it.  

3. API’s

The services that make up GCP offer application programming interfaces so that the code you write can control them. These APIs are what’s called RESTful. In other words they follow the representational state transfer paradigm. 

Basically, it means that your code can use Google services in much the same way that web browsers talk to web servers. 

The APIs name resources and GCP with URLs. Your code can pass information to the APIs using JSON, which is a very popular way of passing textual information over the web. 

And there’s an open system for user log in and access control. The GCP Console lets you turn on and off APIs. 

Many APIs are off by default, and many are associated with quotas and limits. These restrictions help protect you from using resources inadvertently. 

You can enable only those APIs you need and you can request increases in quotas when you need more resources. 

The GCP Console includes a tool called the APIs Explorer that helps you learn about the APIs interactively. It lets you see what APIs are available and in what versions. 

These APIs expect parameters and documentation on them is built in. You can try the APIs interactively even with user authentication. Suppose you have explored an API and you’re ready to build an application that uses it. Google provides client libraries that take a lot of the drudgery out of the task of calling GCP from your code. 

 Libraries

There are two kinds of libraries. 

  • Cloud Client Library
  • Google API Client Library

The Cloud Client Libraries are Google clouds latest and recommended libraries for its APIs. They adopt the native styles and idioms of each language.  Google Cloud Client Libraries are our latest and recommended client libraries for calling Google Cloud APIs. They provide an optimized developer experience by using each supported language’s natural conventions and styles. They also reduce the boilerplate code you have to write because they’re designed to enable you to work with service metaphors in mind, rather than implementation details or service API concepts. 

On the other hand, sometimes a Cloud Client Library doesn’t support the newest services and features. In that case, you can use the Google API Client Library for your desired languages. These libraries are designed for generality and completeness. 

Following are the Cloud Client Libraries available,

  1. Java
  2. Node.js
  3. Python
  4. C#
  5. Go
  6. Ruby
  7. PHP

 Mobile App

There’s a mobile App for Android and iOS that lets you examine and manage the resources you’re using in GCP. 

It lets you build dashboards so that you can get the information you need at a glance.

Tagged : / / / / / / / / / / /

Cloud IAM Roles

by Davin LewisCloud Computing / Google Cloud Platform2 Comments on Cloud IAM Roles

Cloud Identity & Access Management lets you grant granular access to specific Google Cloud resources and helps prevent access to other resources. Cloud IAM lets you adopt the security principle of least privilege, where you grant only necessary permissions to access specific resources.

Primitive Roles

Primitive Roles are broad and impact all resources in the project. They are roles which existed prior to existence of Cloud IAM.

These roles are concentric, owner includes the permissions of editor and editor includes the permission of viewer.

  1. Viewer – Permissions for read-only actions that do not affect state, such as viewing (but not modifying) existing resources or data.
  2. Editor – All viewer permissions, plus permissions for actions that modify state, such as changing existing resources.
  3. Owner – All editor permissions, plus Manage roles and permissions for a project and all resources within the project. Set up billing for a project.

4. Billing Administrator Role

someone to be able to control the billing for a project without the right to change the resources in the project. 

Predefined Roles

In addition to the primitive roles, Cloud IAM provides additional predefined roles that give granular access to specific Google Cloud Platform resources and prevent unwanted access to other resources.

There is a long list of these roles listed here.

Custom Roles

Custom roles can only be used at the project or organization levels. They can’t be used at the folder level. 

To create a custom role, a caller must possess iam.roles.create permission. By default, the owner of a project or an organization has this permission and can create and manage custom roles.

Users who are not owners, including organization admins, must be assigned either the Organization Role Administrator role, or the IAM Role Administrator role.

Tagged : / / / /

Policies & Cloud IAM

by Davin LewisCloud Computing / Google Cloud Platform4 Comments on Policies & Cloud IAM

Organization policies are needed to enforce what resources are available in a Google Cloud trust boundary (folder, project, or other organizational level).

Cloud Identity and Access Management (IAM) enables you to manage access control by defining who (identity) has what access (role) to which resource.

Policies

Top to bottom : Resources inherit the policies of their parent resource. 

Google Cloud Platform Policies are set at the highest level, like the organisation node are cascaded down to the project.

Inheritance is transitive, all the resources in a project inherit the policy.

However policies implemented at a higher level in this hierarchy can’t affect the access that’s granted exceptionally at a lower level. 

For example,

Policy A – Enforced at the Organisation Level – Users can view cloud storage buckets.

Policy B – Enabled at the Project Level – User can modify cloud storage buckets.

Since Project Level Policy B is granting a more generous access at a lower level, it shall take effect and override Policy A.

Cloud IAM

An IAM policy has the following :

  1. “who” part, 
  2. a “can do what” part, and
  3. an “on which resource” part. 

Who

The “who” part of an IAM policy can be defined either by a

a. Google account

b. Google group

c. Service account

d. G Suite

e. Cloud Identity domain. 

Can Do What – Role

The “can do what” part is defined by an IAM role which is a collection of permissions.

To do any meaningful operations, you need more than one permission.For example, to manage instances in a project, you need to create, delete, start, stop, and change an instance. 

So the permissions are grouped together into a role that makes them easier to manage. 

There are three kinds of roles in Cloud IAM:

  1. Primitive roles: Roles historically available in the Google Cloud Console. These roles are Owner, Editor, and Viewer. Avoid using these roles if possible, because they include a wide range of permissions across all Google Cloud services.
  2. Predefined roles: Roles that give finer-grained access control than the primitive roles. For example, the predefined role Pub/Sub Publisher (roles/pubsub.publisher) provides access to only publish messages to a Pub/Sub topic.
  3. Custom roles: Roles that you create to tailor permissions to the needs of your organization when predefined roles don’t meet your needs.

Reference : Concepts Related to Access Management

Tagged : / / / / /

Google Hierarchy Objects : Projects, Folders, Organisation Node

by Davin LewisCloud Computing / Google Cloud Platform1 Comment on Google Hierarchy Objects : Projects, Folders, Organisation Node

Projects, Folders, Organisation Node are the binding blocks of the Google Cloud Hierarchy.

Policies

Policies are inherited downwards in the hierarchy. All Google Cloud platform resources belong to a project.

Projects

Projects are the basis for enabling and using GCP services like managing APIs, enabling billing and adding and removing collaborators and enabling other Google services. 

Each project is a separate compartment and each resource belongs to exactly one. 

Projects can have different owners and users – they’re built separately and they’re managed separately. 

Project ID / Name / Number

Project has three identifying attributes

Each GCP project has a name and a project ID that you assign. 

Project ID : Globally unique : Chosen by you : Immutable

The project ID is a permanent, unchangeable identifier and it has to be unique across GCP. You use project IDs in several contexts to tell GCP which project you want to work with.  In general, project IDs are made to be human readable strings and you’ll use them frequently to refer to projects.

It is a a unique identifier for your project, composed of the project name and a randomly assigned number.

Project Name : Need not be unique : Chosen by you : Mutable

On the other hand, project names are for your convenience and you can assign them. 

Project Number : Globally Unique : Assigned by GCP : Immutable

GCP also assigns each of your projects a unique project number and you’ll see a display to you in various contexts.  It is a number that’s automatically generated by the server and assigned to your project.

Folders

You can organize projects into folders, although you don’t have to. 
For example, you can use folders to represent different departments,
teams, applications or environments in your organization. 

Folders let teams have the ability to delegate administrative rights, 
so they can work independently. 

The resources in a folder inherit IAM policies from the folder. 
So, if project three and four are administered by the same team by design, 
you can put IAM policies into folder B instead. 

Doing it the other way, putting duplicate copies of those policies on 
project three and project four would be tedious and error prone. 

To use folders, you need an organization node at the top of the hierarchy. 

Organisation Node

You probably want to organize all the projects in your company into a single structure. 
Most companies want the ability to have centralized visibility on how resources are being used and to apply policy centrally. That’s what the organization node is for. It’s the top of the hierarchy. 

In part the answer depends on whether your company is also a G Suite customer. 

If you have a G Suite domain, GCP projects will automatically belong to your organization node. Otherwise, you can use Google Cloud Identity to create one.

When you get a new organization node, it lets anyone in the domain create 
projects and billing accounts just as they could before. That’s to avoid surprises and disruption. 

But it’d be a great first step with a new organization node to decide who on your team should really be able to do those things. 

Once you have an organization node, you can create folders underneath it and put it in projects. 

Tagged : / / /

Google Cloud Platform Hierarchy

by Davin LewisCloud Computing / Google Cloud Platform1 Comment on Google Cloud Platform Hierarchy

The Google Cloud Platform Hierarchy is a internal platform resource hierarchy is something you should go out bottom up.

Details

Whatever the resources utilized they are VM’s, cloud storage buckets, tables and big query, they are all organised into projects.

  • Projects rollup into folders.
  • Folders can also multiple other folders.

Together everything can rollup into an organisation node.

GCP Resources -> Projects -> Folders -> Organisation Node

Graphical Representation

cloud-hierarchy-1

Policies can be enforced at any level of this tree.

Tagged : / /