I’m steadily working through the GCP Quests. This week I’ve been working through the Baseline Infrastructure. It walks through creating Buckets, Cloud IAM, using StackDrive and more. My notes are more refined this week and they’re growing more rapidly than last week’s so here’s the first part of the Baseline Infrastructure.
Baseline Infrastructure Quest
Cloud Storage Lab
Storage allows storage (duh) and retrieval of data. Can be used to serve website content, archives, and recovery. Can also distribute large data objects to users with direct download.
These notes will help you create storage buckets, upload objects, create folders and subfolders and make objects publicly available.
We’re not using the cloud shell in this series so notes will all be in the console
Create a Bucket
Open the Google Console
Navigation menu > Storage > Browser > Create Bucket
Name: enter a unique name
There are rules to naming buckets. See the fine print for details.
Short form is, start with alpha; alphanumeric, -, _, .
Storage class: (pick your choice) Multi-regional
Location: (your choice) United States
Create
And now your bucket is created.
Upload an Object into a bucket
Grab a file. Any file. For practice a .jpg or .txt file is best. The image should be saved to your local drive.
If you’ve just created a bucket then you should be in the Bucket’s details screen. If not, then find and open your bucket to view its details.
Drag and drop the file into the bucket’s console or browse your disk for the desired file.
The file will be added to the Bucket’s list.
To Delete the file click the box next to the filename and then click delete. Confirm your desire to delete.
OK, let’s upload the file again.
To rename the file click the hotdog icon (That’s three vertical dots on the far right of the row). Select Rename
(Note that the hotdog button may be hiding so you may need to scroll to the right or use a full screen).
Change the file name and then click RENAME.
Now your bucket will reflect the new name of the file.
Share an Object Publicly
Files in a bucket are private by default. We must edit permissions to make them accessible via URL.
Click the hotdog button and select edit permissions
Click + Add item
We’re going to add a permission for all users:
Under ENTITY select User
Under NAME select allUsers
Under ACCESS select Reader
Click Save
Now that the file is shared publicly a link icon will appear in the “public access” column. Click the file to see it in a new tab.
Create folders in a Bucket
To create a folder click the “Create Folder” link toward the top of the page.
Name the folder and click Create
Folders have the folder icon in the bucket to differentiate them from files.
Create a subfolder by entering the first folder and then create a new folder.
Files can be placed in folders and subfolders.
Be sure the file is publicly accessible by clicking the hotdog button and selecting Edit permissions
Select + Add item
Enter the same permissions as we did previously.
Under ENTITY select User
Under NAME select allUsers
Under ACCESS select Reader
Delete a folder
Open your bucket.
Click the box to the left of the folder you desire to delete.
Click delete again to confirm.
It’s gone!
Cloud Storage: This time with the Shell
Consoles are nice. GUIs have made computers accessible to the general public, which I think we have found is great and terrible all at the same time. As great as GUIs are, the shell is where its at in terms of efficiency and power. If you want to be a power user then learning shell is essential. This section of the notes does exactly what we just accomplished, but using the cloud shell.
You can double check your progress at any time in the console by navigating to your bucket and refreshing the browser as you work. (Navigation menu > Storage)
OK, let’s rock through this.
Open the cloud shell by clicking the Open Shell button
If you’d like to list your active account name use the command
gcloud auth list
List project ID with
gcloud config list project
Let’s create a bucket. Use the gsutil mb command and a unique name to create a bucket.
gsutil mb gs://BUCKET-NAME/
Same bucket naming rules as before apply. In short, alphanumeric, _, -, . Are legal but name must start with alpha and end with alphanumeric
Successful output looks like:
Creating gs://BUCKET-NAME
A 409 ServiceException means you’ve already used that bucket name.
Upload an object into the bucket
In this case, we’re going to get a portrait of Ada Lovelace from her Wikipedia page.
wget --output-document ada.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/a/a4/Ada_Lovelace_portrait.jpg/800px-Ada_Lovelace_portrait.jpg
Use gsutil cp to upload the image from your disk to the bucket.
gsutil cp ada.jpg gs://BUCKET-NAME/
To remove a file use the rm command.
rm ada.jpg
We can download files stored in our buckets to our Cloud Shell with the gsutil cp command.
gsutil cp -r gs://BUCKET-NAME/ada.jpg
Successful operation returns a “Copying gs” output.
Copy an object to a folder in the bucket
Use gsutil cp to create a new folder and copy the image.
gsutil cp gs://BUCKET-NAME/ada.jpg gs://BUCKET-NAME/image-folder/
(Image-folder/ is the name of my new folder.)
If you’re successful then you’ll receive a “Copying gs” output.
List contents of a bucket or folder
Use gsutil ls to list contents of a bucket:
gsutil ls gs://BUCKET-NAME
List details for an object
Use the -l flag to receive a detailed list of bucket contents
gsutil ls -l gs://BUCKET-NAME/ada.jpg
Make objects publicly accessible
Use the gsutil acl ch command to grand all users read permission of an object stored in the bucket.
gsutil acl ch -u AllUsers:R gs://BUCKET-NAME/ada.jpg
An “updated” output returns when successful.
The image or file is now public and available.
Remove Public Access
To remove public access use the same command above with the -d flag.
gsutil acl ch -d AllUsers gs://BUCKET-NAME/ada.jpg
Delete Objects
Use gsutil rm to delete objects.
gsutil rm gs://BUCKET-NAME/ada.jpg
A “removing” output returns upon success.
CLOUD IAM
Cloud IAM is Google cloud’s identity and access management. This is a service that lets users create and manage permissions for their GCP resources. Cloud IAM unifies access control for GCP services into a single system and provides a consistent set of operations.
These notes will help you assign and remove permissions to a second user.
These section will require logging in as two users. To avoid confusion on Google’s platform I recommend using private mode in your browser for this exercise.
USER 1 IAM & admin > IAM
This enters IAM & admin console.
Click +ADD
You’ll find all of the project roles under the Project category.
The roles are Browser, Editor, Owner and Viewer
These are primitive roles
Roles specify control access & mgmt for all GCP services.
It’s a brief summary
Viewer: Read only
Editor: viewer permissions + permission to modify state
Owner: Editor permissions + manage roles and set up billing
Browser (beta): Read access to browse project hierarchy and Cloud IAM policy. Does not have permission to view project resources.
Explore editor roles
Switch to Username 2
Go to IAM Nav menu > IAM & admin > IAM
The table will show the usernames and their roles
Trying to click +ADD will show that you lack permissions to edit.
Switch to Username 1 for the next step
Prepare a resource for access testing
Ensure that you’re in the Username 1 console
Create a bucket (If you receive an error sign out and sign back in as user 1)
Update the following fields.
Name: choose your own
Default storage class: multi-regional
Note the name of the bucket for later use.
Create
Upload a file to the bucket
Rename the file by clicking the hotdog button
Rename to ‘sample.txt’
Click rename.
Verify project viewer access
Switch to Username 2
Nav menu > Storage > Browser
Verify the user can see the bucket
Username 2 should have “viewer” role
Remove project access
Switch to Username 1
To remove a project viewer
Nav menu > IAM & admin > IAM
Click the pencil next to Username 2 (on the right)
Remove the project viewer access for Username 2 by clicking the trash can.
Click save.
The user will disappear from the list.
Verify Username 2 has lost access
Switch to Username 2
Nav menu > Home
Nav back to cloud storage
Nav menu > Storage > Browser
You will see an error.
Add Storage Permissions
Copy username 2 name
Switch to username 1
Nav menu > IAM & admin > IAM
Click + ADD and paste Username 2 into new members field.
In the Roles field, select Storage > Storage Object Viewer from the drop-down menu
Click SAVE
Verify Access
Switch to Username 2
Username 2 doesn’t have Project Viewer role so the user can’t see the project. But the user does have access to Cloud Storage. So let’s activate the cloud shell.
Open the cloud shel and enter the command to list the contents of the bucket.
gsutil ls gs://[YOUR_BUCKET_NAME]
Output will show the sample file.
Try renaming and deleting the file
Stackdriver
Stackdriver monitoring provides visibility into the performance of cloud-powered applications. Stackdriver is used to collect metrics, events, and metadata from GCP, Amazon Web Services, hosted uptime and lots of other services that can be used with GCP. Stackdriver collaborates with Slack, PagerDuty, HipChat and other messaging services.
Use Stackdriver to monitor VMs and collect logs from 3rd party apps.
Setting up Stackdriver
Create a Compute Engine instance
Nav menu > Compute Engine > VM Instance
Create
Fill in the fields with the following information
Name: lamp-1-vm
Region: us-central1 or asia-south1
Zone: us-central1-a or asia south1-a
Machine type: n1-standard-2
Firewall: Select Allow HTTP traffic
Leave any other fields as default
Click Create
The VM will be ready when the green check mark appears next to the instance.
Add Apache2 HTTP Server to the Instance
To the right of the VM, click SSH to open the instances terminal
Set up an Apache2 HTTP Server with the following commands in the SSH window:
sudo apt-get updatesudo apt-get update
sudo apt-get install apache2 php7.0
Press Y to continue
If php7.0 won’t work then use php5
sudo service apache2 restart
In the console, on the VM instances page find the External IP and click the link to see the Apache2 default page.
Create a Stackdriver account
To use Stackdriver the project must live inside a Stackdriver account.
The following steps will create an account with a free trial period:
Nav menu > Monitoring
Click NO THANKS to trying the beta version
Stackdriver will be ready when you see the Stackdriver dashboard.
Click Install Agents in the top banner. There are commands in the SSH window. Run those commands to install the Stackdriver monitoring agent and logging agent.
The required scripts are below
Monitoring agent install script
curl -sSO https://dl.google.com/cloudagents/install-monitoring-agent.sh
sudo bash install-monitoring-agent.sh
A Logging agent does what it sounds like, it logs streams from your VM instances and from selected third-party software packages.
Logging agent install script
curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
sudo bash install-logging-agent.sh
Click Monitoring Overview to return to the Dashboard.
Create an uptime check
Uptime checks verify that resources are accessible. Let’s create an uptime check to verify that the vm is up.
On the Stackdriver console click Create Check button on the dashboard.
Select Uptime Checks in the left-hand menu
Select Uptime Checks Overview
Click Add Uptime Check
Edit the New Uptime Check panel by adding the following
Title: Lamp Uptime Check
Check type: HTTP
Resource Type: Instance
Applies to: Single, lamp-1-vm
Path: leave at default
Check every: 1 min
Leave any other fields at default
Click Test to verify that your uptime check can connect to the resource.
Green check marks indicate that the connection is ready.
Click Save
When asked about creating an alerting policy click No Thanks
The uptime check takes awhile to become active. We will create an alerting policy for a different resource while we wait.
Create an alerting policy
Open the Stackdriver console and refresh the screen
In the left menu, Alerting > Create a Policy
Next we’ll configure conditions, notifications, and documentation
1. Conditions: click Add Condition
A panel will open, edit the following values, leave all others as default.
Target: Start typing “GCE” in the resource type and metric field then select:
* Resource Type: GCE VM Instance (gce_instance)
* Metric: Type “network” then select Network traffic
Configuration
* Condition: is above
* Threshold: 500 bytes
* For: 1 minute
Click Save
2. Notifications: Select Email Address then enter your personal email address in the Email field
3. Documentation: Add a message in documentation, which will be included in the emailed alert
4. Name the policy: type in the name Inbound Traffic Alert
Click Save
This will create the alert policy. We can create a dashboard and chart and then we will check out logging.
Create a dashboard and chart
Display metrics collected by Stackdriver Monitoring in your own charts and dashboards. We’ll create charts for the lab metrics and customize our dashboard.
1. In the left-and menu of the Stackdriver Monitoring Console select
Dashboards > Create Dashboard
2. Click Add Chart in the upper right of the screen
3. Click into the Find resource type and metric field and start typing “CPU” then select CPU Load (1m).
The GCE VM instance is automatically selected as the Resource type. The chart names itself after the metric you’re using, but we can rename it.
4. Click Save
Now we will create a second chart.
1. Select Add Chart in the upper-right menu of the new dashboard
Find resource type and metric field and start typing “Network” then choose “Received Packets”. Leave the other fields with their default values. Chart data will show in the preview section.
2. Click Save
3. You still need to name your new dashboard. Change Untitled Dashboard to “Stackdriver LAMP Qwik Start Dashboard”
View the Logs
Stackdriver Monitoring and Stackdriver Logging share integrations. We can now view the logs for our lab.
In the Stackdriver left-hand menu, click Logging to see the Logs Viewer. Let’s change the focus to see our logs
In the first drop-down menu select GCE VM Instance > lamp-1-vm
Select syslog in the second drop-down menu and then click OK
Leave other fields as their default
Click the Start streaming logs icon. It looks like the play button at the top beside Create metric and create export.
Now we can see our logs in the VM instance.
Start and stop the VM to see how this affects the Logs
You can see this in real time if you orient the compute engine console and the Stackdriver logging console are side by side.
In the console, in the VM instance window, click on the lamp-1-vm instance
Click Stop at the top of the VM instance details window and then confirm to stop the instance. The instance may take a few minutes to stop.
Watch the Logs view tab for when the VM is stopped.
In the VM instance details window, click Start. Confirm.
It will take a few minutes for the instance to restart.
Watch the log messages to monitor the start up.
Check the uptime check results and triggered alerts
We should be able to see if alerts have been triggered now.
View Stackdriver uptime check results
1. In the let pane, click Uptime Checks > Uptime Check Overview. We should now see information about active uptime checks, including web site status.
2. Check the name of the uptime check.
—If Location results message outputs “No checks have run yet” then wait a few minutes and refresh the page.
Check if alerts have been triggered
1. In the Stackdriver console, in the left page click Alerting > Incidents. If there are no open incidents then look in the RESOLVED tab.
2. Still in the Stackdriver console, click Alerting > Events. You should see a number of Events listed.
3. Check your email account. You should see Stackdriver Alerts in your inbox.
This concludes this exercise. Once complete be sure to remove the email notifications from your alerting policy to avoid further emails.
Lab resources may be active for several minutes. You may continue to receive email notifications during that time.