Google Logging
Author: Ronald Fung
Creation Date: 13 June 2023
Next Modified Date: 13 June 2024
A. Introduction
Cloud Logging is a fully managed service that allows you to store, search, analyze, monitor, and alert on logging data and events from Google Cloud and Amazon Web Services. You can collect logging data from over 150 common application components, on-premises systems, and hybrid cloud systems.
Logging includes storage for logs through log buckets, a user interface called the Logs Explorer, and an API to manage logs programmatically. Logging lets you read and write log entries, query your logs, and control how you route and use your logs.
B. How is it used at Seagen
As a biopharma research company, Seagen can benefit from using Google Cloud Logging for their logging and monitoring needs. Here are some ways Seagen can use Google Cloud Logging:
Migrating from Microsoft Azure: Seagen can migrate their existing logs from Microsoft Azure to Google Cloud Logging by exporting the logs from Azure and importing them into Google Cloud Logging. This can be done using the Google Cloud Logging API or other migration tools.
Collecting and analyzing logs: Seagen can collect and analyze logs from their applications, virtual machines, and other resources running on Google Cloud Platform. They can use Google Stackdriver Logging to aggregate and search logs, and can use Google Stackdriver Trace to analyze performance traces and metrics.
Monitoring and alerting: Seagen can monitor their logs and receive alerts in real-time using Google Stackdriver Monitoring. They can set up custom alerts based on specific log events or metrics, and can receive notifications via email, SMS, or other channels.
Managing access and permissions: Seagen can manage access and permissions to their logs using Google Cloud IAM. They can define roles and permissions for different users and groups, and can control access to logs based on resource type, location, or other attributes.
Integrating with other Google Cloud services: Seagen can integrate their logs with other Google Cloud services, such as Google Cloud Pub/Sub, Google Cloud Storage, and Google BigQuery. They can build real-time data pipelines and analytics workflows using these services, and can store and analyze logs at scale.
Overall, by using Google Cloud Logging, Seagen can benefit from a powerful and scalable solution for their logging and monitoring needs. With its support for migrating existing logs, collecting and analyzing logs, monitoring and alerting, managing access and permissions, and integrating with other Google Cloud services, Google Cloud Logging provides a range of options that can meet the needs of a variety of workloads and use cases.
C. Features
Logs Explorer
Logs Explorer enables you to search, sort, and analyze logs through flexible query statements, along with rich histogram visualizations, a simple field explorer, and ability to save the queries. Set alerts to notify you whenever a specific message appears in your included logs, or use Cloud Monitoring to alert on logs-based metrics you define.
Regional log buckets
Use log buckets as part of your local or industry-specific compliance strategy. Log buckets store and process your workload’s logs data only in the region you specify. These buckets feature customizable access control and retention.
Error Reporting
Error Reporting automatically analyzes your logs for exceptions and intelligently aggregates them into meaningful error groups. See your top or new errors at a glance and set up notifications to automatically alert you when a new error group is identified.
Cloud Audit Logs
Cloud Audit Logs helps security teams maintain audit trails in Google Cloud. Achieve the same level of transparency over administrative activities and access to data in Google Cloud as in on-premises environments. Every administrative activity is recorded on a hardened, always-on audit trail, which cannot be disabled by any rogue actor.
Logs Router
Cloud Logging receives log entries through the Cloud Logging API where they pass through the Logs Router. The Logs Router checks each log entry against existing inclusion filters and exclusion filters to determine which log entries to discard, which to ingest, and which to include in exports.
D. Where Implemented
E. How it is tested
Testing Google Cloud Logging involves ensuring that the logging infrastructure is properly configured and optimized for performance, reliability, and security. Here are some steps to test Google Cloud Logging:
Create a test environment: Create a test environment that mimics the production environment as closely as possible, including the applications, virtual machines, and other resources that generate logs. Ensure that the logging infrastructure is properly configured and that the security policies are in place.
Deploy the logging infrastructure: Deploy the logging infrastructure on Google Cloud Platform. Ensure that the infrastructure is properly configured and that it can communicate with other resources, such as applications or APIs.
Test log collection: Test the log collection by generating test log events using log generation tools, such as Stackdriver Logging API or other log generation tools. Ensure that the log events are being collected properly and that there are no errors or missing logs.
Test log search and analysis: Test the log search and analysis by using log search and analysis tools, such as the Stackdriver Logging Console or other log analysis tools. Ensure that the logs are searchable and that there are no errors or timeouts.
Test log monitoring and alerting: Test the log monitoring and alerting by setting up custom alerts based on specific log events or metrics, and by receiving notifications via email, SMS, or other channels. Ensure that the alerts are triggered properly and that there are no false positives or false negatives.
Test log access and permissions: Test the log access and permissions by using Google Cloud IAM to define roles and permissions for different users and groups, and by controlling access to logs based on resource type, location, or other attributes. Ensure that the access control policies are working as expected and that there are no unauthorized access or data breaches.
Test log integration: Test the log integration with other Google Cloud services or third-party tools by using log integration testing tools, such as Google Cloud Pub/Sub or Microsoft Azure Event Grid. Ensure that the logs are integrated properly and that there are no integration issues or errors.
Overall, by thoroughly testing Google Cloud Logging, users can ensure that their logging infrastructure is properly configured and optimized for performance, reliability, and security. Additionally, users can reach out to Google Cloud support for help with any technical challenges they may encounter.
F. 2023 Roadmap
????
G. 2024 Roadmap
????
H. Known Issues
While Google Cloud Logging is a reliable and widely used solution for logging and monitoring needs, there are some known issues that users may encounter. Here are some of the known issues for Google Cloud Logging:
Log ingestion issues: Users may encounter log ingestion issues, such as dropped log events or delayed log events, especially for workloads that generate a high volume of logs. This issue can often be resolved by using the appropriate log ingestion settings, such as batch sizes or batch intervals, and by monitoring the log ingestion rate.
Log search issues: Users may encounter log search issues, such as slow queries or missing logs, especially for workloads that require complex log search queries or real-time log analysis. This issue can often be resolved by using the appropriate log search settings, such as log filters or search indexes, and by optimizing the log search queries.
Alerting issues: Users may encounter alerting issues, such as false positives or false negatives, especially for workloads that require high accuracy or low latency alerts. This issue can often be resolved by using the appropriate alerting policies, such as threshold values or alert conditions, and by testing the alerting policies in a test environment.
Security issues: Users may encounter security issues, such as unauthorized access or data breaches, especially for workloads that require high security. This issue can often be resolved by using the appropriate security policies and access controls, such as firewall rules and IAM roles.
Integration issues: Users may encounter integration issues with other cloud services or third-party tools, such as data pipelines or analytics platforms. This issue can often be resolved by using industry-standard protocols and APIs to enable interoperability between different cloud services and tools.
Overall, while these issues may impact some users, Google Cloud Logging remains a powerful and reliable solution that is widely used by businesses and organizations around the world. By monitoring their performance and security alerts and logs, reviewing their logging configuration and policies, and using best practices and industry standards, users can ensure that their logging infrastructure running on Google Cloud Logging is optimized for performance, reliability, and security. Additionally, users can reach out to Google Cloud support for help with any known issues or other technical challenges they may encounter.
[x] Reviewed by Enterprise Architecture
[x] Reviewed by Application Development
[x] Reviewed by Data Architecture