Google Pub/Sub

  • Author: Ronald Fung

  • Creation Date: 9 June 2023

  • Next Modified Date: 9 June 2024

A. Introduction

Pub/Sub is an asynchronous and scalable messaging service that decouples services producing messages from services processing those messages.

Pub/Sub allows services to communicate asynchronously, with latencies on the order of 100 milliseconds.

Pub/Sub is used for streaming analytics and data integration pipelines to ingest and distribute data. It’s equally effective as a messaging-oriented middleware for service integration or as a queue to parallelize tasks.

Pub/Sub enables you to create systems of event producers and consumers, called publishers and subscribers. Publishers communicate with subscribers asynchronously by broadcasting events, rather than by synchronous remote procedure calls (RPCs).

Publishers send events to the Pub/Sub service, without regard to how or when these events are to be processed. Pub/Sub then delivers events to all the services that react to them. In systems communicating through RPCs, publishers must wait for subscribers to receive the data. However, the asynchronous integration in Pub/Sub increases the flexibility and robustness of the overall system.

To get started with Pub/Sub, check out the Quickstart using Google Cloud console. For a more comprehensive introduction, see Building a Pub/Sub messaging system.

B. How is it used at Seagen

Seagen can use Google Cloud Pub/Sub to create an asynchronous messaging system that allows different components of their application to communicate with each other in a scalable and reliable way. Here are some steps to get started with Google Cloud Pub/Sub:

  1. Create a Google Cloud account: Seagen can create a Google Cloud account in the Google Cloud Console. This will give them access to Google Cloud Pub/Sub and other Google Cloud services.

  2. Create a Pub/Sub topic: Seagen can create a Pub/Sub topic in the Google Cloud Console, which represents a channel for sending messages. They can specify the topic name, subscription settings, and access control policies.

  3. Create a Pub/Sub subscription: Seagen can create a Pub/Sub subscription to the topic, which represents a channel for receiving messages. They can specify the subscription name, delivery settings, and access control policies.

  4. Publish messages: Seagen can publish messages to the Pub/Sub topic, using the Google Cloud Console or the Pub/Sub API. They can specify the message content, metadata, and other properties.

  5. Receive messages: Seagen can receive messages from the Pub/Sub subscription, using the Google Cloud Console or the Pub/Sub API. They can process the messages asynchronously, using different components of their application.

  6. Monitor and report: Seagen can monitor and report on the Pub/Sub topic and subscription, using the Google Cloud Console or the Pub/Sub API. They can track the message flow, monitor the subscription performance, and receive detailed reports on the messaging activity.

Overall, by using Google Cloud Pub/Sub, Seagen can create a scalable and reliable messaging system that allows different components of their application to communicate with each other in real-time. With its support for different message formats, powerful security and monitoring features, and easy-to-use interface, Google Cloud Pub/Sub is an excellent choice for businesses and individuals who need to implement a messaging system that can handle large volumes of messages.

C. Features

At-least-once delivery

Synchronous, cross-zone message replication and per-message receipt tracking ensures at-least-once delivery at any scale.

Open

Open APIs and client libraries in seven languages support cross-cloud and hybrid deployments.

Exactly-once processing

Dataflow supports reliable, expressive, exactly-once processing of Pub/Sub streams.

No provisioning, auto-everything

Pub/Sub does not have shards or partitions. Just set your quota, publish, and consume.

Compliance and security

Pub/Sub is a HIPAA-compliant service, offering fine-grained access controls and end-to-end encryption.

Google Cloud–native integrations

Take advantage of integrations with multiple services, such as Cloud Storage and Gmail update events and Cloud Functions for serverless event-driven computing.

Third-party and OSS integrations

Pub/Sub provides third-party integrations with Splunk and Datadog for logs along with Striim and Informatica for data integration. Additionally, OSS integrations are available through Confluent Cloud for Apache Kafka and Knative Eventing for Kubernetes-based serverless workloads.

Seek and replay

Rewind your backlog to any point in time or a snapshot, giving the ability to reprocess the messages. Fast forward to discard outdated data.

Dead letter topics

Dead letter topics allow for messages unable to be processed by subscriber applications to be put aside for offline examination and debugging so that other messages can be processed without delay.

Filtering

Pub/Sub can filter messages based upon attributes in order to reduce delivery volumes to subscribers.

D. Where Implemented

LeanIX

E. How it is tested

Testing Google Cloud Pub/Sub involves ensuring that messages are being published and received correctly, and that the messaging system is reliable and scalable. Here are some steps to test Google Cloud Pub/Sub:

  1. Create a test topic: Create a test Pub/Sub topic in the Google Cloud Console that mimics the production topic as closely as possible, including the topic name, subscription settings, and access control policies.

  2. Create a test subscription: Create a test Pub/Sub subscription to the topic, specifying the subscription name, delivery settings, and access control policies.

  3. Publish test messages: Publish test messages to the test Pub/Sub topic, using the Google Cloud Console or the Pub/Sub API. Ensure that the messages are being published correctly and that they contain the expected content and metadata.

  4. Receive test messages: Receive test messages from the test Pub/Sub subscription, using the Google Cloud Console or the Pub/Sub API. Ensure that the messages are being received correctly and that they contain the expected content and metadata.

  5. Monitor and report: Monitor and report on the test Pub/Sub topic and subscription, using the Google Cloud Console or the Pub/Sub API. Track the message flow, monitor the subscription performance, and receive detailed reports on the messaging activity.

  6. Repeat the process: Repeat the process as needed, creating additional test topics and subscriptions to test different message formats or to simulate different messaging scenarios.

Overall, by thoroughly testing Google Cloud Pub/Sub, users can ensure that their messaging system is reliable, scalable, and capable of handling large volumes of messages. Additionally, users can reach out to Google Cloud support for help with any technical challenges they may encounter.

F. 2023 Roadmap

????

G. 2024 Roadmap

????

H. Known Issues

While Google Cloud Pub/Sub is a reliable and scalable messaging system, there are some known issues that users may encounter. Here are some of the known issues for Google Cloud Pub/Sub:

  1. Message ordering issues: Users may encounter message ordering issues with Google Cloud Pub/Sub, such as messages being delivered out of order or delayed. These issues can often be resolved by configuring ordering settings, such as using partitioned topics or implementing custom ordering mechanisms.

  2. Access control issues: Users may encounter issues with access control policies, such as policies not being applied correctly or users not being able to access topics or subscriptions due to incorrect permissions. These issues can typically be resolved by reviewing and adjusting access control settings in the Google Cloud Console or using the gcloud command-line tool.

  3. Latency issues: Users may encounter latency issues with Google Cloud Pub/Sub, such as high message delivery times or slow message processing times. These issues can often be resolved by adjusting messaging settings, such as using the appropriate message size or adjusting the message retention period.

  4. Scalability issues: Users may encounter scalability issues with Google Cloud Pub/Sub, such as not being able to scale up or down based on demand. These issues can often be resolved by adjusting the messaging capacity or using automatic scaling policies.

  5. Billing and cost issues: Users may encounter billing and cost issues with Google Cloud Pub/Sub, such as unexpected charges or incorrect usage reports. These issues can often be resolved by reviewing usage reports and monitoring billing statements in the Google Cloud Console.

Overall, while these issues may impact some users, Google Cloud Pub/Sub remains a reliable and scalable messaging system that is widely used by businesses and individuals. By monitoring their Google Cloud Pub/Sub usage and reviewing their usage reports and logs, users can ensure that their messaging resources are secure and accessible, and that they are only paying for the resources they use. Additionally, users can reach out to Google Cloud support for help with any known issues or other technical challenges they may encounter.

[x] Reviewed by Enterprise Architecture

[x] Reviewed by Application Development

[x] Reviewed by Data Architecture