Webhooks for Realtime Cloud Apps Notification With Enterprise Security Products
As enterprises move their assets from the on-prem environment into the cloud using SaaS apps there is a growing problem of securing those assets.
Join the DZone community and get the full member experience.Join For Free
As enterprises move their assets from the on-prem environment into the cloud using sanctioned SaaS apps there is a growing problem of securing those assets in the cloud from accidental exposures and data loss.
A general architecture for products providing security for those apps involves integration via traditional API access mechanism for downloading assets into authorized workload environment, analyzing the content and enforcing security through policy.
You may also like: Enterprise Security Insights: Networks, Architecture, and App Security
In the traditional pull-based API integration approach, there is a delay from the time user does the activity in the cloud to the time the asset becomes available for download in the app and also there is an API rate-limit impact from the cloud app.
What Are Webhooks
Webhooks are automated calls from a cloud app. Those calls are triggered when a specific event happens on the cloud app. For example, if a user uploads a file on a box, the cloud app will trigger an API call to a registered URL.
Use Case and Problem Statement
Traditional security SaaS products use a polling mechanism to pull events from the cloud periodically using a time chunking mechanism. Periodically, time is sliced into small chunks and scheduled in a queue for a specified time interval. A task pulls them from a queue and runs events API call against a cloud app for that period. This is highly inefficient because
- There is no way to know how many events will be present in the time slice.
- The time slice is always changed based on the cloud app.
- Many API calls to the cloud result in 429 errors and subsequent backoff requests from other API calls.
- Non-realtime fetching mechanism for cloud apps.
- No clear picture of how many events are pending processing.
Most apps including Box, Office 365, Google Drive provide the ability to register a URL where they send fast push notification for internal events.
Webhooks are a common solution deployed across many API based service providers that allow customers to hook up a URL.
Switching to a subscription-based push model has few advantages.
- Realtime push notification from the cloud.
- Instant Event discovery.
- No wasted API calls when time slots have no events.
- Fewer API calls resulting in fewer 429 errors.
- More info as part of Event Payload.
Sequence of Steps
- The Cloud App Subscription Service is used for keeping info about each cloud app publisher's APIs and events and register a webhook callback URL.
- The Cloud App Subscription Service subscribes to the Cloud App WebHook.
- As and when an event occurs, HTTP POST callbacks are sent to the registered Prisma SaaS exposed URL.
- The Ingress API Proxy/Gateway will intercept all incoming requests.
- The webhook receiver receives a message.
- The webhook receiver then does initial app-specific event processing and pushes a message to pub/sub-broker for further handling by Worker.
- Pub/Sub broker could be any message broker like Kafka or Redis Pub/Sub.
- Apply Quota and Rate Limit per App.
- Filter Events.
- Authenticate incoming Request OAuth token.
Receiving Controller Service
- Validate Subscription.
- Group Events in batches.
- The route to app-specific webhooks receiver endpoint.
- Initial app-specific event processing.
CloudApp Subscription Service
- Used for setting up a cloud app to register the service URL.
- UI page fronted for this service to set up and authorize the subscription.
- Specialize cloud app-specific receiver handler to massage events.
- Push events to Pub/Sub broker.
- Generic cloud-agnostic message broker.
- Kafka is a good candidate for a topic for each event.
- Cloud App-specific Events topic.
- App workers subscribe to the topic and do further processing.
Prisma SaaS delivers complete visibility and granular enforcement across all user, folder and file activity within sanctioned SaaS applications, providing detailed analysis and analytics on usage without requiring any additional hardware, software or network changes.
Opinions expressed by DZone contributors are their own.