DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Deployment
  4. How Flickr delivers Real-Time Push Notifications using Node.js and Redis

How Flickr delivers Real-Time Push Notifications using Node.js and Redis

Allen Coin user avatar by
Allen Coin
·
Dec. 20, 12 · Interview
Like (0)
Save
Tweet
Share
12.26K Views

Join the DZone community and get the full member experience.

Join For Free
push notifications are, of course, nothing new. recently, joshua cohen at the flickr blog described flickr's process for delivering real-time push notifications using node.js and redis.

flickr's decisions about the underlying architecture centered around four considerations; their push notifications had to have:
  • minimal impact on normal page serving times.
  • near-real-time delivery.
  • deliver thousands of notifications per second.
  • highly available underlying services.

event generation and targeting

they found that their existing push infrastructure was insufficient. the devs found, though, that they could use redis for event generation and targeting:

the event generation phase happens while processing the response to a user request. as such, we wanted to ensure that there was little to no impact on the response times as a result. to ensure this was the case, all we do here is a lightweight write into a global redis queue... everything after this initial redis action is processed out of band by our deferred task system and has no impact on site performance.

message delivery

they built their message delivery using node.js:

flickr’s web-serving stack is php, and, up until now, everything described has been processed by php. unfortunately, one area where php does not excel is long-lived processes or network connections, both of which make delivering push notifications in real time much easier. because of this we decided to build the final phase, message delivery, as a separate endpoint in node.js.

so, the question arose: how do we get messages pending delivery from these php workers over to the node.js endpoints that will actually deliver them? for this, we again turned to redis, this time using its built in pub/sub functionality.

scalability

dividing the workload across individual node.js processes, they published. each message to the redis pub/sub channel as a sharded channel.

whenever we need to add more processing power to the cluster, we can just add more servers and more shards. this also makes it easy to pull hosts out of rotation for maintenance without impacting message delivery: we simply reconfigure the remaining processes to subscribe to additional channels to pick up the slack.

availability

the team wanted to design their push notification system for high availability, which was a particular challenge because they wanted to ensure that they could lose processes, servers, or even data centers without the need for a human operator.

we liked the idea of having one host serve as a backup for another, but we didn’t like having to coordinate the interaction between so many moving pieces. to solve this issue we went with a convention based approach. instead of each host having to maintain a list of its partners, we just use redis to maintain a global lock.

read more on flickr's blog, including code excerpts, here .





Redis (company) push Node.js

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • SAST: How Code Analysis Tools Look for Security Flaws
  • 3 Ways That You Can Operate Record Beyond DTO [Video]
  • Why You Should Automate Code Reviews
  • Streamlining Your Workflow With the Jenkins HTTP Request Plugin: A Guide to Replacing CURL in Scripts

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: