DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • Using Render Log Streams to Log to Papertrail
  • Mastering Node.js: The Ultimate Guide
  • Reactive Event Streaming Architecture With Kafka, Redis Streams, Spring Boot, and HTTP Server-Sent Events (SSE)
  • Node.js: Architectural Gems and Best Practices for Developers to Excel

Trending

  • Monkey-Patching in Java
  • REST vs. Message Brokers: Choosing the Right Communication
  • Software Verification and Validation With Simple Examples
  • Build a Serverless App Fast With Zipper: Write TypeScript, Offload Everything Else
  1. DZone
  2. Coding
  3. JavaScript
  4. Storing Created Files With Node.js

Storing Created Files With Node.js

Writing to cloud storage has never been easier. With options ranging from sending from a file on a disk to using streams, there is a solution for virtually every use case.

Duncan Brown user avatar by
Duncan Brown
·
May. 15, 16 · Analysis
Like (1)
Save
Tweet
Share
7.64K Views

Join the DZone community and get the full member experience.

Join For Free

You have the following requirement: You need to create a file in your web application (say, creating a PDF) and store said file to be accessed later by your web application.

For storage these days, cloud services have come into their own as of late. Take, for example, two offerings by a pair of the world's leading cloud provider: Amazon's S3 Storage service and Microsoft Azure's Storage service. By offloading the storage and management to a remote, cloud-based service, we are able to (affordably) work with the files we need to without burdening our own application/storage servers.

And given current network throughput and available bandwidth, accessing these services via HTTP has made the whole solution a simple, workable one.

So you say to yourself, "Great! I have this super-modern Node.js application that can create my files. Time to save me some blobs and documents in the cloud!"

Not so fast, bucko.

Image title

(Courtesy: frinkiac.com)

Let's have a look at a few ways to handle this scenario, and see if we can't find an ideal way to do it. "Look before you leap," as they say.

Option 1: The Easy Way

One might argue that the quickest (or, at least, the most obvious) way to create and upload your file is the following:

  1. Create the file and save it to local disk temporarily.

  2. Use the desired cloud storage API and point it at the file on disk.

  3. Send.

  4. Delete/clean up any temporary files stored.

Sure, this looks easy, but with any dramatic increase in traffic, you look to have your server I/O-bound as it must do several disk operations in order to save, read, and ultimately delete the temporary file. Not such a hot option.

How can we improve upon this?

Option 2: This Disk-ussion Is Over!

What about cutting out the disk ops altogether? What if we employed a Node.js specialty, i.e. memory streams?

The process would look something like this:

  1. Create a memory stream.

  2. Create the file and .pipe() it to the memory stream above.

  3. Take the memory stream and use your cloud API to read the stream in order to create a new, API-specific stream.

  4. Finish writing to the stream above.

This is definitely an improvement, but the introduction of an intermediate stream feels unnecessary and, at worst, can burden your server's RAM.

Is there a better way still?

Option 3: API Streams

This option is proof that the cloud providers are listening to the needs of developers. For example, in the case of both AWS and Azure, each allows you to pipe directly to an API-specific stream. So now your process goes something like this:

  1. Create an API-specific stream.

  2. Create the file and .pipe() it to the stream from above.

  3. You're done.

No intermediate streams or unnecessary steps! The code itself is brief and to the point.

Conclusion

Writing to cloud storage has never been easier. With options ranging from sending from a file on disk to using streams, there is a solution for virtually every use case.

Happy writing!

Stream (computing) Node.js

Opinions expressed by DZone contributors are their own.

Related

  • Using Render Log Streams to Log to Papertrail
  • Mastering Node.js: The Ultimate Guide
  • Reactive Event Streaming Architecture With Kafka, Redis Streams, Spring Boot, and HTTP Server-Sent Events (SSE)
  • Node.js: Architectural Gems and Best Practices for Developers to Excel

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: