Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Don’t Let Your DataMapper Streaming be Out of Control

DZone's Guide to

Don’t Let Your DataMapper Streaming be Out of Control

· Integration Zone
Free Resource

Discover how Microservices are a type of software architecture where large applications are made up of small, self-contained units working together through APIs that are not dependent on a specific language. Brought to you in partnership with AppDynamics.

Originally authored by Mariano Simone

As Nicolas pointed out in “7 things you didn’t know about DataMapper“, it’s not a trivial task to map a big file to some other data structure without eating up a lot of memory.

If the input is large enough, you might run out of memory: either while mapping or while processing the data in your flow-ref

Enabling the “streaming” function in DataMapper makes this a lot easier (and efficient!).

just enable "streaming"

But just doing this doesn’t let you decide how many records at a time you want to get passed on to the next processor: in the worst-case-scenario, you might end up with just one line at a type processing. If your next processor is a database, you will have as many queries as lines in your file.

There is, however, a little trick to gain fine-grained control on how many lines are being processed: setting the batchSize property of the foreach:

<foreach batchSize="100" doc:name="For Each">
    <logger message="Got a bunch! (of #[payload.size()])" level="INFO" doc:name="Logger"/>
</foreach>

If you want to see this in action, go grab the example app, import it into studio and start playing around ;)


Discover the six challenges and best practices in managing microservice performance, brought to you in partnership with AppDynamics.

Topics:

Published at DZone with permission of Ross Mason, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}