DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • JSON-Based Serialized LOB Pattern
  • Allow Users to Track Fitness Status in Your App
  • Required Capabilities in Self-Navigating Vehicle-Processing Architectures
  • Distribution Design Patterns in Java - Data Transfer Object (DTO) And Remote Facade Design Patterns

Trending

  • Cognitive AI: The Road To AI That Thinks Like a Human Being
  • Send Your Logs to Loki
  • Java Parallel GC Tuning
  • Hugging Face Is the New GitHub for LLMs
  1. DZone
  2. Data Engineering
  3. Data
  4. Getting started with DataWeave: Part 1

Getting started with DataWeave: Part 1

Shana Pearlman user avatar by
Shana Pearlman
·
Sep. 15, 15 · Tutorial
Like (1)
Save
Tweet
Share
7.69K Views

Join the DZone community and get the full member experience.

Join For Free

Data transformation is an inevitable component of connectivity, as most systems don’t speak the same language. Even when the format is similar, as when two RESTful Web APIs exchange JSON payloads, their message structure typically differs, making translation a necessity.

In this the first of a four part series on our new data transformation engine, DataWeave, we will help you learn the basics of this elegant and lightweight expression language. We will first focus our attention on the canonical format which is the immediate result of every expression you execute in DataWeave. In the following posts, we will explore the power of DataWeave expressions and solve some real-world use cases.  We will also be presenting a webinar that introduces DataWeave on September 30, at 10 AM PST.

From DataMapper to DataWeave

Those of you familiar with Anypoint Platform will know DataMapper, our first solution for providing a graphical drag and drop approach to data transformation. We learned, based on customer feedback, that users were looking for a more powerful transformation engine.  With DataWeave, we coupled the transformation engine with the Mule runtime engine to ensure high performance, adopted a language first approach and designed the language with a simple JSON like syntax. As a result, it costs a fraction of the time to write a DataWeave expression compared to a similar use case in DataMapper.

Development Environment

As part of the design time experience for DataWeave you now have the facility to indicate to DataSense the structure of every message by explicitly setting it in the Metadata tab which is present in the other Message Processors. You should also be mindful of the mime-types of the Message data you have to transform. When this is not set explicitly at design time and when you use any of our Connectors, this will default to application/java. For HTTP requests and responses Mule will look at the Content-Type header and set it accordingly. You should set the mime-type explicitly when you are using the set-variable and set-payload processors. Failing to set the mime-type explicitly for xml content will result in it being interpreted as a mere Java string by the engine.Screen Shot 2015-08-30 at 6.21.46 PMAnypoint Studio presents a DataWeave editor with 3 panes. On the left, DataSense displays the structure of the incoming message, together with any example data you have provided. On the right pane, the expected outgoing structure is displayed as per DataSense as well as the design time result of your transformation which constantly refreshes itself as you write the expression.Screen Shot 2015-08-22 at 3.19.49 PMThe middle pane is the transformation pane. This is divided into two sections. The header is where you declare the mime type of the output of your transformation. You can also declare reusable functions and global variables as also namespaces for xml use cases. At a minimum you must declare the %dw 1.0 header and the output mime-type. Below the dotted line is where you author your expression. You only need one expression. There are a number of expression types, but for the majority of your transformations, you will use a semi-literal expression of a DataWeave Object. The same DataWeave transformer can produce multiple outputs from the same incoming Message. You do this by clicking on the plus circle at the bottom right of the transformation pane. Doing so will present you with another transformation pane and you are free to specify a different target (flowVars, etc.) by choosing a different Output from the drop-down at the top of the transformation pane.

Literal Expressions

There are only 3 data types that can result from any DataWeave expression: Simple Types, like Strings and numbers, Arrays and Objects. Each of these can be expressed literally:Screen Shot 2015-08-31 at 7.15.25 PMYou will typically use semi-literal object expressions to define your transformation. To understand this, first consider the example object literal on row 2 above. A DataWeave object is a sequence of key:value pairs. The key is a string without the quotes and the value can be either a simple type, an array, or an object. However, you are not obliged to express the keys and values literally. You can use any expression that returns a string for the key and any expression at all for the value. There are a number of different expression types available to you. We will explore more of these in the next parts of this series. For now we concentrate on the combination of literal expressions and variable references.

Variable Reference Expressions

Key to learning DataWeave is an understanding of the normalization of data which occurs whenever variable references are made to the payload or flowVars, sessionVars, recordVars, inboundProperties or outboundProperties. These expressions convert the incoming data into the canonical DataWeave format. Hence, it makes no difference if your incoming data is JSON or XML or CSV or Java, as these will always be resolved to DataWeave simple types, arrays or objects. You do well to think only in terms of DataWeave data types as these set the context for your expressions and are the result of each expression. Here is an example of how an incoming xml document is resolved to an object. Note the convenient learning experience afforded to you when you declare the output to be application/dw.Screen Shot 2015-08-30 at 11.47.00 AMNote how each element in the xml is normalized as a key:value pair in the object. When the content type of an element is complex, then its normalized value is itself an object. Note also that repeating elements are normalized as repeating key:value pairs. Consider the following CSV. It is normalized as an array of Objects with key:value pairs that correspond to the name of the header and column value respectively.Screen Shot 2015-08-30 at 8.36.33 PMNow, take a look at the result of using a semi-literal expression which wraps the same payload in an object which declares both the forecasts and the city for which they were made:Screen Shot 2015-08-30 at 8.34.58 PM

Output Rendering

The DataWeave engine separates the actual transformation process within the canonical format from the final rendering of the same in the output mime type you defined in the header. It does not matter how complex your expression is. If it’s an object with deep nested structure and expressions and operators which combine expressions, all of these must be executed first and return their value before the outer expression returns its value. It is this final value which is rendered in the output mime-type. You need to be mindful of the constraints imposed by your choice of mime-type. Arrays will not be rendered in xml. You should choose to repeat keys instead. Likewise, repeating keys cannot be rendered in Json. You should generate arrays instead. To render an object to XML, in line with the rule that XML documents may only contain one root element, the object may only contain one key:value pair. Its value can, as we have seen, itself be an object of any complexity. Consider the above transformation, modified accordingly to render a valid XML document.Screen Shot 2015-08-30 at 11.07.17 PM



Object (computer science) Data (computing)

Published at DZone with permission of Shana Pearlman. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • JSON-Based Serialized LOB Pattern
  • Allow Users to Track Fitness Status in Your App
  • Required Capabilities in Self-Navigating Vehicle-Processing Architectures
  • Distribution Design Patterns in Java - Data Transfer Object (DTO) And Remote Facade Design Patterns

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: