DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

The Latest Integration Topics

article thumbnail
How to Integrate Vue.js Applications With Drupal
Vue.js is a popular JavaScript framework to create interactive applications. Did you know it provides data-reactive components with a simple and flexible API?
August 28, 2021
by Chris Bateson
· 4,455 Views · 2 Likes
article thumbnail
Python and Low-Code Development: Smooth Sailing With Jupyter Notebooks
Editor's Note: The following is an article written for and published in DZone's 2021 Low-Code Development Trend Report. If you ride on a sailboat in a steady breeze, it glides through the water effortlessly. Few things can compare to crossing a bay without the noise and commotion of a thundering internal combustion engine. The dream of no-code and low-code development is to effortlessly glide from problem to solution. Back in the ‘80s, no-code/ low-code development was called “end-user computing.” Since the invention of the spreadsheet, we’ve had a kind of low-code computing. The technologies continue to evolve. Let’s take our boat out of the slip and sail around a little. We’ll look at two ways Python is commonly used to create no-code and low-code solutions to problems. The first thing we’ll look at is using JupyterLab to create solutions to problems with minimal code. I liken this to using winches to help lift the sails. It’s not an internal combustion engine, but it is a machine that helps us manipulate heavy, bulky items like sails and anchors. The second thing we’ll sail past is using Python as an integration framework to knit together tools and solutions that aren’t specifically written in Python themselves. In this case, we’re going to be writing integration code, not solution code. This is how sailboats work; given a hull and some masts, you’ll have to select and set the sails that will make the boat move. JupyterLab As a developer, and as a writer about Python, I rely on JupyterLab a lot. It’s often the tool I start with because I get handy, pleasant spreadsheet-like interaction. The “spreadsheet” feature that I’m talking about is the ability to change the value of one cell, and then recompute the remaining cells after that change. This lets me create elegant, interactive solutions where the bulk of the interaction is handled by the notebook’s internal model: a lattice of inter-dependent cells. In this picture, we can see the computation of cell 2 depends on prior results in cell 1. I consider this “low code” because there’s a vast amount of code we don’t need to write. The Jupyter Notebook provides us an interactive framework that’s robust and reliable. More important than that, the framework fits the way a lot of people have grown to use computers by putting values into some cells and checking results in other cells. The big difference between Jupyter and a spreadsheet is Jupyter lets us write extensions and expressions in Python. Recently, I had a boat-related problem crop up that was perfect for this kind of low-code processing. The tank under the pointy part of the boat (the “V-berth”) has a fairly complex shape. The question really is: “How big is it?” The problem is access; I have to make a series of approximations and models. I really need a spreadsheet-like calculator, but I need a lot more mathematical processing than is sensible in a spreadsheet. Consequently, let me apologize to any math-phobic readers. This example involves a lot of math. For folks who don’t like math, think of using a spreadsheet where you never looked at the formula in a given column. A notebook can (and often does) hide the details. This specific example doesn’t try to hide the details. Here’s an example: https://github.com/slott56/replacing-a-spreadsheet. I want to lift up a few key features of this low-code development approach. I created three notebooks, each of which had a common structure. They all have a collection of measurements as the input. As the output, it reports a computed volume. The rest of the cells provide some background to help me make sure the math is right. Here’s the input cell’s content in the “prism.ipynb” notebook. This is cell 8. Python measured = { # Forward triangle, in inches "h_f": 8, "w_f": 10 + Rational(1, 2), # Aft triangle, in inches "h_a": 27, "w_a": 48, # Overall length from forward to aft, in inches. "l_fa": 46, } This is a Python dictionary that has a number of input values. This is a bit more complex-looking than spreadsheet cells, making it solidly “low-code” not “no-code.” The output is computed in cell 10, providing a single numeric result with a format like the following: '50 85/88 gallons' This shows me that the volume of space, given the measurements, is just shy of 51 gallons. The best part about this data is there are two kinds of changes I can make. The most important change is to the measurements, which leads to recomputing the notebook. Anyone who can use a spreadsheet can alter the numbers to the “measured” dictionary to recompute the volume. The other change I can make is to adjust some of the underlying assumptions. This is a more nuanced change to the model that is also implemented in the notebook. I find that for a wide variety of problems, a notebook is the place to start. It lets me gather data, formulate alternative approaches to algorithms and data structures, and work out candidate solutions that are richly interactive. This is an excerpt from DZone's 2021 Low-Code Development Trend Report. For more: Read the Report The JupyterLab project is like a boat with three masts and dozens upon dozens of sails for all conditions and situations. There are a lot of features that can be used to create interactive solutions to problems. The idea is to write only the code that’s directly related to the problem domain and leverage the notebook’s capabilities to present the solution so that people can make decisions and take actions. In addition to the notebook for a low-code interactive user experience, we can look at Python as an engine for integrating disparate applications together. Python as Integration Engine Python’s standard library includes modules to help us work with OS resources, including other applications. Rather than build or modify existing code, we can treat applications as opaque boxes and combine them to create an integrated solution. To a limited extent, integration applications is what shell scripts do. There’s a world of difference, however, between integration with a shell script and integration with Python. Shell scripting involves the almost impossible-to-understand shell programming language. (See this article on replacing shell scripts with Python for more thoughts on this.) When we integrate applications with Python, we can easily introduce additional computations and transformations. This can help to smooth over gaps, remove manual operations, and prevent potential errors. I’m a fan of code like the following: Python command = [ "markdown_py", "-v", "-e", "utf-8" ] temp = options.input.with_suffix(".temp") output = options.input.with_suffix(".html") params = dict(CSS=str(options.style), TITLE=options.title) with temp.open('w') as temporary, options.input.open() as source: subprocess.run(command, stdout=temporary, stdin=source) This is part of a larger and more complex script to publish a complex document written in markdown. It has a lot of code examples, which are a lot easier to read in HTML format. I must do some pre-processing of the markdown and some post-processing of the HTML. It seems easiest to execute the markdown_py command from inside a Python script, avoiding a complex python-bash-python kind of process. Since I’m not modifying the underlying applications, I find this fits with a low-code approach. I’m using the source application (markdown_py) for the thing it does best — adjusting the inputs and outputs using Python. Conclusion We can use Python in a variety of ways. It’s a programming language, so we can build code. More importantly, we can use the vast number of pre-built Python libraries to create low-code solutions. We can use a Jupyter Notebook as a low-code way to create a sophisticated interactive experience for users, and we can use Python to integrate other applications. Sailing isn’t effortless. The boat glides only when the sails are set properly, and we keep the rudder in the right position. Just as skill and expertise are required to make a boat move, so too is careful attention needed to write the minimal Python code to solve an information processing problem. Steven Lott, Writer, Python Guru & Retiree @slott on DZone | @s_lott on Twitter | slott-softwarearchitect.blogspot.com Steven has been programming since the 70s, when computers were large, expensive, and rare. As a former contract software developer and architect, he worked on hundreds of projects from very small to very large. He’s been using Python to solve business problems for over 20 years. His titles with Packt Publishing include Python Essentials, Mastering Object-Oriented Python, Functional Python Programming, Python3 Object-Oriented Programming, and Python for Secret Agents. Steven is currently a technomad who lives in various places on the east coast of the US.
August 27, 2021
by Steven Lott
· 10,686 Views · 2 Likes
article thumbnail
Testcontainers: From Zero To Hero [Video]
Do you want to make the most out of your integration tests by using the popular Testcontainers library? Learn how to use it from the ground up.
August 27, 2021
by Marco Behler
· 8,363 Views · 4 Likes
article thumbnail
Token-Based Security, OAuth 2.0, OIDC and IdentityServer4
We'll learn the basics of token-based security, OAuth, OIDC basics, and set up an ASP .NET Core Web Application with IdentityServer4 package.
August 27, 2021
by Jawad Hasan Shani DZone Core CORE
· 6,760 Views · 2 Likes
article thumbnail
How To Implement and Design Twitter Search Backend Systems using Java Microservices?
In this article, you’ll learn about implement and design Twitter search backend systems functionality how to work using Java microservices with explanations.
August 25, 2021
by James Warner
· 5,533 Views · 3 Likes
article thumbnail
Popular APIs for the Internet of Things (IoT) Systems
A mobile app developer wishes to build IoT applications using APIs or programming interfaces which helps them a lot in their work.
August 23, 2021
by Rahul Panchal
· 27,437 Views · 4 Likes
article thumbnail
9 Best Free and Open Source Tools for Reporting
Using data reporting tools has its pros and cons but with such a variety of options, everyone can find one which covers all needs.
August 21, 2021
by Angelina Shevchuk
· 29,222 Views · 9 Likes
article thumbnail
Unleash the power of Anypoint DataGraph
With Anypoint DataGraph, you can reuse multiple APIs in a single request. Enterprise architects can easily unify APIs into one data service all without writing more code. Developers can consume multiple APIs from the data service in a single GraphQL request.
Updated August 21, 2021
by Jitendra Bafna
· 25,265 Views · 3 Likes
article thumbnail
Getting Started With Protobuf Using C# Runtime Library for Protocol Buffers
In this post, take a look at how we can quickly get started with Protobuf using C# runtime library for Protocol Buffers with only a few steps.
August 20, 2021
by Kapil Khandelwal
· 19,525 Views · 2 Likes
article thumbnail
SaaS User Management Tools Comparison for 2021
Are you looking for a way to manage users, permissions, and roles on an ongoing basis? Read here about our comparison of SaaS User Management tools.
August 17, 2021
by Or Hillel
· 7,185 Views · 1 Like
article thumbnail
What is MuleSoft and Anypoint Platform Capabilities and Strengths
Anypoint Platform is a Multitenant Integration Platform as a Service and Cloudhub is the designed to provide enterprise multitenant, secure, elastic and high available Integration Platform as a Service.
Updated August 15, 2021
by Jitendra Bafna
· 28,233 Views · 4 Likes
article thumbnail
MuleSoft API Led Connectivity Architectural and Design Patterns
API Led Connectivity is an architectural style that connects data to application through reusable and purposeful API.
Updated August 13, 2021
by Jitendra Bafna
· 76,898 Views · 6 Likes
article thumbnail
Custom Rate Limiting for Microservices
Enforcing rate limits on microservices is a common requirement in the API economy. In this article, we are going to build a custom rate limiting solution.
August 13, 2021
by Randhir Singh
· 30,152 Views · 9 Likes
article thumbnail
Integrating Pub/Sub With MuleSoft
iPaaS tools are built on synchronous RESTful APIs. For your marketing app to be integrated with your CRM app, you need to have appropriate RESTful endpoints.
August 12, 2021
by Himanshu Gupta
· 17,134 Views · 4 Likes
article thumbnail
Using OKTA as Client Provider in Mulesoft
There are many ways to secure your API in Mulesoft and in this article we are going to address the topic of identity and access control with OpenID connect.
August 12, 2021
by Gaurav Dhimate
· 16,710 Views · 42 Likes
article thumbnail
API Security 101: Lack of Resources and Rate Limiting
Data, data, everywhere. How the lack of rate limiting contributes to severe security issues...
August 12, 2021
by Vickie Li
· 9,991 Views · 3 Likes
article thumbnail
Container Attached Storage (CAS) vs. Shared Storage: Which One to Choose?
An Overview of Storage in Kubernetes Kubernetes supports a powerful storage architecture that is often complex to implement unless done right. The Kubernetes orchestrator relies on volumes-abstracted storage resources - that help to save and share data between ephemeral containers. Since these storage resources abstract the underlying infrastructure, volumes enable dynamic provisioning of storage for containerized workloads. In Kubernetes, shared storage is typically achieved by mounting volumes and connecting to an external filesystem or block storage solution. Container Attached Storage (CAS) is a relatively newer solution that allows Kubernetes administrators to deploy storage as containerized microservices in a cluster. The CAS architecture makes workloads more portable and simpler to modify storage based on application needs. Because CAS is deployed per workload or per cluster, it also eliminates the cross workload and cluster blast radius of traditional shared storage. This article compares CAS with traditional shared storage to explore their similarities, differences and architecture overview. Container Attached Storage: Container Attached Storage (CAS) is a solution for stateful workloads that deploys storage as a cluster running in the cloud or on-premises. Unlike traditional storage options where storage is a shared filesystem or block storage running externally, CAS enables storage controllers that can be managed by Kubernetes. These storage controllers can run anywhere with a Kubernetes distribution, whether on top of traditional shared storage systems, or managed storage services like Amazon EBS. Data stored in CAS is accessed directly from containers within the cluster, thereby significantly reducing Read/Write times. Architecture Overview: CAS leverages the container orchestrator’s environment to enable persistent storage. The CAS software has storage targets in containers that run as services. If desired, these services are replicated as microservice-based storage replicas that can easily be scheduled and scaled independently of each other. CAS services can be orchestrated using Kubernetes or any other orchestration platform as containerized workloads, ensuring the autonomy and agility of software development teams. For any CAS solution, the cluster is typically divided into two layers: The control plane consists of the storage controllers, storage policies, and instructions on how to configure the data plane. Control plane components are responsible for the provisioning volumes and other storage associated tasks. The data plane components receive and execute instructions from the control plane on how to save and access container information. The main element of the data plane is the Storage Engine which implements pooled storage. The engine is essentially responsible for the Input-Output volume path. Some popular storage engines of OpenEBS include Mayastor, cStor, Jiva and OpenEBS LocalPV. Some prominent users of OpenEBS include the CNCF, ByteDance(Tiktok), Optro, Flipkart, Bloomberg and others. Features: Container Attached Storage is built to primarily run on Kubernetes and other cloud-native container orchestrators. This makes the solution inherently platform-agnostic and portable, thereby making it an efficient storage solution that can be deployed on any platform without the inconvenience of vendor lock-in. CAS decomposes storage controllers into constituent units that can be scaled and run independently. Every storage controller is attached to a Persistent Volume and typically runs within the user-space, achieving storage granularity and independence from underlying operating systems Control plane entities are deployed as Custom Resource Definitions that deal with physical storage entities such as disks Data plane entities are deployed as a collection of PODs running in the same cluster as the workload The CAS architecture can offer synchronous replication in order to add additional availability. When to Use: Container Attached Storage is steadily becoming the de-facto standard for persistent storage of stateful Kubernetes workloads. CAS is most like the Direct Attached Storage that many current workloads expect, such as NoSQL, logging, machine learning pipelines, Kafka and Pulsar. Many workload communities and users have embraced CAS. CAS also allows small teams to retain control over their workloads. In short, CAS may be preferred where: The workloads expect local storage Teams want to be able to efficiently turn local storage, including disks or cloud volumes, into volumes on demand for Kubernetes workloads Performance is a concern The loose coupling of the architecture is desired to be maintained at the storage layer Increased density of workloads on hosts is desired Small team autonomy is desired to be maintained Traditional Shared Storage: Shared storage was designed to allow multiple users/machines to access and store data in a pool of devices. Shared storage provided additional availability to workloads that themselves were unable to provide for their own availability; additionally, shared storage was able to work around the poor performance of the underlying disk which at the time we're able to deliver no more than 150 I/O operations per second. Today’s underlying drives can be 10,000 times more performant; massively faster than the performance requirements of most workloads. A shared storage infrastructure typically consists of block storage systems in Storage Area Networks (SANs) or file system based storage devices in Network Attached Storage (NAS) configurations. Adoption While the storage industry was once a rapidly growing industry, with growth rates in excess of 30% - 50% YoY in the late 1990s and early 2000s. In the 2010s this growth rate moderated and in certain years stopped entirely. In the 2020s growth started again, however, at a rate much slower than the exponential growth in the amount of data storage. Meanwhile, Direct Attached Storage and Cloud storage each grew more quickly in terms of capacity shipped and overall spending. Architecture Overview In traditional shared storage, all nodes in a network share the same physical storage resources but have their own private memory and processing devices. Files and other data can be accessed by any machine connected to the central storage. For a Kubernetes application, traditional shared storage is first implemented by using monolithic storage software to virtualize physical storage resources, which could either be bare-metal servers, SAN/NAS networks or block storage solutions. The software then connects to Persistent Volumes that store cluster data. Each Persistent Volume (PV) is bound to a Persistent Volume Claim (PVC) which application PODs use to request a portion of the shared storage. Both CAS and shared storage can utilize the Container Storage Interface (CSI). CSI is used to issue the commands to the underlying storage such as the need to provision a PV or to expand or snapshot that capacity. Features: Embraces centralized, consolidated storage for Block and File Storage systems, allowing administration from a single interface. Traditional storage is distinctly divided into 3 layers: the Hosts tier which has client machines, the Fabric layer which includes switches and other networking devices, and the storage layer which includes the controllers used to read/write data onto physical disks. Shared storage integrates redundancy into the design of storage devices, allowing systems to withstand failure to a sizable degree. To scale up traditional shared storage, additional storage devices should be deployed and configured into the existing array. When to Use Shared storage is used to manage large amounts of data generated and accessed by a number of different machines. This is because traditional shared storage enables high performance for large files with no bottlenecks or downtimes. Shared storage is also the go-to storage solution for organizations that depend on collaboration between teams. As data and files are managed centrally, shared storage allows efficient version control and consolidated information management. Traditional Shared Storage is also used to eliminate the need for multiple drives containing the same information, which helps reduce redundancies, thus increasing storage capacity. CAS vs. Shared Storage The two storage options vary greatly in how they persist application data. While traditional shared storage relies on an external array of storage devices to persist data, CAS uses containers within an orchestrated environment. Following are a few similarities and differences between CAS and Traditional Shared Storage: Similarities: Both CAS and traditional shared storage offer high availability storage for applications. CAS allows high availability using Data POD replicas that ensure storage is always available for the CAS cluster. While traditional shared storage uses a redundant design to ensure that the storage system can withstand failure. Both options provide quick storage options for critical applications. CAS uses agile microservices to ensure quick I/O times while shared storage allows multiple machines to quickly read and write data on a shared pool of storage devices, reducing the need to create connections between individual machines. Both solutions accommodate software-defined storage which leverages the performance of physical devices with the agility of software. Both can utilize the Container Storage Interface (CSI) to issue the commands to the underlying storage. Both can be Open Source, extending the openness of Kubernetes to the data layer. It appears that container attached storage is somewhat more likely to be open source however that is yet to be determined conclusively. Differences CAS follows a container-based microservice framework for storage, which means teams can take advantage of the agility and portability of containers to ensure faster, more efficient storage. On the contrary, traditional shared storage involves different virtual or physical machines reading/writing into a shared pool of storage devices, thereby increasing latency and reducing access speeds. CAS is platform-agnostic. This means CAS-based storage solutions can run either on-premises or the cloud, without requiring extensive configuration changes. While shared storage relies on Kernel modifications, making it is inefficient to deploy for workloads across different environments. While traditional shared storage relies on consolidated monolithic storage software, CAS runs on the userspace, enabling independent management capabilities for efficient storage administration at the granular level. CAS allows linear scalability since storage containers can be brought up as required, while in traditional shared storage, scaling involves adding newer devices to an existing storage array Summary Designed in Kubernetes, CAS enables agility, granularity and linear scalability, making it a favourite for cloud-native applications. Traditional shared storage offers a mature stack of storage technology that mainly falls short in persisting storage for stateful applications due to the inherent lack of linear scalability. CAS is a novel solution that enables the implementation of storage controllers to exist in userspace, allowing maximum scalability. OpenEBS, a popular CAS based storage solution, has helped several enterprises run stateful workloads. Originally developed by MayaData, OpenEBS is now a CNCF project with a vibrant community of organizations and individuals alike. This was also evident from CNCF’s 2020 Survey Report that highlighted MayaData (OpenEBS) in the top-5 list of popular storage solutions. Resources: Canonical definition of Container Attached Storage: https://www.cncf.io/blog/2020/09/22/container-attached-storage-is-cloud-native-storage-cas/ To read Adopter use-cases or contribute your own, visit: https://github.com/openebs/openebs/blob/master/ADOPTERS.md. CNCF 2020 Survey Report: https://www.cncf.io/wp-content/uploads/2020/11/CNCF_Survey_Report_2020.pdf OpenEBS LocalPV Quick Start Guide: https://docs.openebs.io/docs/next/localpv.html This article has already been published on https://blog.mayadata.io/container-attached-storage-cas-vs.-shared-storage-which-one-to-choose and has been authorized by MayaData for a republish.
August 10, 2021
by Sudip Sengupta DZone Core CORE
· 4,859 Views · 3 Likes
article thumbnail
Creating a Secure REST API in Node.js
Step by step guide to create a REST API using Node.js and create your first app in Express API.
Updated August 6, 2021
by Michael Smit
· 20,957 Views · 12 Likes
article thumbnail
Helpful Tools for Apache Kafka Developers
Great productivity-boosting tools for Apache Kafka developers, a feature-rich HTTP Kafka client, a command line processor, a Kafka Streams topology visualizer, and more.
August 5, 2021
by Dave Klein
· 7,916 Views · 8 Likes
article thumbnail
JMeter: Setup for Load Testing With Certificate-based Authentication
Steps to do load testing using JMeter on endpoints that required client authentication based on a security certificate. Let's get started!
August 4, 2021
by Ananda Joardar
· 15,649 Views · 1 Like
  • Previous
  • ...
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • ...
  • Next
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook
×