DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

SBOMs are essential to circumventing software supply chain attacks, and they provide visibility into various software components.

Related

  • Clustered Quartz Scheduler With Spring Boot and MongoDB
  • How to Marry MDC With Spring Integration
  • How Spring and Hibernate Simplify Web and Database Management
  • Functional Endpoints: Alternative to Controllers in WebFlux

Trending

  • How to Embed SAP Analytics Cloud (SAC) Stories Into Fiori Launchpad for Real-Time Insights
  • Top Tools for Front-End Developers
  • Threat Modeling for Developers: Identifying Security Risks in Software Projects
  • Server-Driven UI: Agile Interfaces Without App Releases
  1. DZone
  2. Coding
  3. Frameworks
  4. Camel Clustered File Ingestion With JDBC and Spring

Camel Clustered File Ingestion With JDBC and Spring

Reading files with Apache Camel is not so easy when you need to deploy your route to multiple servers. Thankfully, Camel has the concept of an idempotent consumer.

By 
Mary Cochran user avatar
Mary Cochran
·
Sep. 10, 17 · Tutorial
Likes (2)
Comment
Save
Tweet
Share
6.6K Views

Join the DZone community and get the full member experience.

Join For Free

Reading a file is a common use for Apache Camel. From using a file to kick off a larger route to simply needing to store the file content, the ability to only read a file once is important. This is easy when you have a single server with your route deployed, but what about when you deploy your route to multiple servers? Thankfully, Camel has the concept of an idempotent consumer.

A useful way to implement this is with an idempotent repository. This repository will keep track of the file being read and not allow another server to read it. It works as a lock on the file. After the file reading has been completed, there are options to either remove the row from the database (which would allow for files with the same name to be ingested later) or to keep the row there (and not allow for files of the same name to be later ingested).

So how do you set up an idempotent repository using JDBC and Spring?

First, you will need to create the table and necessary Spring beans. Here, I am using the JDBCMessageIdRepository. At the time of development, this bug was still an issue, so I could not use the JPA version. This has since been fixed and you can easily swap one out for the other. This example assumes you already have a bean for your datasource setup. You can use the same fileConsumerRepo bean for multiple routes in your deployment.

CREATE TABLE CAMEL_MESSAGEPROCESSED ( processorName VARCHAR(255), messageId VARCHAR(100), createdAt TIMESTAMP );
@Bean
public JdbcMessageIdRepository fileConsumerRepo() {
   JdbcMessageIdRepository fileConsumerRepo = null;
   try {
        fileConsumerRepo = new JdbcMessageIdRepository(dataSource(), "fileConsumerRepo");
    } catch (Exception ex) {
        LOGGER.info("############ Caught exception inside Creating fileConsumerRepo ..." + ex.getMessage());
    }
    if (fileConsumerRepo == null) {
        LOGGER.info("############ fileConsumerRepo == null ...");
    }
    return fileConsumerRepo;
}

In addition, make sure to include org.apache.camel.processor.idempotent.jpa in your packages to scan or persistence unit depending on your exact setup. In my case, I was using a LocalContainerEntityManagerFactoryBean for JPA.

@Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory() throws NamingException {
    final LocalContainerEntityManagerFactoryBean em = new LocalContainerEntityManagerFactoryBean();
    em.setPersistenceUnitName("camel");
    final HibernateJpaVendorAdapter vendorAdapter = new HibernateJpaVendorAdapter();
    vendorAdapter.setDatabase(Database.ORACLE);
    vendorAdapter.setShowSql(false);
    vendorAdapter.setGenerateDdl(false);
    em.setJpaVendorAdapter(vendorAdapter);
    em.setPackagesToScan(new String[] { "com.my.package.model","org.apache.camel.processor.idempotent.jpa" });
    em.setJpaProperties(additionalProperties());
    em.setDataSource(this.dataSource());
    return em;
}

Now, your idempotent repository is ready for a Camel to use. Below is an example:

from("file:/my/test/dir?moveFailed=.error&autoCreate=true&readLockLoggingLevel=WARN&shuffle=true&readLock=idempotent&idempotentRepository=#fileConsumerRepo&readLockRemoveOnCommit=true")
    .routeId("ingestionFile")
    .convertBodyTo(String.class)
    .log(LoggingLevel.INFO, "File received");

Let's break down the options:

Image title

Now deploy your route to multiple servers and watch the magic! You can do this easily using EAP or Karaf by using a port offset. If you are looking for more examples, check out this sample project put together by fellow Red Hatter Josh Reagan. And whether you are new to containers or have experience, downloading this cheat sheet can assist you when encountering tasks you haven't done lately.

Spring Framework clustered

Published at DZone with permission of Mary Cochran, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Clustered Quartz Scheduler With Spring Boot and MongoDB
  • How to Marry MDC With Spring Integration
  • How Spring and Hibernate Simplify Web and Database Management
  • Functional Endpoints: Alternative to Controllers in WebFlux

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: