DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • How To Use GitLab for Simultaneous Execution of Jobs (Part 2)
  • Building a REST Service That Collects HTML Form Data Using Netbeans, Jersey, Apache Tomcat, and Java
  • Spring Boot - How To Use Native SQL Queries | Restful Web Services
  • Testing REST Controller Methods With JUnit 5 [Video]

Trending

  • JWT Token Revocation: Centralized Control vs. Distributed Kafka Handling
  • Essential Complexity Is the Developer's Unique Selling Point
  • Unleashing the Power of Microservices With Spring Cloud
  • Debugging Tips and Tricks: A Comprehensive Guide
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. DevOps and CI/CD
  4. How to Schedule Pipeline Execution Using the SDC REST APIs

How to Schedule Pipeline Execution Using the SDC REST APIs

This tutorial seeks to clarify how to schedule Streamsets Data Collector pipeline execution with the SDC REST API method.

Guglielmo Iozzia user avatar by
Guglielmo Iozzia
·
Jul. 31, 17 · Tutorial
Like (4)
Save
Tweet
Share
11.63K Views

Join the DZone community and get the full member experience.

Join For Free

A hot topic in the sdc-user group during the past weeks has been about how to schedule the start and stop of SDC pipelines. Usage of the SDC REST APIs has been suggested in some threads, but because the general impression I have is that the audience doesn't have a clear idea about them, I decided to write this post to help and clarify once and for all how to do it.

Streamsets Data Collector REST APIs

SDC provides REST APIs, which allow you to do a lot of things. The full list of APIs and detailed info on how to invoke them can be accessed from the SDC dashboard, by clicking first on the help icon and then on the RESTful API link.

Image title

They are grouped in six different categories:

  •  acl

  •  definitions

  •  manager

  •  preview

  •  store

  •  system

The manager section contains the API to start and stop a pipeline:

Image title

Image title

In order to start a pipeline using the specific REST API, you need to make a POST request the following way (in this case we are using cUrl for this purpose):

curl -u <username>:<password> -X POST https://<sdc_host>:<port:>/rest/v1/pipeline/<pipeline_id>/start -H "X-Requested-By:sdc"

Same way to stop a pipeline:

curl -u <username>:<password> -X POST https://<sdc_host>:<port:>/rest/v1/pipeline/<pipeline_id>/stop -H "X-Requested-By:sdc"

You have to specify the credentials of a user which has permission to start or stop a pipeline. The X-Requested-By header attribute is mandatory for POST requests to these REST APIs.

Scheduling

Now that we know what are the commands to trigger a pipeline start or stop it couldn't be hard to set a schedule for them. Save the commands above to scripts and then prepare a Cron expression for them to be used with crontab. This is the general syntax for a Cron expression:

* * * * * command or script to be executed

- - - - -

| | | | |

| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)

| | | ------- Month (1 - 12)

| | --------- Day of month (1 - 31)

| ----------- Hour (0 - 23)

------------- Minute (0 - 59)

Examples (to start a pipeline every day at 00:15 and stop it at 00:30):

15 0 * * * /home/guglielmo/scripts/start_pipeline.sh

30 0 * * * /home/guglielmo/scripts/stop_pipeline.sh

Of course, you could reuse the same expressions with any other scheduler (like Rundeck) you should have in-house.

REST Web Protocols Pipeline (software) Schedule (computer science) Execution (computing)

Opinions expressed by DZone contributors are their own.

Related

  • How To Use GitLab for Simultaneous Execution of Jobs (Part 2)
  • Building a REST Service That Collects HTML Form Data Using Netbeans, Jersey, Apache Tomcat, and Java
  • Spring Boot - How To Use Native SQL Queries | Restful Web Services
  • Testing REST Controller Methods With JUnit 5 [Video]

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: