Node-RED Hosting in the Cloud With a Twitter Feed Analyzer

due to applications and components in the global iot ecosystem getting more and more interrelated, it becomes essential to be capable of wiring them together to achieve automation. advanced connectivity of systems allows us to eliminate repetitive programming tasks and therefore free time for research and development.

one of the tools to make different systems interact with each other is node-red – a simple programming tool for binding online services, hardware devices, and apis in an interactive and visually appealing fashion. it provides a browser-based editor and a wide set of nodes to build and deploy various functional flows.

within this article, we’ll take a look at a solution we’ve implemented for a quick node-red delivery and will deploy a sample flow of a twitter streaming data analyzer to store and alert upon posts on any theme you are interested in.

a simple way to run node-red instance within a cloud

at jelastic, the ready-to-go node-red development environment is provided as a single-click installation package. it is available within the jelastic jps collection on github alongside with other preconfigured solutions and could be installed on any platform of 4.9.5 jelastic version and higher.

all the required resources and settings for running node-red on jelastic are declared within the package manifest.jps file, where the main points to consider are:

• nodered/node-red-docker : docker image with the latest tag is used as a solution base
• the created node-red instance is assigned 16 dynamic cloudlets , which enable it to scale vertically up to 2 gib of ram and 6.4 ghz of cpu.
• the automatically attached endpoint allows to access node-red the admin panel through a randomly allocated port via shared load balancer (eliminating the necessity to purchase a public ip address)

so, let’s bring it to life – log into your jelastic cloud account and proceed with the following steps.

within the main dashboard page, click import at the top and switch to the url tab. here, insert link to the corresponding manifest.jps file:

https://github.com/jelastic-jps/node-red/blob/master/manifest.jps

click import to continue.

in the opened installation dialog box:

• type environment as the name to be used as its internal hostname.
• optionally, specify environment display name (so-called alias , to be shown within the dashboard and ssh terminal)
• select an environment region (if multiple ones are available)

click install and wait a bit to be shown a notification message on the process completion. click open in browser to access the node-red development environment and start creating your first project (so-called flow) .

we’ll learn how to interact with this wizard through a hands-on example of a twitter content tracker – building such a useful and practical tool will also help you to get started with node-red as quick as possible.

deploying twitter feed analyzer as a node-red flow

so, the application we are going to build is intended to search through twitter feeds based upon a specified criteria (keyword, hashtag, and/or username), conduct sentiment analysis of observed posts, and process them accordingly. depending on the message's tone, action should be:

• upon detecting negative content: alert via email message.
• for positive and neutral results: store a mention within a dedicated file inside a container for each type of characteristic separately

for analysis, we’ll use #cloud , #cloudcomputing , and #paas hashtags as a base for the corresponding hosting area investigation. obviously, you are free to specify any other search parameters depending on your requirements – e.g. your company name.

the whole application structure will look like below: here, the following component types (nodes) are involved:

• twitter – used to authenticate your twitter account and specify search keywords.
• sentiment – designed to analyze posts and return a corresponding value (negative, positive or neutral).
• switch – intended to evaluate sentiment scores and take a corresponding action according to the defined tone.
• two file nodes – provided to store positive and negative messages.
• email – allows receiving instant alert messages with negative feedback.

altogether, such a flow enables you to quickly react to negative feedback and, simultaneously, to learn about user experience, best practices, news, and events within neutral and positive posts. this way, building such a tool will help you to get relevant information in time and stay tuned to what’s happening in the market.

so, below, we’ll consider the parameters that should be adjusted for each of these node types and consequently implement the above workflow.

twitter node

within the node-red flow editor, expand the social section and drag and drop the twitter icon to place it on the sheet. this will be the very base of our application. double-click the picked node and specify the following information in the opened dialog box:

• twitter id – click on the pencil icon to authorize the app with your twitter account login and password ; after this operation successfully passes, select add to finish
• search – choose an area for conducting search, for example, all public tweets
• for – specify a search term (keyword, hashtag, and/or username); multiple entries сould be separated with spaces (for the bot to look for their joint mention) or commas without spaces (to detect separate terms)
• name – optionally, give some custom name to the node (or leave the default one) select done above to proceed.

tip: for more comprehensive feedback analysis, append an additional debug node to the main twitter one so that fetched posts can be reviewed directly through the editor. as a result, records of all semantic types will be available within the sidebar debug tab, being stored within its default msg.payload log.

sentiment node

now, expand the analysis section and choose a sentiment node – it will analyze found posts and return a corresponding sentiment value to determine whether a tweet is negative, positive, or neutral. this component requires no additional adjustments unless you’d like to change its name (otherwise, double-click this block, specify the preferred denomination, and click done ).

switch node

now let’s add another switch element from the function section. it is intended to evaluate the received sentiment score and take a corresponding action according to the defined tone.

tip: the average sentiment score ranges from -5 (which means a very negative tweet) to +5 (a very positive feedback), with 0 being a neutral value. scores outside this range are considered as extreme values.

to adjust the switch node configs, double-click it and provide the following details in the opened dialog box:

• name – pick a custom node name (or leave the default one)
• property – specify the msg.sentiment.score value and +add the following rules to determine which sentimental value should be considered as neutral , positive, or negative respectively:
  • == 0
  • > 0
  • < 0

select checking all rules at the bottom and click done .

file nodes

now, we are going to set up logging of the observed neutral and complimentary messages. but before the corresponding node insertion, you need to create files for such mentions to be stored in – thus, open the config manager for your node-red container and enter the desired destination directory.

here, select new file within options menu, type a name for it (e.g. neutral ) and press enter . repeat these actions to create another file for positive tweets.

note: your application should have sufficient permissions to access and write to the created files. to ensure this, connect to your node-red container via ssh gate as follows:

• generate an ssh keypair.
• add your public ssh key to the dashboard.
• access your account via ssh protocol.

once the connection is established, run the chown node-red {filename} command within the destination directory.

then, expand storage wizard section and drag the comprised file node to the sheet. double-click this block to adjust the appropriate parameters:

• filename – specify a full path to a destination storing file.
• action – select append to file to add new tweets to the end of a file. optionally, tick  the following additional options are available for this settings:
  • add newline (\n) to each payload? – for a new line to be added to every message (can be turned off if required)
  • create directory if it doesn’t exist? – for a new directory to be created if it doesn’t exist
• name – enter the index number of the required switch node condition so that tweets with the corresponding semantic value will be written to this file (e.g. 1 for neutral posts in this example)

click done to confirm. repeat these operations to configure one more file node that will process and save positive tweets.

email node

the last required email component could be located under the social section. place it within the work area and double-click to provide the following details:

• to – type an email address the alerts on negative tweets should be sent to
• server & port – leave the default values ( smtp.gmail.com host and 465 port number correspondingly)
• userid & password – provide credentials to an email account from which the notifications will be sent
• name – type the index number of the corresponding switch node condition according to the post type you’d like to receive the alerts on (in our case, semantic range for negative tweets was defined third)

note: to ensure that the email node will definitely work properly, your email account (i.e. the one alerts will be sent from) should meet the following requirements:

• have the access to less secure apps setting turned on
• being protected with a strong 8-character password (containing uppercase/lowercase letters, numbers, and symbols)

when you are finished, click done .

note: due to inherent twitter api limits, you could be shown the rate limit hit warning, appearing under the twitter node, if changing nodes settings and/or clicking deploy too often. this does not affect the actual flow operability – just wait a minute for this warning to be replaced with your search keyword(s) instead.

checking results

so, your self-made twitter bot has started to work, tracking the current posts and processing the information you’ve asked about. subsequently, results of this analysis could be checked as follows:

• for neutral/positive mentions – click config for the node-red instance within your jelastic dashboard and navigate to the appropriate file; its content can be reviewed just through the embedded file manager:
  
• for negative tweets – just track new incomings within your email box

that’s it! the node-red flow is now up and running to analyze streaming data from twitter.