Generating Tweets Using a Recurrent Neural Net (torch-rnn)
Generating Tweets Using a Recurrent Neural Net (torch-rnn)
What would happen if you trained an RNN with all your past Twitter tweets, and then used it to generate new tweets? Let's find out!
Join the DZone community and get the full member experience.Join For Free
Bias comes in a variety of forms, all of them potentially damaging to the efficacy of your ML algorithm. Read how Alegion's Chief Data Scientist discusses the source of most headlines about AI failures here.
Even if you're not actively following recent trends in AI and machine learning, you may have come across articles by a researcher who experiments with training neural nets to generate interesting things such as:
Cooking recipes, including culinary wisdom such as, "Brown salmon in oil. Add creamed meat and another deep mixture.
Recipe titles such as Chocolate Pickle Sauce and Completely Meat Chocoate Pie.
Craft beer names such as La Cat Tas Oo Ma Ale.
So, what's going on here? What's being used is something called a Recurrent Neural Net to generate text in a specific style. It's trained with input data which it analyzes to recognizes patterns in the text, constructing a model of that data. It can then generate new text following the same patterns, sometimes with rather curious and amusing results.
A commonly referred-to article on this topic is by Andrej Karpathy titled The Unreasonable Effectiveness of Recurrent Neural Networks. It's well worth a read to get an understanding of the theory and approach.
There are many RNN implementations that you can download and start training with any input data you can imagine. Here's a few to take a look at:
char-rnn by Andrej Karpathy
textgenrnn, a character RNN python module
And many more.
So, it occurred to me: What would happen if you trained an RNN with all your past Twitter tweets, and then used it to generate new tweets? Let's find out!
Let's try it out with torch-rnn. Rhe following is a summary of these installation steps.
sudo apt-get -y install python2.7-dev
sudo apt-get install libhdf5-dev
git clone https://github.com/torch/distro.git ~/torch --recursive cd ~/torch; bash install-deps; ./install.sh #source new PATH for first time usage in current shell
Now, clone the torch-rnn repo:
git clone https://github.com/jcjohnson/torch-rnn.git
Install torch deps:
luarocks install torch luarocks install nn luarocks install optim luarocks install lua-cjson
git clone https://github.com/deepmind/torch-hdf5 cd torch-hdf5 luarocks make hdf5-0-0.rockspec
pip to install Python deps:
sudo apt-get install python-pip
From inside the torch-rnn dir:
pip install -r requirements.txt
Now, do the following steps from docs to preprocess your text input:
python scripts/preprocess.py \ --input_txt my_data.txt \ --output_h5 my_data.h5 \ --output_json my_data.json
For my input tweet text, this looks like:
python scripts/preprocess.py \ --input_txt ~/tweet-text/tweet-text.txt \ --output_h5 ~/tweet-text/tweet-text.h5 \ --output_json ~/tweet-text/tweet-text.json
This gives me:
Total vocabulary size: 182 Total tokens in file: 313709 Training size: 250969 Val size: 31370 Test size: 31370
Now, to train the model:
th train.lua \ -input_h5 my_data.h5 -input_json my_data.json
For my input file containing my tweet text this looks like:
th train.lua -input_h5 ~/tweet-text/tweet-text.h5 -input_json ~/tweet-text/tweet-text.json
This gave me this error:
init.lua:389: module 'cutorch' not found:No LuaRocks module found for cutorch no field package.preload['cutorch']
Trying to manually install
cutorch, I got errors about Cuda toolkit:
CMake Error at /usr/share/cmake-3.5/Modules/FindCUDA.cmake:617 (message): Specify CUDA_TOOLKIT_ROOT_DIR
Checking the docs:
By default this will run in GPU mode using CUDA; to run in CPU-only mode, add the flag -gpu -1
-gpu -1 and trying again, now, I've got this output as it runs:
Epoch 1.44 / 50, i = 44 / 5000, loss = 3.493316
...one line every few seconds.
After some time, it completes a run, and you'll find files like this in your cv dir beneath where you ran the previous script:
checkpoint_1000.json checkpoint_1000.t7 checkpoint_2000.json checkpoint_2000.t7 checkpoint_3000.json checkpoint_3000.t7 checkpoint_4000.json checkpoint_4000.t7 checkpoint_5000.json checkpoint_5000.t7
Now, to run and get some generated text:
th sample.lua -checkpoint cv/checkpoint_5000.t7 -length 500 -gpu -1 -temperature 0.4
Breaking this down:
-checkpoint: As the model training runs, it saves these point in time snapshots of the model. You can run the generation against any of these files, but it seems that the last file it generates gives you the best results.
-length: How many characters to generate from the model.
-gpu -1: Turn off the GPU usage.
-temperature: Ranges from 0.1 to 1 and with values closest to zero, the generation is less creative. The closer to 1, the generated output is, let's say, more creative.
Let's run a couple of example. Let's do 140 chars at
The programming to softting the some the programming to something the computer the computer the computer to a computer the com
And now, let's crank it up to
z&loDOps be sumpriting sor's a porriquilefore AR2 vanerone as dathing 201lus: It's buct. Z) https://t.co/gEDr9Er24N Amatere. PEs'me tha
Now, we've some pretty random stuff including a randomly generated shortened URL, too.
Using a value towards the middle, like 0.4 to 0.5 gets some reasonably interesting results that are not too random, but somewhat similar to my typical tweet style. What's interesting is my regular retweets of software development quotes from @CodeWisdom have heavily influenced the model, so based on my 3,000+ tweets, it generates text like:
RT @CodeWisdom followed by random generated stuff
Given that the following text is clearly not content from @CodeWisdom, it wouldn't be appropriate to use this text as-is and post it as a new tweet. Since I'm looking to take this text and use it as input for an automated Twitter-bot, as interesting as this generated pattern is in that it does look like the majority of my tweets, I've filtered out anything that starts with RT @text.
I've already implemented a first attempt at a Twitter bot using this content with an AWS Lambda running on a timed schedule, you can check it out here:
I'll be following up with some additional posts on the implementation of my AWS Lambda soon.
Published at DZone with permission of Kevin Hooke , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.