Transformer Language Models Should Absolutely Not Quit Their Day Jobs to Pursue Musical Passion Full Time
Because trust me, they won't be able to make a living.
Join the DZone community and get the full member experience.Join For Free
Photo credit by Unsplash/Owen Beard
Automation may be on the verge of making quite a few jobs obsolete, but if you write creatively for a living, trust me, you have nothing to worry about.
When a pair of researchers at Google sought to create a machine that could convincingly create its own song lyrics, they turned to two separate machine-learning models – Transformer Language Models (TLMs) to be exact.
While one TLM was “trained to capture the structure of lyrics” using a “large set of lyrics spanning multiple genres and decades,” the other was “trained to provide a richer vocabulary” than that available in the first dataset, which (obviously) used the text from 20 popular 19th and early 20th century books. The machine then put it all together by inserting these words into the lyrical structures used in its training.
As this blog post from Google explains, TLMs are “novel neural network architecture[s] based on a self-attention mechanism.” Not only does this mechanism make them “particularly well suited for language understanding,” but it also “requires less computation to train and is a much better fit for modern machine learning hardware, speeding up training by up to an order of magnitude.”
It’s also the same type of model used for OpenAI’s GPT-2, which was so good at generating human-sounding language that the organization actually withheld its release for fear of potential misuse. Google’s team, however, does not have to share these misgivings.
Here are some sample lyrics for you to
i'm your big and brave and handsome romeo
you know my secret secret
you have my second estate
you suit your high origin
you have my cursed youth
you have my life
come on, uh
you remember the voice of the widow
i love the girl of the age
i have a regard for the whole
i have no doubt of the kind
i am sitting in the corner of the mantelpiece
you told me you loved nobody else, you never would
you told me you wanted everything else, you never would
he put his hand on the pillow of the marquis
he put his cap on the ground like a stone
he put his hand on the latch of a door
he put his key in the lock as a key
Using a key as a key, huh? Perhaps the fact that my brain is attempting to decipher some deeper meaning here suggests that these researchers may be on to something. Or probably more realistically, this is just what happens to the human brain after spending seven years completing a master’s degree in English lit. Everything has a hidden meaning that must be extracted at all costs. Everything.
While the researchers themselves were encouraged by the improvement they had made over a previous attempt using a less sophisticated Recurrent Neural Network, as you can tell, they still have a long way to go.
TLMs may be great at literal language processing, but creative language is clearly not their forte. At least not yet anyway.
Do you have a favorite piece of AI-related fun? We’d love to hear about it in the comments, and who knows, it may even end up as the subject of a future article. We highly recommend AI Weirdness.
Opinions expressed by DZone contributors are their own.