Our classification is based on two basic questions.
1: Can the poem be determined grammetrically? For this we use a scheme developed by Donald Wesling:
2: Are these grammetric patterns isochronous? To answer this question we use a scheme from Richard Cureton based on Lerdahls and Jackendoffs "Generative Theory of Tonal Music":
These two questions can be operationalized if we add the "rhythmic phrasing" of poems, i.e.the variation of "time spans" (rubato) and the "prolongation" (legato - portato - staccato) into Wesling's scheme. How this works is shown in the diagram below by the blue brackets:
During our project we have developed software that is able to aline the audio signals and the poem text using different OpenSourceTools (Wavesurfer, CMU Sphinx, PoS-Tagger, ToBI, WEKA) to extract relevant features like syllables, lines or prosodic features based on this alignment:
In the diagram shown above, this extraction is illustrated by the example of a poem by Bas Böttcher. We are also working on solutions for the difficult application of our software to a relatively small data corpus - lyrikline - and for the further development from supervised to unsupervised machine learning. We use a bidirectional recurrent neural network (RNN):