The Kiel Intonation Model (KIM),
its Implementation in TTS Synthesis and its Application to the Study of Spontaneous Speech

Klaus J. Kohler, IPdS Kiel, Germany

1. Introduction
2. KIM - The Kiel Intonation Model
• Overview
• Stress
• Intonation
• Phonetic variation
• Microprosody
3. Linguistic environment and symbolic input to the model
4. Implementation of the model in the RULSYS/INFOVOX TTS system for German
5. References
6. Acknowledgement

This document is based on a paper presented at the ATR Workshop on Computational Modeling of Prosody for Spontaneous Speech Processing, Kyoto/Japan, 12-14 April 1995

Abstract

Following on from general considerations of requirements for prosodic modelling, this paper outlines the Kiel Intonation Model for German, incorporating stress and intonation, timing and articulatory reduction. It puts the model into a linguistic environment and sets out the prosodic notation system. This symbolization framework is also the basis for prosodic labelling of speech data. The paper sketches the implementation of the model in the RULSYS/INFOVOX TTS system for German including the generation of spontaneous speech. The categories of the model are demonstrated by audio examples of sentences and texts in KIM-based TTS synthesis. The integrated framework of prosodic research at Kiel is illustrated by an excerpt from a spontaneous dialogue recording, its prosodic label file and its TTS simulation.

1 Introduction

The modelling of prosody has to take the following points into account.

1. Prosodic universals
   The study of prosody has grown out of dealing with individual languages, especially with English, more than with any other language. Categories and operations (e.g. prosodic rules) are - to a large extent - determined by the particular linguistic structures. What we need for a general prosodic theory, however, are independently motivated categories and operations. Candidates are pitch direction (falling, rising) and synchronization of pitch 'peaks' and 'valleys' with syllable timing, in each case independently of the functional use they may be put to in individual languages (e.g. tone or intonation), further, the prominence-lending features of pitch movement and segment duration for the functional use of sentence stress (focus), and timing factors at various levels (global speech rate, utterance-final lengthening, stress/syllable timing).


2. A unified theory integrating segmental and prosodic aspects, as well as intonation and timing among the latter
   We have become used to thinking in and dealing with dichotomies in the study of speech: segmentals versus prosody, intonation versus timing. It is quite clear that all these levels of description form an intricate mutually conditioning network. Prosody (e.g. stress, timing, especially speech rate) provides conditioning factors for articulatory reduction; on the other hand segmental structures determine the manifestation of prosodic categories (different synchronization of pitch 'peaks' and 'valleys' in high/low or short/long vowels, curtailing of falls in pitch 'peaks' before voiceless consonants). Global utterance timing has not only an influence on individual segment durations, but also on their qualitative realization and on the manifestation of pitch patterns (upward scaling of F0 and reduction of F0 range in increased speed). Contrariwise, segmental features determine their timing in global utterance speed: vowel and consonant durations are adjusted differently, not by a uniform proportionate factor across all segments; depending on the segmental type and context, shortening in fast speech also involves assimilation and elision of articulatory gestures over and above their changes in timing.


3. A prosodic phonology as an interlevel between syntax/semantics/pragmatics and the phonetic signal: phonetic substance - phonological form - linguistic function
   The phonetic-semantic relationship is not direct in the sense that the measured values themselves represent syntactic or semantic categories, but the link operates via formal elements that, on the one hand, are related to features of meaning, but are, on the other hand, defined by phonetic ranges. This phonetic substantiation of phonological categories is just as essential as the recognition of structure in phonetic substance. Both phonetic substance and phonetic structure (or signal measures and phonological form) are required for an adequate description of the phonetic-syntactic/semantic relationship, and consequently for prosodic modelling.


4. Integration of prosodic modelling into a linguistic environment
   It follows from point 3 that prosodic modelling requires strong linkswith syntactic, semantic, and, particularly in the case of spontaneous speech, also pragmatic levels.

2 KIM - The Kiel Intonation Model

2.0 Overview

This chapter gives a brief outline of a prosody model which we have been developing at Kiel for German (for further details see Kohler 1991a,b, 1997a,b). It incorporates the following domains:

2.1 Stress

Within stress we have to differentiate between lexical and sentence stress. At the abstract level of phonological specifications in the lexicon, every German word has at least one vowel that has to be marked as potentially stressable, as being able to attract the feature specifications of sentence stress. Lexical stress is thus not a distinctive stress feature, it only marks a position that can attract such a feature at the sentence level, but need not.

Vowels receive combinations of the stress features <+/-FSTRESS> and <+/-DSTRESS> (referring to the association of sentence stress with the two important parameter domains of F0 and duration). In sentence-stressed words the lexically stressed vowel is <+FSTRESS,+DSTRESS>, in unstressed content words <-FSTRESS,+DSTRESS>; in unstressed function words als well as in lexically unstressed syllables that do not get sentence stress the combination is <-FSTRESS,-DSTRESS>. In partially deaccented sentence stresses <+DEACC> is added to the two positive stress features, all other vowels are <-DEACC>. Words that are to get additional emphasis receive the feature <+EMPH> in their lexical stress position, all other vowels <-EMPH>.

Whether <+DSTRESS>, responsible for longer duration, is associated with <+FSTRESS>, marking the vowel as the recipient of intonation features ('peak' and 'valley' contours), or as <-FSTRESS>, not providing the vowel with this potential, depends on the rules of grammar and context of situation in speech communication. They have to be supplied by the linguistic environment of the prosodic phonology (see Section 3). The same applies to the attribution of <+DEACC>.

To distinguish degrees of emphasis, <+EMPH> vowels are given the graded stress level feature <@STRLEV>, with @ = 1,2,...7; <-EMPH> vowels are <0STRLEV>. These vowels are made the more prominent, the higher the stresslevel. In F0 'peak' contours, this greater prominence is achieved by raising the F0 maximum, and if the 'peak' is nonfinal in a 'peak' series, by having a faster descent as well as by lowering the F0 minimum between 'peaks', proportinally to stress level. In the case of F0 'valley' contours, the final F0 point is raised in accordance with stress level. Emphasis is used to put words and phrases within sentences in focus, particularly when the expansion of intonation contours on certain structural elements is coupled with the deaccentuation of others.

In summary, the following sentence stres features are proposed in a prosodic phonology of German for a comprehensive contrastive categorization:
<+/-FSTRESS>
<+/-DSTRESS>
<+/-DEACC>
<+/-EMPH>
<@STRLEV>, with @ = 0, 1, ...7.
The feature pair <+/-EMPH> constitutes the link with the intonationfeatures.

2.2 Intonation

2.2.1 Pitch categories at sentence stresses

All vowels with 'primary' or 'secondary' (= deaccented) sentence stress, i.e. with the feature specification <+FSTRESS,+DSTRESS> receive intonation features, which may be either 'valleys' or 'peaks', specified as <+/-VALLEY>, and in the case of 'peaks' (<-VALLEY>), they may contain a unidirectional F0 fall, classified as <+TERMIN>, or rise again at the end, resulting in a (rise-) fall-rise, categorized as <-TERMIN>. 'Valleys' may have a low rise, to indicate, e.g., continuation, or a high rise, used, e.g., in questions, with the specifications <+/-QUEST>. All 'peaks' and 'valleys' may have their turning points (F0 maximum in 'peaks' or F0 minimum in 'valleys') early or later with reference to the onset of <+VOK,+FSTRESS>, categorized as <+/-EARLY>, and finally, for 'peaks' <-EARLY> may be around the stressed vowel centre or towards its end, classified by the feature opposition <+/-LATE>. The categorization of <-VALLEY> into <+EARLY> and <-EARLY>, with a further subdivision of the latter into <+/-LATE>, captures the grouping of 'late' and 'medial' vs. 'early peaks', as it showed up in perceptual experiments with stepwise 'peak' shift from left to right (Kohler 1990b,1991b).

'Peaks' are characterized by a quick F0 rise confined to the vicinity of a sentence-stressed syllable. This rise precedes the onset of the latter, and is usually narrow in temporal extension, for an 'early peak'; it extends into the first half of the stressed nucleus in the case of a 'medial peak'. In the 'late peak', it starts after the stressed vowel onset and continues into the second half of the nucleus or beyond, the exact timing of the maximum 'peak' value depending on vowel type (duration according to quantity and quality), subsequent voiced/ voiceless consonants and number of immediately following unstressed syllables. There may even be a low stretch of F0 in the stressed vowel before the rise. After the 'peak' maximum is reached the F0 descends immediately, especially on subsequent unstressed syllables. But for chains of 'peaks' see 2.2.2.

'Valleys', on the other hand, have a continuous rise, starting before the stressed-syllable nucleus (early) or inside it (non-early) and extending as far as the beginning of the following sentence-stressed syllable. If there are several unstressed syllables between two sentence stresses a 'valley' is thus realised as a more gradual F0 ascent compared with the much quicker rise for a 'late peak'. The less distance there is between stressed syllables the more difficult it becomes to distinguish between a 'valley' + 'peak' and a 'late peak' + 'peak' sequence, especially if there is no F0 dip in between the first and second stress F0 maxima, as in a hat pattern (see 2.2.2).

In summary, the elements of the intonation component of the prosodic phonology of German are given the following distinctive features for a comprehensive contrastive categorization:
<+/-TERMIN>
<+/-VALLEY>
<+/-QUEST>
<+/-EARLY>
<+/-LATE>
<+/-EMPH>
<@STRLEV>, with @ = 0, 1, ...7.

2.2.2 Pitch category concatenation

In a concatenation of pitch 'peaks' without prosodic boundaries between them (see 2.3.3), F0 may fall to a low or an intermediate level and then rise again for the next 'peak'. This fall will be effected on intervening unstressed syllables between the two 'peaks', reaching the lowest point, to start the next rise, in the vicinity of the following stressed syllable, depending on 'peak' position. If there are no unstresed syllables separating the two 'peaks', the dip can be accommodated between all 'peak' combinations, except for 'late' + 'early'/'medial', where a hat pattern is created; it combines the rise of the 'late peak' and the fall of the 'early peak' in a two-stress sequence.

This boundary case of the absence of an F0 descent between 'peaks' canalso be extended to concatenations with intervening unstressed syllables. In such a hat pattern, an 'early peak' is not possible initially, and a 'late' one is excluded non-initially. If there are more than two stresses incorporated in a hat the non-initial and non-final ones are unspecified as to 'peak' position because they neither have a rise nor a fall but are simply integrated into the downstepped sequence of 'peak' maxima (see 2.3.2). In the categorization of pitch patterns they are nevertheless grouped together with 'peaks'. If in a two-stress rise-fall it is difficult to decide whether the rise represents a 'valley' or a 'late peak' in a hat pattern, the latter solution is chosen.

When prosodic boundaries intervene any sequencing of 'peaks' and/or 'valleys' is possible, but the hat pattern is then excluded since it represents a very high degree of cohesion. On the other hand, a 'late peak' with a full F0 descent marks a dissociation from a following 'peak' and will then normally be linked with a prosodic boundary, i.e. final lengthening and, usually, F0 reset afterwards.

2.2.3 Pre-head

Unstressed syllables preceding the first sentence stress in a prosodic phrase may be either low or high: they represent different types of 'pre-head'.

2.3 Phonetic variation

The acoustic manifestations of the distinctive prosodic categories vary according to segmental and prosodic context, depending on the temporal alignment of 'peaks' and 'valleys' - defined by a small number of significant F0 points - with different syllable types and sequences, on downstepping, speech rate, prosodic boundaries and, finally, articulation-induced microprosody.

2.3.1 Temporal alignment of 'peaks' and 'valleys'

Taking the default, 'medial peak' as a reference, two significant F0 points are defined. The first one, TF0, is positioned at the beginning of the syllable containing the <+FSTRESS> vowel, the second, T2F0, near the vowel centre, the exact timing after voiced vowel onset depending on vowel quantity, vowel height, number of following unstressed syllables and position in the utterance. The calculation of the time point T2F0 after vowel onset is carried out on the basis of the segmental duration rules for German. They have adopted the principle proposed by Klatt (1979) for the rule synthesis of English (see also Kohler 1988), defining different classes of segments (e.g. diphthongs vs. long vs. short vowels, low vs. high vowels) by different pairs of values for intrinsic duration (Di) and for minimal duration (Dmin) and generating actual segment durations in various segmental, prosodic and syntactic contexts by the application of the following rules:

(1) <DUR> ^ <(Di-Dmin)*PRCNT/100+Dmin>

(2) <PRCNT> ^ <PRCNT*PRCNT1/100>.

In (1), PRCNT = 100 initially; the rules then change the PRCNT values successively by introducing a rule-specific PRCNT1 value into (2). This way all the factors influencing segmental durations (tempo, position in the word and sentence, stress, segmental context) can be captured in specific rules by inserting a new PRCNT1 value each time. This model assumes that all the factors affecting duration operate independently of each other and that it is only the amount exceeding the minimal duration of a segment that is adjusted by these factors. The two assumptions provide a good approximation of segment timing in languages like German and English, and certainly result in prosodically acceptable speech synthesis.

T2F0 for 'medial peaks' is now derived from the basic vowel-type related duration. The only percentage factor that enters the calculation is the one referring to speech rate; it is normally set at 100, a speeding up lowers, a slowing down increases the factor, i.e. it is essentially the intrinsic vowel duration that determines the point in time after <+VOK,+FSTRESS> onset (T2F0) where the 'medial peak' is positioned. But this has to be adjusted in the case of aspiration. On the one hand aspiration lengthens the total vowel duration, compared with vowels in non-aspirated contexts, but this increase is not as large as the total aspiration phase; on the other hand it shortens the stop closure duration compared with unaspirated cases, but again not by the total amount. So the larger part of the aspiration (AH) should be added to the vowel, but some of it attached to the plosive, and the F0 'peak' placement has to take this ambivalence into account:

(3) <+VOK,+FSTRESS> ^ <T2F0=((Di-Dmin)*PRCNT/100+Dmin)*0.6+TLAH*0.75>,

i.e. three quarters of the period up to the last aspiration time point are added to T2F0, shifting it further to the right by this amount.

Sentence-final 'medial peaks' receive a third F0 point, T3F0, at 150 ms after the 'peak' maximum in a medium speech rate (see 2.3.4, 2.0(6), 3(6')); in all non-final cases, the maximum 'peak' point of one <+FSTRESS> connects with the left-base point of the next <+FSTRESS>. As the absolute F0 'peak' position is not affected by vowel duration modifications due to voiced/voiceless context, number of syllables in the word, sentence position etc., its relative position changes with vowel shortening or lengthening, moving closer towards or further away from, the end. This way the microprosodic F0 truncation before voiceless obstruents is automatically built into the rules.

Thus, in utterance-final short-vowel monosyllables, ending in voiceless consonants, an F0 'peak' fall is truncated, and has to be so to create the same perceptual 'peak' pattern as in other, non-truncating contexts. The listener obviously takes the underlying constancy in absolute F0 'peak' positioning within the same vowel type, and the difference across different vowel types, into account, disregarding contextual adjustments. He can always calculate the final F0 point that should have been reached, if the F0 contour had not been curtailed, from the F0 decline per unit of time up to the cut-off, and he can then compare this value with the likely low end of the speaker's speech voice range (about 60 - 80 Hz in a male voice, an octave higher in a female one). If the comparison is within a narrow margin the fall is terminal, otherwise it is not. Since the lower end of a speaker's F0 range is a reliable reference value, truncation of falling F0 patterns can be uniquely restored perceptually.

That no longer applies to rising F0. Here the end point is not calculable, because there is a large margin for the end point, anything up to or even above 1.5 octaves is possible. The position of the ceiling is not fixed, by contrast with the base line. That means that the intended high value of a 'valley' always has to be physically reached, it cannot be deduced from what precedes, and, moreover, it does not change with different 'valley' positions from 'early' to 'late', which is quite different in 'peaks'. There is thus a fundamental difference between 'peaks' and 'valleys' in the fixation of their offsets.

An 'early peak' has its maximum value at the <+FSTRESS> syllable onset, TF0 100 ms before, and T3F0 - in sentence-final position - in an area where the 'medial peak' has its maximum. A 'late peak' has TF0 at the same point as a 'medial peak', then an additional low F0 point T2F0 is inserted where the 'medial peak' has its centre, and the 'late' summit (T3F0) occurs 100 ms later or at the end of the last voiced segment in a non-final monosyllabic word if this distance is less than 100 ms. If there is an unstressed syllable following, the summit coincides with the unstressed vowel voice onset. In utterance-final position, a fourth F0 point, T4F0, occurs 100 ms after the summit.

'Valleys' have their left and centre F0 points at the same positions as TF0 and T2F0 in 'medial peaks'. In an 'early valley' the left point is the lowest, whereas in a 'non-early valley' it is the centre pont; in both cases, the right, high point is located at the end of the last voiced segment.

2.3.2 Downstepping

Declination, i.e. the temporally fixed decline of F0, prominent in, e.g., the Dutch and Lund models, is not a feature of spontaneous speech production. It has therefore been replaced by downstepping in KIM, i.e. a structurally determined pitch lowering from sentence stress to sentence stress, independent of the time that elapses between them.

Results of interactive testing make it clear that perception orientates itself at structurally positioned and downstepped 'peaks', not at a time-based declination. The downstepping values used in KIM are 6% from 'peak' maximum to 'peak' maximum, and 18% from a 'peak' maximum to the next base. In 'valleys' both the low and the high F0 value are downstepped by 6%.Downstepping can be interrupted at any point by the feature <+EMPH> or by resetting.

2.3.3 Prosodic boundaries

One of the functions of prosody is the sequential structuring of utterances and discourse, i.e. the signalling or prosodic boundaries and - at least partially - their hierarchical organization. To decode the syntagmatic chunking of messages in accordance with the speaker's intention the listener requires signals that index degrees of cohesion or separation, respectively, between phrases, clauses, utterances and turns. The parameters that achieve this are pause duration, phrase-final segmental lengthening and scaling of F0 end points at the respective boundaries. They can be controlled by parametric rules in the prosodic model upon appropriate symbolic input.

As at this stage the linguistically and phonetically relevant categorization of these boundaries is not well understood the modelling cannot reduce the categories in this domain to the same small number as in the other areas of prosody discussed so far, but has to allow sufficient degrees of freedom for experimentation with data modelling. At each of the three parameters three degrees are therefore recognised, controlled by digit notation in the symbolic input to the model (see Section 3). As our knowledge of prosodic boundary marking increases the degrees of freedom can be reduced by establishing constraints between the three parameters in the signalling of the necessary and sufficient number of phonologically relevant distinctions.

2.3.4 Speech rate

Speech rate changes within the same speaker not only alter the segmental durations (at varying degrees for different segment types, e.g. vowels vs. consonants) but also the positioning of T2F0 within a stressed vowel, moving it farther or less far into it in slower and faster speed, respectively (see 2.3.1). This also implies slower or faster rises and falls, which lower or raise the perceived pitch level. The faster movements in turn mean that there comes a point when the complete F0 excursions can no longer be carried out. Since the 'peak' and 'valley' maxima are the essential target values controlled by the speaker, the levelling of F0 movements in fast speech particularly affects the low values. Finally, faster speech also means greater effort, which may produce a higher level of activation at the vocal folds right away. The correlation of F0 level with speech rate perception has been shown in Kohler (1986b,c,).

The control of speech rate is also coupled with articulatory reduction and elaboration (Kohler 1990a). The speech rate category therefore activates whole blocks of parametric rules that deal with F0 timing, F0 patterning, segment durations and segmental adjustments (coarticulation, reduction, reinforcement, elision). To start with, the model distinguishes four degrees, one of them - reduced - taking reduction phenomena (at an otherwise medium speech rate timing) into accout, as one way of changing speed of articulation. This level will probably have to be subdivided into subcategories according to the degree of formality and spontaneity of speaking, comprising different rule modules for the respective degree. A good deal more research into spontaneous speech is necessary before an adequate categorization can be set up in the model.

2.4 Microprosody

In the generation of intonation, KIM separates two levels

• the defining of phonology-controlled prosodic patterns by a small number of significant F0 points,
• the output of continuous F0 contours influenced by articulation- related modifications (Kohler 1990b).

This dichotomy implies the assumption that the underlying F0 'peak' and 'valley' patterns develop independently and in a very concrete physical and physiological sense in speech production, and are modified microprosodically by output constraints in the vocal apparatus. In particular, we have to distinguish five areas of microprosodic adjustments to the basic significant point patterns discussed so far.

1. In close vowels with 'medial peaks', the summit is raised by a factor of 1.08, compared with all other vowels.
2. An interpolation (cosine) is carried out between the significant F0 points.
3. After the voiceless obstruents, the F0 value at vowel onset is raised by an additive constant of 15 Hz, the increment trailing off to 0 towards T2F0.
4. In voiced plosives all F0 values are lowered by 10 Hz, in other voiced consonants by 5 Hz.
5. F0 is masked in voiceless stretches.

3 Linguistic environment and symbolic input to the model

KIM is integrated into a pragmatic, semantic and syntactic environment. The input into the model are symbolic strings in phonetic notation with additional pragmatic, semantic and syntactic markers. The pragmatic and semantic markers trigger, e.g., the pragmatically or semantically conditioned use of 'peak' and 'valley' types or of sentence focus. Lexical stress position can largely be derived by rule, and syntactic structure rules mark deaccentuation and emphasis in word, phrase, clause and sentence construction. Phrasal accentuations are thus derived from the syntactic component preceding the prosodic model, and are given special symbolizations in the input strings to the model. The following 7 bit ASCII characters are used to represent the prosodic categories of 2.0 (using the corresponding numbering).