Using Machine Learning to Automatically Identify Important Conversation Topics During Therapy Sessions

2. Post Session Questionnaires (PSQ) for therapists – treatment alliance ruptures were identified after each session by having therapists give feedback (PSQ questionnaire). This questionnaire is a commonly used tool in psychotherapy studies and has been shown to be capable of predicting various situations and processes. 

3. Hopkins Abbreviated Symptom Checklist – the researchers examined the severity of clients' mental symptoms using an 11-item weekly questionnaire.

Data processing: identification, definition, and categorization

Data processing required the integration of research strategies and artificial intelligence, to allow the generation of lists of identifiable conversation topics out of the raw material.

To automatically identify conversation topics, the millions of words in the raw material needed to be divided into semantically meaningful categories. This was done with the aid of artificial intelligence tools that automatically derive categories from the text. One of the significant advantages of this strategy is its ability to reduce textual data into a tabulated list of topics, link therapy sessions to specific topics, and provide a dynamic picture of how these topics change from session to session throughout a course of therapy.

In order to make the data meaningful, a preliminary process was required. This involved omitting both words that are semantically meaningless as well as rare and non-informative words. Since any analysis based on a high rate of rare or special words can interfere with the organization of topics and lead to results that are difficult to interpret semantically, the researchers omitted words that appeared less than 10 times across the database. In addition, common words that were found in more than 90% of the documents were omitted because they were expected to reduce the linguistic distinction of the identified topics. 

The researchers used topic modeling to automatically identify conversation topics. For this purpose, they predefined the number of topics as k and entered a few acceptable k values - (50, 100, 200, 300) after which they examined the distribution results for the conversation topics. In order to find a k value that would produce a relevant list of topics, the researchers examined if the set k value produced too few conversation topics per text, or a large ratio of words attributed to multiple conversation topics across the text, indicating that the set k value was too high. In this manner, the researchers discovered that setting k to 200 yielded the clearest results for analysis.

Thus, the researchers analyzed the documents based on a probability model. This way, the computer model generated a table of 200 conversation topics, in which each topic was represented by a set of words that appeared in the document. The words that appeared under each topic were semantically related and relevant to the content of the conversation and the issues that came up during the session, indicating successful machine learning of the texts.

The next stage after the machine learning-aided processing phase was testing the chosen research hypotheses.

The first research hypothesis sought to find a relationship between the table of conversation topics automatically suggested by the computer and the clients’ functioning levels, as well as therapeutic alliance ruptures. In accordance with this hypothesis, the researchers did find a relationship between changes in conversation topics and functioning levels. They also were able to identify an association between an increase in topics that reflect higher functioning levels (grouped under "positive experience") and a reduction of symptom severity. In addition, they found that a higher number of topics identified with therapeutic alliance ruptures (grouped under "treatment") was associated with higher self-reported symptom severity.

The computer model was able to correctly predict client functioning levels based on conversation topics that came up in 75% of the cases, and predict therapeutic alliance ruptures in 65% of the cases. The study results shed light on concepts and ideas related to client functioning levels. For example, the four topics most frequently associated with low functioning levels are loneliness, suffering, physical difficulties and anger, and are all related to experiences that are perceived as negative. Conversation topics found to be related to high functioning levels, however, are pleasure, leisure, experience, celebration and choice, and are all related to experiences that are perceived as positive. The success of the computer model is also reflected through the four topics that were found to be most closely related to therapeutic alliance ruptures – communication, goal setting, need for help, and problems. This finding corresponds with previous descriptions in the research literature and therapeutic alliance characteristics. We should remember that these conversation topics were found automatically, which again illustrates the computer model's ability to identify significant aspects of the clinical session.

Results - the second research hypothesis

The second research hypothesis sought to examine the manner in which the conversation topics that came up during sessions were related to treatment outcomes. For this end, the researchers compared computer-generated tables of conversation topics to the symptom severity questionnaires clients filled out and the feedback provided by therapists.

The researchers found no direct relationship between specific topics and treatment outcomes, but like the research hypothesis, they did discover that changes in conversation topics were reflected by client-reported symptom severity throughout the treatment process.

More specifically, the findings show that a higher number of conversation topics reflecting higher functioning levels associated with positive experiences was also reflected by reduced symptom severity. Although many classic psychotherapeutic models emphasize the client's ability to cope with negative and painful experiences, the recent years have seen increasing awareness that expanding the client's ability to experience both positive and negative emotions has a therapeutic effect.

From the research laboratory to practical insights: applying the findings to two test cases

Finally, to illustrate the use of conversation topic models by clinicians and researchers, the researchers examined the links between conversation topics and functioning levels and therapeutic alliance ruptures throughout the therapeutic process in one case that recorded good outcomes and another that did not.

To this end, the researchers examined the change in conversation topics throughout the course of treatment. They discovered that in the case of successful treatment, conversation topics that expressed lower functioning levels became less frequent over time, whereas conversation topics that expressed higher functioning levels became more frequent.

However, in the case of unsuccessful treatment, conversation topics that expressed lower functioning levels became more frequent over time, whereas conversation topics that expressed higher functioning levels did not come up at all.

In terms of therapeutic alliance ruptures, in the case of successful treatment, the researchers identified a pattern through which conversation topics identified with therapeutic alliance ruptures came up in the mid-stages of the treatment course but were fewer at the beginning and at the end of the treatment course.

However, in the case of unsuccessful treatment, conversation topics identified with treatment alliance ruptures were evidently present and repeatedly came up during the course of treatment.

The bottom line

The use of advanced machine learning technologies is a relatively new aspect in the field of psychotherapy studies, and recent evidence has indicated their contribution to such research as well as their ability to deepen the understanding of mechanisms that promote positive changes during treatment.

In the present study, the researchers from Bar-Ilan University, led by Prof. Dana Atzil-Slonim, showed that conversation topics that come up during therapy sessions can be automatically identified and can be used to predict treatment gains and outcomes.