Predicting patient drug experience with deep learning

Reading the sentiment of 185,000 online medication reviews with recurrent neural networks, and finding a data leak in the research we built on.

For our course Advanced Machine Learning for Public Policy, my team built models that classify patient reviews of medications as positive, neutral or negative. Reviews like these could help doctors spot strong reactions to a drug across thousands of patients. We started from the Drugs.com dataset published by Grasser et al. (2018), 215,063 reviews from 2008 to 2017, and wrote a scraper to extend it through May 2021, adding 56,520 new reviews.

While cleaning the data, we found that 86,585 reviews, about 40% of the original dataset, were duplicates: for example, the same review listed under both Seroquel and its generic, quetiapine. After a normal train/test split, 60% of the test set also appeared in training. A logistic regression barely noticed (0.782 accuracy with the duplicates, 0.775 without, Figures 2 and 3), but an LSTM fell from 0.90 to 0.80 once they were removed, suggesting that deep learning results reported on this dataset may be inflated.

On the cleaned and extended data, 184,998 reviews, we trained bidirectional LSTMs on GloVe word embeddings, then a mixed-data model that reads each review through two embeddings (GloVe and one trained on tweets about drugs) alongside 1,016 features from its metadata, about 25 million trainable parameters in all. Both reached about 0.80 accuracy. The hard part was the middle: the final model read 91% of positive and 81% of negative reviews correctly, but only 27% of neutral ones, which make up 12% of the data and often read like praise or complaint.

Built with Angelo Cozzubo and Noah Selman. Read the final paper, or the code on GitHub.