Learning Objects

This tutorial aims to introduce basic ways to preprocess data before we model data using R. We will cover:

1. How to read, clean, and transform text data

2. How to preprocess data such as tokenization, removing stop words, lemmatization, stemming, and representing words in R

3. How to get basic statistics from texts using lexicon methods

In previous tutorial, we have covered some basic things about R and how to do simple statistical computing as well as how to use selenium to do webscraping.

You can download the RMarkdown file from the link: https://yongjunzhang.com/files/soc201/Lab12.Rmd; You can modify the codes for your own analysis in your RStudio.

Basic Intro to R and Preprossing Textual Data with R

if (!requireNamespace("pacman"))
  install.packages('pacman')

## Loading required namespace: pacman

library(pacman)
packages<-c("tidyverse","tidytext","stringr","sentimentr","syuzhet",
            "tm","stopwords")
p_load(packages,character.only = TRUE)

Query Twitter Data

In week 10, we talked about how you can retrieve data from twitter, so let us copy and paste those codes. We are going to get some twitter data to run a demo today.

# We need twitterR package to access and query data from twitter
library(twitteR)

## 
## Attaching package: 'twitteR'

## The following objects are masked from 'package:dplyr':
## 
##     id, location

consumer_key <- "your_consumer_key" # replace with your own
consumer_secret <- "your_consumer_secret"
access_key <- "your_access_token"
access_secret <- "your_access_secret"
source("twitter.R") # I store my authentification info in a local R file
setup_twitter_oauth(consumer_key, consumer_secret, access_key, access_secret)

## [1] "Using direct authentication"

# let use searchTwitter function to search covid19 
# here is the documentation of this function <https://www.rdocumentation.org/packages/twitteR/versions/1.1.9/topics/searchTwitter>
# we store the data as covid_twitter
twitter <- searchTwitter("#ChinaVirus",n=1000,lang = "en")

## Warning in doRppAPICall("search/tweets", n, params = params, retryOnRateLimit =
## retryOnRateLimit, : 1000 tweets were requested but the API can only return 544

# we convert is to a data frame using twListToDF
twitter_df <- twListToDF(twitter)
# let us see the data
knitr::kable(twitter_df[1:5,])

CDC / WHO data from around the world shows the IFR of #ChinaVirus a.k.a. #WuhanFl… |FALSE | 0|NA |2021-11-12 13:22:43 |FALSE |NA |1459149659968983040 |NA |Twitter for Android |dreamr6936 | 33|TRUE |FALSE |NA |NA | |RT @dr_SDRK: @thebossoriginal @MVZexplorer @cselston @NancyMAGAmonkey @kimmarinesis @OneGeorgiaPeach @RhondaFurin

Here is the Master Mind… |FALSE | 0|NA |2021-11-12 12:38:42 |FALSE |NA |1459138582916915207 |NA |Twitter for iPhone |RhondaFurin | 2|TRUE |FALSE |NA |NA | |RT @dr_SDRK: @thebossoriginal @MVZexplorer @cselston @NancyMAGAmonkey @kimmarinesis @OneGeorgiaPeach @RhondaFurin

Here is the Master Mind… |FALSE | 0|NA |2021-11-12 12:14:45 |FALSE |NA |1459132555567894529 |NA |Twitter for iPhone |dr_SDRK | 2|TRUE |FALSE |NA |NA |

write_csv(twitter_df,"twitter_df.csv")

We are able to get 1000 tweets. If you cannot set up your twitter developer accounts, you can just download my saved csv file to play with. You can use the following chunk of codes to read the data from my websites.

twitter_df <- read_csv(url("https://yongjunzhang.com/files/soc201/twitter_df.csv"))

If you are analyzing your own text data, you are use read_csv functions to read .csv files. There are others functions to read different datasets.

R tidyverse package provides a series of useful data wrangling tools. You can check it here https://www.tidyverse.org/.

# if you can use ?mutate() to see the details of the function, basically it creates a new variable for you
# you may also want to get rid of all other messy stuff except numbers and words
data <- twitter_df %>% 
  mutate(text=str_replace_all(text,"[^a-zA-Z0-9#@]"," ") %>% 
           str_squish) %>% 
  mutate(
    uid=row_number(),
    text=str_replace_all(tolower(text),"(https)|(http).*? ","") %>% 
           str_squish
    )

# Show a table for visual check
knitr::kable(data[1:3,],cap="Twitter Data on #ChinaVirus")

library(tidytext)

# Let us say we are interested in text description. We need to restructure it as one-token-per-row format. The unnest_tokens function is a way to convert a dataframe with a text column to be one-token-per-row:

tidy_data <- data %>%
  # let us only keep unique id and text
  select(uid,text) %>% 
  # one token per row. This function uses the tokenizers package to separate each line into words. The default tokenizing is for words, but other options include characters, ngrams, sentences, lines, paragraphs, or separation around a regex pattern.
  unnest_tokens(word, text) %>% 
  # remove stop words
  anti_join(tidytext::get_stopwords("en",source="snowball")) %>% 
  # you can also add your own stop words if you want
  # check here to see tibble data structure <https://tibble.tidyverse.org/>
  anti_join(tibble(word=c("co","t","rt","w")),by="word") %>% 
  # let us stem words
  mutate(word=SnowballC::wordStem(word))

## Joining, by = "word"

knitr::kable(tidy_data[1:10,],cap="#Twitter Data for ChinaVirus")

head(tidy_data %>% 
  count(word, sort = TRUE))

##        word   n
## 1 chinaviru 338
## 2    hacker 144
## 3       amp 125
## 4     world 106
## 5    scienc  97
## 6      viru  89

# define a nice color palette
#install.packages("wordcloud")
library(wordcloud)

## Loading required package: RColorBrewer

library(wordcloud2)
pal <- brewer.pal(8,"Dark2")
# plot the 50 most common words
tidy_data %>% 
  count(word, sort = TRUE) %>% 
  wordcloud2(color=pal,size=.7)

library(ggplot2)

tidy_data %>%
  count(word, sort = TRUE) %>%
  filter(n > 100) %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n)) +
  geom_col() +
  xlab(NULL) +
  coord_flip()

 data %>%
  select(uid,text) %>%
  unnest_tokens(bigram, text,token = "ngrams", n = 2) %>%
  count(bigram,sort = TRUE) %>% 
  mutate(bigram= reorder(bigram, n)) %>%
  filter(n>50) %>% 
  ggplot(aes(bigram, n)) +
  geom_col() +
  xlab(NULL) +
  coord_flip()

Structural Topic Model

Let us say, you are interested in the potential themes in those tweets. In other words, what are those tweets talking about.

In this part we heavily rely on stm’s tutorial by Molly Roberts, Brandon Stewart and Dustin Tingley. We will go through the tutorial and show you how to do stm in R librabry stm.

The basic idea of topic models is to assume that a document is a distribution of topics and a topic is a distribution of words. In thise sense, a tweet is composed of certain topics and a topic is composed of certain words. We are trying to figure out potential topics in these tweets.

Let us install stm first.

#library(devtools)
#install_github("bstewart/stm",dependencies=TRUE)
library(stm)

Before we run topic models, we need to preprocess data. STM provides several functions to automatically do stemming, stopwords removal, low frequency words removal, etc for you.

Let us use the textProcessor to preprocess texts. Here is the function:

textProcessor(documents, metadata = NULL, lowercase = TRUE, removestopwords = TRUE, removenumbers = TRUE, removepunctuation = TRUE, ucp = FALSE, stem = TRUE, wordLengths = c(3, Inf), sparselevel = 1, language = “en”, verbose = TRUE, onlycharacter = FALSE, striphtml = FALSE, customstopwords = NULL, custompunctuation = NULL, v1 = FALSE)

#Preprocessing
#stemming/stopword removal, etc.
#Josh-cc, if you don't know the details of a function, you can use ? to check the documentation of that function. ?textProcessor
processed <- textProcessor(data$text, metadata=data)

## Building corpus... 
## Converting to Lower Case... 
## Removing punctuation... 
## Removing stopwords... 
## Removing numbers... 
## Stemming... 
## Creating Output...

Let us use prepDocuments to perform several corpus manipulations including removing words and renumbering word indices. here is the function:

prepDocuments(documents, vocab, meta = NULL, lower.thresh = 1, upper.thresh = Inf, subsample = NULL, verbose = TRUE)

#structure and index for usage in the stm model. Verify no-missingness. can remove low frequency words using 'lower.thresh' option. 
#See ?prepDocuments for more info
out <- prepDocuments(processed$documents, processed$vocab, processed$meta, lower.thresh=1)

## Removing 1038 of 1625 terms (1038 of 5882 tokens) due to frequency 
## Removing 2 Documents with No Words 
## Your corpus now has 542 documents, 587 terms and 4844 tokens.

#output will have object meta, documents, and vocab 
docs <- out$documents
vocab <- out$vocab
meta <-out$meta

Now, let us use stm function fit a stm model.

The function takes sparse representation of a document-term matrix, an integer number of topics, and covariates and returns fitted model parameters. Covariates can be used in the prior for topic prevalence, in the prior for topical content or both.

stm(documents, vocab, K, prevalence = NULL, content = NULL, data = NULL, init.type = c(“Spectral”, “LDA”, “Random”, “Custom”), seed = NULL, max.em.its = 500, emtol = 1e-05, verbose = TRUE, reportevery = 5, LDAbeta = TRUE, interactions = TRUE, ngroups = 1, model = NULL, gamma.prior = c(“Pooled”, “L1”), sigma.prior = 0, kappa.prior = c(“L1”, “Jeffreys”), control = list())

#run an stm model using the 'out' data. 20 topics. Asking how prevalaence of topics varies across documents' meta data, including 'rating' and day. !! option s(day) applies a spline normalization to day variable.

# max.em.its should be at least 100. We use 15 just as demo
pFit <- stm(out$documents,out$vocab,K=20, max.em.its=15, data=out$meta,seed=2020)

## Beginning Spectral Initialization 
##   Calculating the gram matrix...
##   Finding anchor words...
##      ....................
##   Recovering initialization...
##      .....
## Initialization complete.
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 1 (approx. per word bound = -5.208) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 2 (approx. per word bound = -4.238, relative change = 1.863e-01) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 3 (approx. per word bound = -4.034, relative change = 4.819e-02) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 4 (approx. per word bound = -3.970, relative change = 1.567e-02) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 5 (approx. per word bound = -3.935, relative change = 8.912e-03) 
## Topic 1: harsha, kakar, happen, know, everyth 
##  Topic 2: chines, mani, sleep, fake, troll 
##  Topic 3: just, can, chinavirus, mask, say 
##  Topic 4: death, chinavirus, amp, china, must 
##  Topic 5: china, chinavirus, covid, virus, wuhanvirus 
##  Topic 6: chinavirus, like, flu, cdc, around 
##  Topic 7: chinavirus, joebiden, better, form, potus 
##  Topic 8: peopl, lie, american, feel, hidenbidenlieden 
##  Topic 9: amp, scienc, india, play, coffin 
##  Topic 10: chinavirus, virus, flu, call, mandat 
##  Topic 11: chinavirus, coronavirus, scienc, taiwan, amp 
##  Topic 12: vaccin, chinavirus, don, stop, doesn 
##  Topic 13: tuckercarlson, fact, potus, dailycal, genflynn 
##  Topic 14: hacker, world, news, virus, alert 
##  Topic 15: trump, year, origin, least, india 
##  Topic 16: covid, watch, won, due, fund 
##  Topic 17: doesn, media, china, week, blast 
##  Topic 18: even, year, like, crazi, silent 
##  Topic 19: chinavirus, die, make, myocard, first 
##  Topic 20: china, chinavirus, copi, indigen, invent 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 6 (approx. per word bound = -3.913, relative change = 5.517e-03) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 7 (approx. per word bound = -3.905, relative change = 2.198e-03) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 8 (approx. per word bound = -3.898, relative change = 1.670e-03) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 9 (approx. per word bound = -3.893, relative change = 1.265e-03) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 10 (approx. per word bound = -3.889, relative change = 1.039e-03) 
## Topic 1: harsha, kakar, happen, know, shall 
##  Topic 2: chines, mani, account, sleep, other 
##  Topic 3: can, will, mask, stop, say 
##  Topic 4: death, amp, china, chinavirus, must 
##  Topic 5: china, chinavirus, covid, virus, usa 
##  Topic 6: like, cdc, chinavirus, flu, around 
##  Topic 7: chinavirus, joebiden, potus, better, form 
##  Topic 8: peopl, lie, american, feel, hidenbidenlieden 
##  Topic 9: amp, scienc, india, coffin, final 
##  Topic 10: chinavirus, virus, flu, call, tell 
##  Topic 11: chinavirus, coronavirus, taiwan, info, caught 
##  Topic 12: vaccin, chinavirus, doesn, don, stop 
##  Topic 13: tuckercarlson, fact, potus, dailycal, genflynn 
##  Topic 14: hacker, world, news, virus, alert 
##  Topic 15: year, trump, origin, least, india 
##  Topic 16: covid, watch, won, due, fund 
##  Topic 17: doesn, media, china, week, blast 
##  Topic 18: even, year, like, crazi, last 
##  Topic 19: die, chinavirus, make, myocard, first 
##  Topic 20: china, everyth, chinavirus, copi, indigen 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 11 (approx. per word bound = -3.885, relative change = 1.052e-03) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 12 (approx. per word bound = -3.881, relative change = 9.934e-04) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 13 (approx. per word bound = -3.878, relative change = 9.381e-04) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Completing Iteration 14 (approx. per word bound = -3.877, relative change = 1.987e-04) 
## ............................................................................................................
## Completed E-Step (0 seconds). 
## Completed M-Step. 
## Model Terminated Before Convergence Reached

Let us see PROPORTION OF EACH TOPIC in the entire CORPUS.

## Just insert your STM output
plot.STM(pFit, type="summary", n=5,xlim=c(0,1))

Now it is time to interpret the stm model.

###LIST OF TOP WORDS for topics 1, 7, & 10
labelTopics(pFit, c(17, 19, 6))

## Topic 17 Top Words:
##       Highest Prob: media, china, week, blast, doesn, yet, includ 
##       FREX: week, bomb, comic, globaltimesnew, none, speak, media 
##       Lift: bomb, comic, globaltimesnew, none, oct, speak, agent 
##       Score: anyth, bomb, comic, globaltimesnew, none, speak, media 
## Topic 19 Top Words:
##       Highest Prob: die, chinavirus, make, myocard, first, heart, terrorist 
##       FREX: myocard, heart, make, die, bellspalsi, courtneymilan, defens 
##       Lift: bellspalsi, courtneymilan, defens, fullofstar, heart, jerri, liter 
##       Score: bellspalsi, myocard, die, make, heart, courtneymilan, fullofstar 
## Topic 6 Top Words:
##       Highest Prob: like, cdc, flu, around, chinavirus, world, show 
##       FREX: show, cdc, data, ifr, march, sinc, like 
##       Lift: data, ifr, march, pariscum, sinc, truthmav, wuhanfl 
##       Score: pariscum, show, data, ifr, march, sinc, truthmav

Let us do wordcloud, but I am not suggesting you to do this in your published research.

###WORDCLOUD for a specified TOPIC
cloud(pFit, topic=17)

Let us find some texts that are most representative for a particular topic using findThoughts function:

Outputs most representative documents for a particular topic. Use this in order to get a better sense of the content of actual documents with a high topical content.

findThoughts(model, texts = NULL, topics = NULL, n = 3, thresh = NULL, where = NULL, meta = NULL)

#object 'thoughts1' contains 2 documents about topic 1. 'texts=shortdoc,' gives you just the first 250 words

thoughts1 <- findThoughts(pFit,
                          texts=meta$text,
                          n=2,
                          topics=17)$docs[[1]]
#will show you the output
plotQuote(thoughts1, width=40, main="Topic 17")

Sentiment Analysis

Some of you might be interested in analyzing sentiment of tweets. For instance, you may want to analyze the public opinion toward election results 2020.

Here let us do a very short demo about how you can do sentiment analysis in R using sentimentr package.

Let us use syuzhet package as the baseline. Matthew Jockers created the syuzhet package that utilizes dictionary lookups for the Bing, NRC, and Afinn methods as well as a custom dictionary. He also utilizes a wrapper for the Stanford coreNLP which uses much more sophisticated analysis.

You can check here for a tutorial for syuzhet: https://cran.r-project.org/web/packages/syuzhet/vignettes/syuzhet-vignette.html

We are going to use its functions:

• get_sentences():implements the openNLP sentence tokenizer

• get_tokens:tokenize by words instead of sentences

• get_sentiment():includes two parameters–a character vector (of sentences or words) and a “method.” The method you select determines which of the four available sentiment extraction methods to employ. In the example that follows below, the “syuzhet” (default) method is called.Other methods include “bing”, “afinn”, “nrc”, and “stanford”.

• get_nrc_sentiment: implements Saif Mohammad’s NRC Emotion lexicon. The NRC emotion lexicon is a list of words and their associations with eight emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive) (See http://www.purl.org/net/NRCemotionlexicon).

library(syuzhet)
# Let use tokenize our origanl texts first
my_example_text <- data$text
s_v <- get_sentences(my_example_text)
head(s_v)

## [1] "rt @sindhichokrovb yes on a serious note why these idiots of so called iim amp intelligent institutes do not trust their own capabilities"
## [2] "yes on a serious note why these idiots of so called iim amp intelligent institutes do not trust their own capabili t co od16v8jagg"       
## [3] "rt @truthmav like i said since march 2020 it s a flu cdc who data from around the world shows the ifr of #chinavirus a k a #wuhanfl"      
## [4] "rt @dr sdrk @thebossoriginal @mvzexplorer @cselston @nancymagamonkey @kimmarinesis @onegeorgiapeach @rhondafurin here is the master mind" 
## [5] "rt @dr sdrk @thebossoriginal @mvzexplorer @cselston @nancymagamonkey @kimmarinesis @onegeorgiapeach @rhondafurin here is the master mind" 
## [6] "no record of unvaxxed who recovered from commie #chinavirus spreading virus a few million cases amp not one occurren t co ee9a4arorr"

syuzhet_vector <- get_sentiment(s_v, method="syuzhet")
head(syuzhet_vector)

## [1]  2.05  1.80 -0.25  0.00  0.00 -0.50

Let us try different methods (use different lexicons)

bing_vector <- get_sentiment(s_v, method="bing")
head(bing_vector)

## [1]  1  1  1  1  1 -1

afinn_vector <- get_sentiment(s_v, method="afinn")
head(afinn_vector)

## [1]  4  4  0  0  0 -1

nrc_vector <- get_sentiment(s_v, method="nrc", lang = "english")

## Warning: `filter_()` was deprecated in dplyr 0.7.0.
## Please use `filter()` instead.
## See vignette('programming') for more help

head(nrc_vector)

## [1]  1  1  0  1  1 -1

Let us compare the difference

rbind(
  sign(head(syuzhet_vector)),
  sign(head(bing_vector)),
  sign(head(afinn_vector)),
  sign(head(nrc_vector))
)

##      [,1] [,2] [,3] [,4] [,5] [,6]
## [1,]    1    1   -1    0    0   -1
## [2,]    1    1    1    1    1   -1
## [3,]    1    1    0    0    0   -1
## [4,]    1    1    0    1    1   -1

Let us plot the result

plot(
  syuzhet_vector, 
  type="h", 
  main="Example Plot Trajectory", 
  xlab = "Narrative Time", 
  ylab= "Emotional Valence"
  )

let us use the package’s simple_plot to plot the trend.

simple_plot(syuzhet_vector)

Let us see get_nrc_sentiment results

nrc_data <- get_nrc_sentiment(s_v)

## Warning: `group_by_()` was deprecated in dplyr 0.7.0.
## Please use `group_by()` instead.
## See vignette('programming') for more help

## Warning: `data_frame()` was deprecated in tibble 1.1.0.
## Please use `tibble()` instead.

barplot(
  sort(colSums(prop.table(nrc_data[, 1:8]))), 
  horiz = TRUE, 
  cex.names = 0.7, 
  las = 1, 
  main = "Emotions in Sample Tweets", xlab="Percentage"
  )