Twitter Analytics Project

Thu, 15 Dec 2022 00:00:00 +0000

The original unedited notebook is available on Google Colab. Some of the code, readouts, and .head() calls are suppressed on this page for readability.

Table of Contents

Introduction/Background

In this project, we analyze the data of over 2 million tweets relating to customer support. The dataset consists of IDs for each tweet, IDs for unique authors (anonymized apart from company support accounts), and text for the tweets, along with fields identifying the IDs of other tweets that reply to the tweet in question or the previous tweet which is being replied to. We use these associations to construct conversations from the relationships between tweets. We then use natural language processing (NLP) and sentiment analysis to analyze the corpus of tweets, running predictive models on these sentiments and other features of the data, like the response times and the sentiment of related tweets.

Part of our motivation was to look at an analysis of social media sentiment: Twitter has a reputation for being a toxic/hateful place, and customer support is also known for being a particularly unsavory field. We were also interested in how sentiment transmitted in a conversation–would very negative tweets attract faster support, or affect the company’s response? Would a company responding more positively make the customer also more positive? We use regression and classification models to try and answer these questions by analyzing the predictive power of the models. Separately, we were also curious if there is a detectable change in the sentiment and ratio of positive/negative/neutral words between corporate Twitter accounts and individuals, so we also used a classification model to try and predict this.

Data Cleaning / Data Wrangling

We first import relevant libraries.

import pandas as pd
import numpy as np
import time
import seaborn as sns
import matplotlib.pyplot as plt
import ssl
import nltk
from nltk.corpus import stopwords
nltk.download('stopwords')
nltk.download('punkt')
nltk.download('vader_lexicon')
stopwords = set(stopwords.words('english'))

The tweet csv can be access here. This data must be downloaded and placed in the root directory in order to read the csv file into the df_tweets dataframe.

df_tweets = pd.read_csv('twcs.csv')

Below, we can see the columns for this dataset.

tweet_id is an individual numerical id for each tweet in the dataset
author_id is the name (for companies) or user id (for non-companies) of the tweet author
inbound tells whether the tweet was sent by a company or a non-company author (True if from non-company, false otherwise)
created_at is the date of the tweet creation
text is the content of the tweet
response_tweet_id is a comma-separated string containing the tweet ids of the responses to the selected tweet (if it exists)
in_response_to_tweet_id contains the tweet id of the tweet to which the current tweet is a response (if it exists)

# First five rows of the data
df_tweets.head()

	tweet_id	author_id	inbound	created_at	text	response_tweet_id	in_response_to_tweet_id
0	1	sprintcare	False	Tue Oct 31 22:10:47 +0000 2017	@115712 I understand. I would like to assist y...	2	3.0
1	2	115712	True	Tue Oct 31 22:11:45 +0000 2017	@sprintcare and how do you propose we do that	NaN	1.0
2	3	115712	True	Tue Oct 31 22:08:27 +0000 2017	@sprintcare I have sent several private messag...	1	4.0
3	4	sprintcare	False	Tue Oct 31 21:54:49 +0000 2017	@115712 Please send us a Private Message so th...	3	5.0
4	5	115712	True	Tue Oct 31 21:49:35 +0000 2017	@sprintcare I did.	4	6.0

From the datatypes of the columns, we can see that many of the id columns are in object form, and the created_at column is in string form. We address this issues later on.

# Checking datatypes of columns
df_tweets.dtypes

tweet_id                     int64
author_id                   object
inbound                       bool
created_at                  object
text                        object
response_tweet_id           object
in_response_to_tweet_id    float64
dtype: object

# Get number of rows of dataset
df_tweets.shape[0]

# Get the number of nulls per column
df_tweets.isna().sum()

tweet_id                         0
author_id                        0
inbound                          0
created_at                       0
text                             0
response_tweet_id          1040629
in_response_to_tweet_id     794335
dtype: int64

We can see that there are many null values in the response tweet id columns. This makes sense, as not every tweet will be a response to another tweet / have responses. To address this, we convert all null values to 0 (no tweet id has value 0 so this is a valid way to label such null values). We convert the in_response_to_tweet_id column to integers and response_tweet_id to a list of integers.

# Fill nulls and change to integer type
df_tweets = df_tweets.fillna('0')
df_tweets['in_response_to_tweet_id'] = df_tweets['in_response_to_tweet_id'].astype('int64')

# Convert string to list of integers
df_tweets['response_tweet_id'] = df_tweets['response_tweet_id'].str.split(',')
df_tweets['response_tweet_id'] = df_tweets['response_tweet_id'].apply(lambda x: list(map(int, x)))
df_tweets.head()

# Sort the rows by tweet_id
df_tweets = df_tweets.sort_values(by='tweet_id').reset_index(drop=True)
df_tweets.head()

To ensure that we can efficiently track times and time deltas in our analysis, we convert the date string in the created_at columns into a datetime object.

df_tweets['created_at'] = pd.to_datetime(df_tweets['created_at'], format='%a %b %d %H:%M:%S %z %Y')
df_tweets.head()

In the text column, we make three observations.

Each tweet’s first word consists of an at sign followed by the author_id to which the tweet is directed. This is not relevant to the actual content of the text, so we remove this.
Some tweets contain emojis. As the main focus of this project is centered around the sentiments of the textual content, we remove emojis from any tweets that have them by removing all non-ascii characters.
Some tweets contain non-English words. As the main focus of this project is centered around analysis of English text, we can also remove these characters by removing all non-ascii characters.

# Regular expression to remove words starting with an at sign and any non-ascii characters (e.g. emojis, non-English)
df_tweets['text'] = df_tweets['text'].str.replace('(\@\w+.*?)|[^\x00-\x7F]', "")
df_tweets.head()

Adding columns for tweet tree identification and tree depth

Due to the nature of tweet chains, we wanted a way to differentiate tweets based on their tweet response tree and their depth within the tweet chain. To do so, we created two new columns: tweet_group and tweet_wave. tweet_group contains a unique numerical identifier for each tweet tree, and tweet_wave contains the depth within the tree (starting at 1). To start, we first find the root nodes of these trees (a.k.a. the first tweets in the chain), which is done by finding in_response_to_tweet_id columns with a value of 0 (signifying that the tweet is not in response to a tweet, as specified in our null-removing step). For each of these root nodes, we assign a unique group index, which is maintained by a global group_index and managed by the group function. We then set the tweet_wave of the roots to 1. This is done in preparation for the next step, which utilize a BFS-life implementation to expand out on the tweet trees depth by depth.

# Create temporary array with select columns
df_temp = df_tweets[['response_tweet_id', 'tweet_id', 'in_response_to_tweet_id']]

# Global index used to assign tweet group number to root nodes
group_index = 1

# Function used to assign tweet group to nodes and increment group_index
def group(tweet):
  global group_index
  group_index += 1
  return group_index - 1

# Set tweet group values for root nodes
df_temp["tweet_group"] = df_temp["in_response_to_tweet_id"].apply(lambda x: group(x) if not x else 0)

# Set tweet wave of root nodes to 1
df_temp["tweet_wave"] = df_temp["tweet_group"].apply(lambda x: 1 if  x else 0)
df_temp = df_temp.drop(columns=['in_response_to_tweet_id'])
df_temp.head(25)

	response_tweet_id	tweet_id	tweet_group	tweet_wave
0	[2]	1	0	0
1	[0]	2	0	0
2	[1]	3	0	0
3	[3]	4	0	0
4	[4]	5	0	0
5	[5, 7]	6	0	0
6	[0]	7	0	0
7	[9, 6, 10]	8	1	1
8	[0]	9	0	0
9	[0]	10	0	0

We now do a BFS-like implementation to expand on the tweet trees to a depth of 10. By doing a series of joins, we create a new dataframe df_tempTotal that contains the tweets with tweet tree depths from 1-10 and their corresponding tweet groups.

# Initialize df_tempTotal dataframe
df_tempTotal = df_temp[df_temp['tweet_wave'] == 1]

# Initialize df_tempRecur dataframe, which will be used in every iteration of the BFS as the left dataframe of the joins
df_tempRecur = df_tempTotal.copy()

# Do BFS for 9 iterations for a total of 10 levels of the BFS, including root nodes
for i in range(1, 10):
  # Explode the response_tweet_ids to give each response_tweet_id its own row and simplifying the joins
  df_tempRecur = df_tempRecur.explode('response_tweet_id')
  df_tempRecur = df_tempRecur.drop(columns=['tweet_wave', 'tweet_id'])

  # Join df_tempRecur to df_temp to get the next depth tweet ids
  df_tempRecur = df_tempRecur.merge(df_temp, left_on='response_tweet_id', right_on='tweet_id')

  # Drop and rename extraneous columns to extract only the newly added tweet ids
  df_tempRecur = df_tempRecur.drop(columns=['tweet_group_y', 'response_tweet_id_x'])
  df_tempRecur = df_tempRecur.rename(columns={'tweet_group_x' : 'tweet_group', 'response_tweet_id_y' : 'response_tweet_id'})

  # Set the tweet_wave of newly added tweet ids to the tree depth
  df_tempRecur['tweet_wave'] = i + 1

  # Append the new tweet ids and relevant information to df_tempTotal
  df_tempTotal = df_tempTotal.append(df_tempRecur)

df_tempTotal

	response_tweet_id	tweet_id	tweet_group	tweet_wave
7	[9, 6, 10]	8	1	1
17	[17]	18	2	1
19	[19]	20	3	1
28	[28]	29	4	1
30	[30]	31	5	1
...	...	...	...	...
16344	[0]	2982342	792579	10
16345	[0]	2986328	792626	10
16346	[2983409]	2983410	792716	10
16347	[2985953]	2985952	793591	10
16348	[0]	2985964	793593	10

We merge df_tempTotal to our original df_tweets. Note that in the process, we essentially filter out any tweets that have a tree depth greater than 10. We do this for a few reasons.

Retrieving tweets with depths 1-10 gets us 2743800 rows out of a total of 2811774 rows in the original dataframe. This captures most of the data, and thus we are only losing a relatively small number of rows.
For the sake of our analysis, we determined that tweet chains that are too long could potentially be outliers in our main analysis. In addition, some of these tweet chains can have extremely long lengths (25+). Thus we determined that analyzing depths 1-10 would provide us with the most general analysis of our tweet chain data.

df_tempTotal = df_tempTotal.drop(columns=['response_tweet_id'])
df_tweets = df_tweets.merge(df_tempTotal, on='tweet_id')
df_tweets

del(df_temp)
del(df_tempTotal)

Tweet text columns (tokenization, sentiment analysis)

We use the nltk library to tokenize the tweet text for future analysis.

def tokenize_content(content):
  token = nltk.word_tokenize(content)
  token = [word for word in token if word.isalpha()]
  token = [word.lower() for word in token]
  token = [word for word in token if word not in stopwords]
  return token

df_tweets['tokenized_text'] = df_tweets['text'].apply(tokenize_content)
df_tweets.head()

We use nltk’s SentimentIntensityAnalyzer to determine the sentiment polarity scores of each tweet.

from nltk.sentiment import SentimentIntensityAnalyzer
sia = SentimentIntensityAnalyzer()

def retrieve_sentiment(text):
  return sia.polarity_scores(text)

# Polarity column temporarily stores the dictionary containing polarity scores for each text
df_tweets['polarity'] = df_tweets['text'].apply(lambda x: retrieve_sentiment(x))

# Add columns for each polarity score in the sentiment analysis
df_tweets = df_tweets.join(pd.json_normalize(df_tweets['polarity'])).drop(
    'polarity', axis='columns')
df_tweets = df_tweets.rename(columns={'compound' : 'sentiment'})

df_tweets.head()

Exploratory Data Analysis

Company vs. Non-Company Tweet Sentiments

We first use boxplots and violinplots to visualize and compare the average sentiments of company tweets and non-company tweets. Here, we can see that in general, the company tweets tend to have a more positive sentiment than the non-company tweets, with the medians and both quartiles being higher for the company tweets. In particular, we can see that the median company sentiment nearly lines up with the upper quartile of the non-company sentiments, signifying that 75% of the non-company tweets has a more negative sentiment than the median company sentiment. Similarly, we can also see that 75% of the company tweets has a more positive sentiment than the median non-company tweet sentiment.

Looking at the violinplots, we can see that both types of tweets tended to be neutral for the most part. Outside the neutral zones, we find that non-company tweets has a somewhat even distribution of positive/negative sentiments, while company tweets tended to have a slightly higher concentration of positive sentiment tweets.

This results generally make sense. We would expect non-company tweeters to be tweeting to companies about service issues or other problems, which explains the lower sentiments. Companies on the other hand have to preserve solid PR and positivity in their responses, explaining their generally positive sentiments.

Both company and non-company tweets also had very similar ranges in their sentiments(-1 to 1). Given the large number of tweets in the dataset, it is plausible to have this kind of range for both categories, but one other contributing factor may be that some of these tweets are very short (<5 words), and thus a single one of these words may carry much more weight than a word in a longer sentence, contributing to very high/low sentiments.

df_temp = df_tweets[['sentiment', 'inbound']]
df_temp['inbound'] = df_temp['inbound'].apply(lambda x: "Non-company" if x else "Company")
sns.boxplot(data = df_temp, x="sentiment", y="inbound").set(
    title='Company vs. Non-Company Tweet Sentiments', ylabel='Tweet Sender')
plt.xlabel("Sentiment")
plt.show()
sns.violinplot(data = df_temp, x="sentiment", y="inbound").set(
    title='Company vs. Non-Company Tweet Sentiments', ylabel='Tweet Sender')
plt.xlabel("Sentiment")

Text(0.5, 0, 'Sentiment')

Average Sentiments of Company Tweets

Next, we average the tweet sentiments for each company and sort them from highest to lowest average sentiment. A bar graph is used to compare sentiments between every company that appears in the dataset. A table of the top 10 companies with highest / lowest average sentiments is also provided below. We note that the top three are HPSupport, TwitterSupport, and AskTarget. We also noted that two companies (SW_Help and KFC_UKI_Help) has negative average sentiments. It is also interesting to note that TwitterSupport seems to have generally very positive sentiment tweets on their own platform.

df_temp = df_tweets[['inbound', 'sentiment', 'author_id']]
df_temp = df_temp[df_temp['inbound'] == False]
df_temp = df_temp.drop(columns='inbound')
df_temp = df_temp.groupby(by='author_id').mean().sort_values(by='sentiment', ascending=False).reset_index()
display(df_temp.head(10))
display(df_temp.tail(10))
# very big graph, but shows all companies
f, ax = plt.subplots(figsize=(20, 9))
df_tempa = df_temp.head(54)
g = sns.barplot(data=df_tempa, x='author_id', y='sentiment')
g.set(title='Average Sentiments of Company Tweets', xlabel='Companies', ylabel='Sentiment')
g.set_xticklabels(df_tempa['author_id'].tolist(), rotation=90)
# ensure same scale
plt.ylim([-0.1, 0.65])
g

	author_id	sentiment
0	HPSupport	0.620603
1	TwitterSupport	0.603739
2	AskTarget	0.598224
3	AskCiti	0.577591
4	HiltonHelp	0.561960
5	askpanera	0.556811
6	Walmart	0.550349
7	CenturyLinkHelp	0.545466
8	Kimpton	0.535690
9	AldiUK	0.530867

	author_id	sentiment
98	VirginTrains	0.123603
99	GreggsOfficial	0.110357
100	MTNC_Care	0.094765
101	JackBox	0.057037
102	GWRHelp	0.050760
103	TacoBellTeam	0.032794
104	LondonMidland	0.022948
105	nationalrailenq	0.019662
106	SW_Help	-0.005786
107	KFC_UKI_Help	-0.088667

f, ax = plt.subplots(figsize=(20, 9))
df_tempb = df_temp.sort_values(
    "sentiment", ascending = True).head(54).sort_values(
        "sentiment", ascending = False)
g = sns.barplot(data=df_tempb, x='author_id', y='sentiment')
g.set(title='Average Sentiments of Company Tweets', xlabel='Companies', ylabel='Sentiment')
g.set_xticklabels(df_tempb['author_id'].tolist(), rotation=90)
plt.ylim([-0.1, 0.65])
g

<AxesSubplot: title={'center': 'Average Sentiments of Company Tweets'}, xlabel='Companies', ylabel='Sentiment'>

Sentiments of First Non-Company Tweets

Next, we attempted to look at the general distribution of the first non-company tweets in each tweet group. From the boxplots and violinplots below, we found that the distribution was fairly balanced overall, with most tweet sentiments being in the neutral zone and similar numbers of positive/negative tweets.

df_3 = df_tweets[['inbound', 'sentiment','tweet_wave']]
df_3 = df_3[(df_3['inbound'] == True) & (df_3['tweet_wave'] == 1)]
sns.boxplot(x=df_3['sentiment']).set(title='Sentiments of First Non-Company tweets', xlabel='sentiment')
plt.xlabel("Sentiment")
plt.show()
sns.violinplot(x=df_3['sentiment']).set(title='Sentiments of First Non-Company tweets', xlabel='sentiment')
plt.xlabel("Sentiment")
plt.show()

Difference in Sentiment between Company and User First Tweets By Company

Next, we looked at the difference between the sentiment in the first tweet (from user) to the first tweet response from the company. We created two dataframes, one with the user tweet and one with the company tweet, and joined them on tweet_group, an indication that both belonged to the same conversation. We then calculated the sentiment change by taking the difference, and then plotted.

# get the first tweets
df_4 = df_tweets[['tweet_group', 'in_response_to_tweet_id', 'sentiment']]
df_4 = df_4[df_4['in_response_to_tweet_id'] == 0]

# get the company tweets
df_5 = df_tweets[['tweet_group', 'author_id', 'inbound', 'sentiment', 'created_at']]
df_5 = df_5[df_5['inbound'] == False]
df_5 = df_5.sort_values(
    "created_at", ascending = True).groupby('tweet_group').head(1)

# merge to match first tweet with the first company tweet in response
df_6 = pd.merge(df_4, df_5, on = "tweet_group")
df_6 = df_6.drop(columns=['in_response_to_tweet_id', 'inbound', 'created_at'])

# calculate mean difference in sentiment by company
df_6['sentiment_diff'] = df_6['sentiment_y'] - df_6['sentiment_x']
df_6 = df_6.drop(columns=['sentiment_x', 'sentiment_y'])
df_6 = df_6.groupby(by='author_id').mean().sort_values(by='sentiment_diff', ascending=False).reset_index()

# plot first half
f, ax = plt.subplots(figsize=(20, 9))
df_6a = df_6.head(54)
g1 = sns.barplot(data=df_6a, x='author_id', y='sentiment_diff')

g1.set_xticklabels(df_6a['author_id'].tolist(), rotation=90)
display(df_6.head(10))
plt.title("Sentiment Difference Between Customer and Company First Tweets by Company")
plt.xlabel("Company")
plt.ylabel("Change in Compound Sentiment Score")
plt.ylim([-0.1, 0.75])
g1

	author_id	tweet_group	sentiment_diff
0	HPSupport	377784.992693	0.701777
1	CenturyLinkHelp	394297.025868	0.686102
2	askpanera	416177.813800	0.656815
3	AskCiti	394322.851147	0.630343
4	HiltonHelp	308925.981690	0.618681
5	AskTarget	451522.805701	0.595897
6	KeyBank_Help	349224.621622	0.593961
7	TwitterSupport	364036.941638	0.589568
8	comcastcares	410544.276653	0.562788
9	AppleSupport	412262.559967	0.559475

<AxesSubplot: title={'center': 'Sentiment Difference Between Customer and Company First Tweets by Company'}, xlabel='Company', ylabel='Change in Compound Sentiment Score'>

# plot second half
f, ax = plt.subplots(figsize=(20, 9))
df_6b = df_6.sort_values(
    "sentiment_diff", ascending = True).head(54).sort_values(
        "sentiment_diff", ascending = False)
g2 = sns.barplot(data=df_6b, x='author_id', y='sentiment_diff')
g2.set_xticklabels(df_6b['author_id'].tolist(), rotation=90)
plt.title("Sentiment Difference Between Customer and Company First Tweets by Company")
plt.xlabel("Company")
plt.ylabel("Change in Compound Sentiment Score")
plt.ylim([-0.1, 0.75])

(-0.1, 0.75)

Sentiment of User’s Last Tweet

We were interested in seeing if the sentiment of user tweets changed over time, so this is the distribution of sentiment for the last tweet that each user in the dataset tweeted.

df_7 = df_tweets[["author_id", "inbound", "text", "created_at", "sentiment"]]
df_7 = df_7[df_7["inbound"] == True]

df_7 = df_7.sort_values("created_at", ascending = False).groupby("author_id").head(1)
sns.boxplot(x=df_7['sentiment'])
plt.xlabel("Sentiment")
plt.title("Sentiment of Last User Tweet (Non-Company)")

Text(0.5, 1.0, 'Sentiment of Last User Tweet (Non-Company)')

Change in Sentiment from First Tweet to Last Tweet for Non-Companies

Similarly, we wanted to visualize the difference between the sentiment in their first tweet and last tweet, and but the boxplot was surprisingly not helpful as the middle 50% all saw no change. However, there were points at the extremes, going from positive to very negative and vice versa.

df_8 = df_tweets[["author_id", "inbound", "text", "created_at", "sentiment"]]
df_8 = df_8[df_8["inbound"] == True]

df_8 = df_8.sort_values("created_at", ascending = True).groupby("author_id").head(1)

df_8 = pd.merge(df_7, df_8, on = "author_id")
df_8["sent_diff"] = df_8["sentiment_x"] - df_8["sentiment_y"]
df_8["sent_diff"].describe()
sns.boxplot(x=df_8['sent_diff'])
plt.xlabel("Sentiment Difference")
plt.title("Change in Sentiment of Last Tweet from First Tweet for Users")

Text(0.5, 1.0, 'Change in Sentiment of Last Tweet from First Tweet for Users')

## reducing dimensions for RAM sake
df_tweets = df_tweets.drop(["tweet_id", "text", "response_tweet_id",
                            "in_response_to_tweet_id", "tokenized_text"], axis = 1)

df_tweets.to_csv("df_tweets_clean.csv", encoding='utf-8', index=False)

Modeling: Regression

# df_tweets = pd.read_csv("df_tweets_clean.csv")
df_tweets["created_at"] = pd.to_datetime(df_tweets["created_at"])

For each relationship that we want to study, we will run two different machine learning models—Elastic Net, and Random Forests.

Elastic Net will allow for the interpretability of linear regression while avoid the problem of overfitting in unregularized regression. It will also allow the prediction to adopt a full LASSO or Ridge regression if appropriate.

The Random Forest is sometimes regarded as a superior prediction model due to its use many trees for predictions, so this exercise will allow us to test that in direct comparison to Elastic Net. Additionally, if random forests are a better prediction model, it will be more helpful in identifying if a relationship really exists between the inputs and outputs.

PySpark initialization

Note: We will often delete dataframes that will not be used again and use the garbage collector method gc.collect() in order to minimize RAM usage which could cause the entire runtime to crash.

# clearing RAM
del(df_3)
del(df_4)
del(df_5)
del(df_6)
del(df_6a)
del(df_6b)
del(df_7)
del(df_8)
del(df_temp)
del(df_tempa)
del(df_tempb)
gc.collect()

!apt install libkrb5-dev
# !python -m wget -o https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
!tar xf C:\Users\takhi\Downloads\spark-3.3.1-bin-hadoop3.tgz
!pip install findspark
!pip install sparkmagic
!pip install pyspark

# Graph section
# import networkx as nx
# SQLite RDBMS
# import sqlite3
# Parallel processing
# import swifter
import pandas as pd

import os
os.environ['SPARK_HOME'] = "C:/Users/takhi/Downloads/spark-3.3.1-bin-hadoop3"
os.environ["JAVA_HOME"] = "C:/Users/takhi/.jdks/corretto-1.8.0_322"
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
import pyspark
from pyspark.sql import SQLContext

from pyspark.sql import SparkSession
from pyspark.sql.types import *
import pyspark.sql.functions as F

spark = SparkSession.builder.config("spark.driver.memory", "14g").getOrCreate()

%load_ext sparkmagic.magics

try:
    if(spark == None):
        spark = SparkSession.builder.appName('Initial').getOrCreate()
        sqlContext=SQLContext(spark)
except NameError:
    spark = SparkSession.builder.appName('Initial').getOrCreate()
    sqlContext=SQLContext(spark)

from pyspark.ml.feature import VectorAssembler
from pyspark.ml import Pipeline
from pyspark.ml.regression import LinearRegression
from pyspark.ml.regression import RandomForestRegressor
from pyspark.mllib.evaluation import RegressionMetrics
from pyspark.mllib.evaluation import MulticlassMetrics
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.ml.tuning import ParamGridBuilder
from pyspark.ml.tuning import CrossValidator
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.classification import LogisticRegression

Model 1

This first model attempts to predict the sentiment of the first company tweet based on the sentiment parameters of the first user tweet, including the compound sentiment score and the ratio of positive, negative and neutral words. The total ratio adds up to 1, so the regressions involving this sentiment omit the neu variable to avoid multicollinearity.

random_seed = 42

We first get the desired dataframe using Pandas, and then use the procedure from homework in order to convert it to Spark

df_model = df_tweets.sort_values(
    "created_at", ascending = True).groupby(["tweet_group", "inbound"]).head(1)
df_model = df_model[["inbound", "pos", "neg", "neu", "sentiment", "tweet_group"]]
df_m1a = df_model[df_model["inbound"] == True]
df_m1b = df_model[df_model["inbound"] == False]

df_model = pd.merge(df_m1a, df_m1b, on = "tweet_group")
# pos + neg + neu = 1 according to nltk devs, so we omit neu because of
# multicollinearity
features = df_model[["pos_y", "neg_y", "sentiment_y"]]
y = df_model[["sentiment_x"]]
columns_to_use = ["pos_y", "neg_y", "sentiment_y"]

This code is to serve as a base to compare the training and test mean squared error (MSE). We can contextualize the training or test MSE as low or high based on the MSE that a model would have if it predicted the overall mean every single time.

base = df_model[["sentiment_x"]].to_numpy()
print(np.mean(np.square(np.mean(base) - base)))

0.20163048376044265

model_sdf = spark.createDataFrame(df_model)
model_sdf.printSchema()

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:474: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():
C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():


root
 |-- inbound_x: boolean (nullable = true)
 |-- pos_x: double (nullable = true)
 |-- neg_x: double (nullable = true)
 |-- neu_x: double (nullable = true)
 |-- sentiment_x: double (nullable = true)
 |-- tweet_group: long (nullable = true)
 |-- inbound_y: boolean (nullable = true)
 |-- pos_y: double (nullable = true)
 |-- neg_y: double (nullable = true)
 |-- neu_y: double (nullable = true)
 |-- sentiment_y: double (nullable = true)

import gc
del (df_model)
del (df_m1a)
del (df_m1b)
gc.collect()

---------------------------------------------------------------------------

NameError                                 Traceback (most recent call last)

Cell In[15], line 2
      1 import gc
----> 2 del (df_model)
      3 del (df_m1a)
      4 del (df_m1b)


NameError: name 'df_model' is not defined

assembler = VectorAssembler(inputCols = columns_to_use, outputCol = "features")
pipe = Pipeline(stages = [assembler])
model_sdf = pipe.fit(model_sdf).transform(model_sdf)

For each model, we use an 80/20 train/test split.

train_sdf, test_sdf = model_sdf.randomSplit([0.8, 0.2], seed = random_seed)

Elastic Net

Each model is also cross validated in order to find optimal values for the hyperparameters. For Elastic Net, this is the regParam which binds the constraint to the optimization equation as well as the relative weight of the L1 and L2 constraints, entered as the elasticNetParam. We then use the best performing model for prediction

elastic = LinearRegression(maxIter=10, featuresCol = "features", labelCol = "sentiment_x")
paramGrid = ParamGridBuilder() \
    .addGrid(elastic.regParam, [0, 0.33, 0.667, 1]) \
    .addGrid(elastic.elasticNetParam, [0, 0.33, 0.667, 1]) \
    .build()
crossval = CrossValidator(estimator=elastic,
                          estimatorParamMaps=paramGrid,
                          evaluator=RegressionEvaluator(labelCol = "sentiment_x"),
                          numFolds=3)
elastic = crossval.fit(train_sdf)

We calculate both training and test predictions and eventually MSE in order to evaluate if our model is overfitting to training data

train_pred = elastic.transform(train_sdf)
test_pred = elastic.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sentiment_x))
train_metrics = RegressionMetrics(train_label)

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sentiment_x))
test_metrics = RegressionMetrics(test_label)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training MSE: ", train_metrics.meanSquaredError)
print("Test MSE: ", test_metrics.meanSquaredError)
gc.collect()

Training MSE:  0.19408484605684143
Test MSE:  0.19483644218840396





1509

Elastic net, as a regularized form of linear regression, will have coefficients that allow us to have some form of interpretability. If the MSE is lower than the base MSE calculated above, then the interpretation of these coefficients could be very interesting.

print(elastic.bestModel.coefficients)
columns_to_use

[0.7463819294866374,-0.8301102154938762,-0.17270048241776875]





['pos_y', 'neg_y', 'sentiment_y']

Random Forest

We then use a similar process for a random forest regression model, optimizing the hyperparameters of maxDepth and numTrees. We would like to have a wider array of options in the parameter grid, but computational concerns and

rf = RandomForestRegressor(featuresCol = "features",
                            labelCol = "sentiment_x", seed = random_seed)
paramGrid = ParamGridBuilder() \
    .addGrid(rf.maxDepth, [4, 6]) \
    .addGrid(rf.numTrees, [20, 30]) \
    .build()
crossval = CrossValidator(estimator=rf,
                          estimatorParamMaps=paramGrid,
                          evaluator=RegressionEvaluator(labelCol = "sentiment_x"),
                          numFolds=3)
rf = crossval.fit(train_sdf)

train_pred = rf.transform(train_sdf)
test_pred = rf.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sentiment_x))
train_metrics = RegressionMetrics(train_label)

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sentiment_x))
test_metrics = RegressionMetrics(test_label)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training MSE: ", train_metrics.meanSquaredError)
print("Test MSE: ", test_metrics.meanSquaredError)
gc.collect()

Training MSE:  0.19041896237273437
Test MSE:  0.19139729255344803





656

Remarks

Based on the MSE of using the mean to predict the outcome variable, the regressions indicate that sentiment of the user tweet is not a very good predictor the sentiment of the first company tweet. Using the mean, the MSE would be 0.202, while the Elastic Net slightly improved on that with a test MSE 0.195 and the Random Forest slightly better than that with an MSE of 0.191. The similar training and test MSEs indicate a lack of overfitting in both models, but also show that the data could not be fit very well in general with these predictors.

Model 2

The second model seeks to identify if the sentiment of a user’s second tweet can be predicted by the sentiment of the company’s first tweet. The concept behind this is that the type of way that a company responds may allow us to predict how a person responds—perhaps if a company is very positive the sentiment of the user’s reply will also be more positive/greater in value.

This model and all other models follow the same structure as Model 1 (creating dataset, transforming to Spark DF, cross-validation tuning, fitting model, predicting, calculating accuracy) so they are not annotated.

df_model = df_tweets.sort_values(["created_at"], ascending = True).groupby(["tweet_group", "inbound"]).head(2)
# get company first tweet
df_model_extra = df_model[df_model["inbound"] == False].groupby("tweet_group").head(1)
# get second user tweet
df_model = df_model.sort_values("created_at", ascending = False).groupby("tweet_group").head(1)
#join
df_model = pd.merge(df_model[["sentiment", "tweet_group"]],
                 df_model_extra[["tweet_group", "sentiment", "pos", "neg"]],
                 on = "tweet_group")
features = df_model[["pos", "neg", "sentiment_y"]]
y = df_model[["sentiment_x"]]
columns_to_use = ["pos", "neg", "sentiment_y"]
del (df_model_extra)
gc.collect()

# calculating what MSE would be if we used the overall mean to predict
base = df_model[["sentiment_x"]].to_numpy()
print(np.mean(np.square(np.mean(base) - base)))

0.15583840129965396

model_sdf = spark.createDataFrame(df_model)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:474: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():
C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():

assembler = VectorAssembler(inputCols = columns_to_use, outputCol = "features")
pipe = Pipeline(stages = [assembler])
model_sdf = pipe.fit(model_sdf).transform(model_sdf)

train_sdf, test_sdf = model_sdf.randomSplit([0.8, 0.2], seed = random_seed)

Elastic Net

elastic = LinearRegression(maxIter=10, featuresCol = "features", labelCol = "sentiment_x")
paramGrid = ParamGridBuilder() \
    .addGrid(elastic.regParam, np.arange(0, 1, 0.25)) \
    .addGrid(elastic.elasticNetParam, np.arange(0, 1, 0.25)) \
    .build()
crossval = CrossValidator(estimator=elastic,
                          estimatorParamMaps=paramGrid,
                          evaluator=RegressionEvaluator(labelCol = "sentiment_x"),
                          numFolds=3)
elastic = crossval.fit(train_sdf)

train_pred = elastic.transform(train_sdf)
test_pred = elastic.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sentiment_x))
train_metrics = RegressionMetrics(train_label)

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sentiment_x))
test_metrics = RegressionMetrics(test_label)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training MSE: ", train_metrics.meanSquaredError)
print("Test MSE: ", test_metrics.meanSquaredError)
gc.collect()

Training MSE:  0.09065836855585652
Test MSE:  0.0899119900442909





1718

print(elastic.bestModel.coefficients)
columns_to_use

[0.21960564506192556,0.11552338275744405,0.5825977763540658]





['pos', 'neg', 'sentiment_y']

Random Forest

rf = RandomForestRegressor(featuresCol = "features",
                            labelCol = "sentiment_x", seed = random_seed)
paramGrid = ParamGridBuilder() \
    .addGrid(rf.maxDepth, [4, 5, 6]) \
    .addGrid(rf.numTrees, [10, 20, 30]) \
    .build()
crossval = CrossValidator(estimator=rf,
                          estimatorParamMaps=paramGrid,
                          evaluator=RegressionEvaluator(labelCol = "sentiment_x"),
                          numFolds=3)
rf = crossval.fit(train_sdf)

train_pred = rf.transform(train_sdf)
test_pred = rf.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sentiment_x))
train_metrics = RegressionMetrics(train_label)

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sentiment_x))
test_metrics = RegressionMetrics(test_label)

print("Training MSE: ", train_metrics.meanSquaredError)
print("Test MSE: ", test_metrics.meanSquaredError)
gc.collect()

Training MSE:  0.09046014672851513
Test MSE:  0.08964395931128473





913

Remarks

In this case, the models do appear to have some predictive power. The base MSE is 0.156, while the Elastic Net and Random Forest both had a test MSE of 0.090, a significantly smaller value. This also means that both models appeared to fit the data equally well, and the coefficients of the Elastic Net seem to indicate having a higher overall sentiment and greater ratio of positive words were associated with largest effect in increasing the sentiment of the user’s second tweet. Peculiarly, the ratio of negative words also had a positive coefficient, which suggests that sentiment is just overall higher for the second tweet. This makes sense, as users are likely more negative tweeting out into the void than they are when their issues are recognized by the company they address. However, it does generally suggest that more positive tweets from the company are met with more positive responses.

Model 3

The last of the regression models predicting sentiment in response to tweets, this model attempts to see if there is a more delayed effect of the company’s sentiment on the user, by predicting the sentiment of a user’s third tweet based on the first two company tweets. For example, perhaps one general response and a positive second response are more likely to elicit a positive third tweet from the user.

df_model = df_tweets[df_tweets["inbound"] == False].sort_values(
    ["created_at"], ascending = True).groupby(["tweet_group"]).head(2)
df_model_extra = df_tweets[df_tweets["inbound"] == True].sort_values(
    "created_at", ascending = True).groupby("tweet_group").head(3)
# extra will have the third tweet of user
df_model_extra = df_model_extra.sort_values("created_at", ascending = False).groupby("tweet_group").head(1)
df_model = df_model[["sentiment", "pos", "neg", "tweet_group", "created_at"]]
# join first and second company tweets
df_model = pd.merge(df_model.sort_values("created_at", ascending = True).groupby("tweet_group").head(1),
                    df_model.sort_values("created_at", ascending = False).groupby("tweet_group").head(1),
                    on = "tweet_group")
df_model = pd.merge(df_model_extra[["tweet_group", "sentiment"]], df_model, on = "tweet_group")


features = df_model[["pos_x", "neg_x", "sentiment_x", "pos_y", "neg_y", "sentiment_y"]]
y = df_model[["sentiment"]]
columns_to_use = ["pos_x", "neg_x", "sentiment_x", "pos_y", "neg_y", "sentiment_y"]
del (df_model_extra)
gc.collect()

# calculating what MSE would be if we used the overall mean to predict
base = df_model[["sentiment"]].to_numpy()
print(np.mean(np.square(np.mean(base) - base)))

0.18928519870713995

model_sdf = spark.createDataFrame(df_model)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:474: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():
C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():

assembler = VectorAssembler(inputCols = columns_to_use, outputCol = "features")
pipe = Pipeline(stages = [assembler])
model_sdf = pipe.fit(model_sdf).transform(model_sdf)

train_sdf, test_sdf = model_sdf.randomSplit([0.8, 0.2], seed = random_seed)

Elastic Net

elastic = LinearRegression(maxIter=10, featuresCol = "features", labelCol = "sentiment")
paramGrid = ParamGridBuilder() \
    .addGrid(elastic.regParam, np.arange(0, 1, 0.25)) \
    .addGrid(elastic.elasticNetParam, np.arange(0, 1, 0.25)) \
    .build()
crossval = CrossValidator(estimator=elastic,
                          estimatorParamMaps=paramGrid,
                          evaluator=RegressionEvaluator(labelCol = "sentiment"),
                          numFolds=3)
elastic = crossval.fit(train_sdf)

train_pred = elastic.transform(train_sdf)
test_pred = elastic.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sentiment))
train_metrics = RegressionMetrics(train_label)

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sentiment))
test_metrics = RegressionMetrics(test_label)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training MSE: ", train_metrics.meanSquaredError)
print("Test MSE: ", test_metrics.meanSquaredError)
gc.collect()

Training MSE:  0.18309974987869396
Test MSE:  0.18250699907308532





1700

print(elastic.bestModel.coefficients)
columns_to_use

[0.14790908185889448,-0.4021527367743378,-0.05429529225476363,0.5669464108344088,-0.3032378924075626,-0.11079966572632083]





['pos_x', 'neg_x', 'sentiment_x', 'pos_y', 'neg_y', 'sentiment_y']

Random Forest

rf = RandomForestRegressor(featuresCol = "features",
                            labelCol = "sentiment", seed = random_seed)
paramGrid = ParamGridBuilder() \
    .addGrid(rf.maxDepth, [4, 5, 6]) \
    .addGrid(rf.numTrees, [10, 20, 30]) \
    .build()
crossval = CrossValidator(estimator=rf,
                          estimatorParamMaps=paramGrid,
                          evaluator=RegressionEvaluator(labelCol = "sentiment"),
                          numFolds=3)
rf = crossval.fit(train_sdf)

train_pred = rf.transform(train_sdf)
test_pred = rf.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sentiment))
train_metrics = RegressionMetrics(train_label)

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sentiment))
test_metrics = RegressionMetrics(test_label)

print("Training MSE: ", train_metrics.meanSquaredError)
print("Test MSE: ", test_metrics.meanSquaredError)
gc.collect()

Training MSE:  0.1792486218230749
Test MSE:  0.17873323893850637





1033

Remarks

Based on the MSEs, there does not seem to be much predictive power associated with the sentiment of a company’s first and second tweets on a user’s third tweet. The base MSE was 0.189, while the Elastic Net had a test value of 0.183 and the Random Forest performed slightly better at 0.179. The overall difference does not appear very significant, so while the models may have captured some of the variation, they do not appear to be very powerful.

Model 4

The fourth and last regression model attempts to predict the time difference between a user’s first tweet and the company’s first response based on the sentiment of the user’s first tweet. Perhaps a more negative/angrier tweet will elicit a more rapid response than one that is more neutral. It is important to note here, however, that there is some sample bias, as the dataset only consists of tweets asking for customer support that have been replied to—perhaps sentiment is a better predictor of whether or not a company will reply rather than how fast it replies.

df_model = df_tweets.sort_values(
    "created_at", ascending = True).groupby(["tweet_group", "inbound"]).head(1)
df_model = df_model[["inbound", "pos", "neg", "neu", "sentiment", "tweet_group", "created_at"]]
df_model_extra = df_model[df_model["inbound"] == True]
df_model = df_model[df_model["inbound"] == False]

df_model = pd.merge(df_model[["tweet_group","created_at"]],
                    df_model_extra[["tweet_group", "sentiment", "created_at", "pos", "neg"]], on = "tweet_group")
## company time minus user time
df_model["time_delta"] = df_model["created_at_x"] - df_model["created_at_y"]
## convert from ns to minutes
df_model["time_delta"] = df_model["time_delta"].apply(lambda x: x.seconds / 3600)

features = df_model[["sentiment", "pos", "neg"]]
y = df_model[["time_delta"]]
columns_to_use = ["sentiment", "pos", "neg"]
del (df_model_extra)
gc.collect()

# calculating what MSE would be if we used the overall mean to predict
base = df_model[["time_delta"]].to_numpy()
print(np.mean(np.square(np.mean(base) - base)))

23.417744062597894

model_sdf = spark.createDataFrame(df_model)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:474: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():
C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():

assembler = VectorAssembler(inputCols = columns_to_use, outputCol = "features")
pipe = Pipeline(stages = [assembler])
model_sdf = pipe.fit(model_sdf).transform(model_sdf)

train_sdf, test_sdf = model_sdf.randomSplit([0.8, 0.2], seed = random_seed)

Elastic Net

elastic = LinearRegression(maxIter=10, featuresCol = "features", labelCol = "time_delta")
paramGrid = ParamGridBuilder() \
    .addGrid(elastic.regParam, np.arange(0, 1, 0.25)) \
    .addGrid(elastic.elasticNetParam, np.arange(0, 1, 0.25)) \
    .build()
crossval = CrossValidator(estimator=elastic,
                          estimatorParamMaps=paramGrid,
                          evaluator=RegressionEvaluator(labelCol = "time_delta"),
                          numFolds=3)
elastic = crossval.fit(train_sdf)

train_pred = elastic.transform(train_sdf)
test_pred = elastic.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.time_delta))
train_metrics = RegressionMetrics(train_label)

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.time_delta))
test_metrics = RegressionMetrics(test_label)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training MSE: ", train_metrics.meanSquaredError)
print("Test MSE: ", test_metrics.meanSquaredError)
gc.collect()

Training MSE:  23.45191416873251
Test MSE:  23.214754248884862





1891

print(elastic.bestModel.coefficients)
columns_to_use

[0.06779660584706401,-0.05526114749735003,-0.7479872231535638]





['sentiment', 'pos', 'neg']

Random Forest

rf = RandomForestRegressor(featuresCol = "features",
                            labelCol = "time_delta", seed = random_seed)
paramGrid = ParamGridBuilder() \
    .addGrid(rf.maxDepth, [4, 5, 6]) \
    .addGrid(rf.numTrees, [10, 20, 30]) \
    .build()
crossval = CrossValidator(estimator=rf,
                          estimatorParamMaps=paramGrid,
                          evaluator=RegressionEvaluator(labelCol = "time_delta"),
                          numFolds=3)
rf = crossval.fit(train_sdf)

train_pred = rf.transform(train_sdf)
test_pred = rf.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.time_delta))
train_metrics = RegressionMetrics(train_label)

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.time_delta))
test_metrics = RegressionMetrics(test_label)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training MSE: ", train_metrics.meanSquaredError)
print("Test MSE: ", test_metrics.meanSquaredError)
gc.collect()

Training MSE:  23.40218635081998
Test MSE:  23.175258943545114





2455

%reset Out

Note: the time delta was converted to hours for two reasons: so that the MSEs would be more comparable and reasonable as smaller units like minutes and seconds would be highly sensitive to any outliers due to the nature of squaring, and also because a timeframe of hours seemed more appropriate to judge the performance of a company’s tweet—it would not make sense for them to respond to the first tweet on the timescale of multiple days considering the pace of social media, nor would it be reasonable to expect responses within a few minutes or seconds.

Remarks

It appears that the reply times of companies are relatively independent of a user’s tweet sentiment, as the base MSE of 23.4 is barely larger than the test MSE of 23.2 that both the Elastic Net and Random Forest have. However, this may be slightly expected—as mentioned previously, the sentiment may predict whether or not a tweet is replied to overall instead.

Modeling: Classification

For classification we will similarly be using Elastic Net and Random Forests, although in this context we will be using Logistic Regression and the Random Forest Classifier from PySpark. We would like to test the ability of neural networks and FNNs, but due to the computational constraints of Colab we have omitted this from our analysis.

Model 1

This classification model seeks to find if there are it is possible to identify company and non-company accounts based on the sentiment of a tweet. The rationale behind this is that companies may have much higher proportions of ‘safer’ and neutral words in their tweets than users do. This may potentially allow a classification model to identify if an author is corporate based solely on a tweet.

Because of variability from tweet to tweet, this may not have good predictive power, so it is combined with Model 2 which aggregates over all tweets by an author instead.

df_model = df_tweets[["inbound", "sentiment", "pos", "neu"]]
df_model["inbound"] = df_model["inbound"].astype(float)
print(df_model["inbound"].mean())

features = df_model[["sentiment", "pos", "neu"]]
y = df_model[["inbound"]]
columns_to_use = ["sentiment", "pos", "neu"]
gc.collect()

0.54570121728989


C:\Users\takhi\AppData\Local\Temp\ipykernel_11608\2617133014.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_model["inbound"] = df_model["inbound"].astype(float)





24

model_sdf = spark.createDataFrame(df_model)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:474: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():
C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():

assembler = VectorAssembler(inputCols = columns_to_use, outputCol = "features")
pipe = Pipeline(stages = [assembler])
model_sdf = pipe.fit(model_sdf).transform(model_sdf)

train_sdf, test_sdf = model_sdf.randomSplit([0.8, 0.2], seed = random_seed)

Elastic Net

elastic = LogisticRegression(maxIter=10, featuresCol = "features", labelCol = "inbound")
paramGrid = ParamGridBuilder() \
    .addGrid(elastic.regParam, [0, 0.333, 0.667, 1]) \
    .addGrid(elastic.elasticNetParam, [0, 0.333, 0.667, 1]) \
    .build()
crossval = CrossValidator(estimator=elastic,
                          estimatorParamMaps=paramGrid,
                          evaluator=MulticlassClassificationEvaluator(labelCol = "inbound"),
                          numFolds=3)
elastic = crossval.fit(train_sdf)

train_pred = elastic.transform(train_sdf)
test_pred = elastic.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.inbound))
train_metrics = MulticlassMetrics(train_label)
cm_train = np.array(train_metrics.confusionMatrix().toArray())

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.inbound))
test_metrics = MulticlassMetrics(test_label)
cm_test = np.array(test_metrics.confusionMatrix().toArray())

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training Accuracy: ", cm_train.trace() / cm_train.sum())
print("Test Accuracy: ", cm_test.trace() / cm_test.sum())
gc.collect()

Training Accuracy:  0.6688046636606256
Test Accuracy:  0.6689975074893836





3727

print(elastic.bestModel.coefficients)
columns_to_use

[-2.3134871288472585,2.1983159361975084,0.010341737571913463]





['sentiment', 'pos', 'neu']

Random Forest

rf = RandomForestClassifier(featuresCol = "features",
                            labelCol = "inbound", seed = random_seed)
paramGrid = ParamGridBuilder() \
    .addGrid(rf.maxDepth, [4, 6]) \
    .addGrid(rf.numTrees, [20, 30]) \
    .build()
crossval = CrossValidator(estimator=rf,
                          estimatorParamMaps=paramGrid,
                          evaluator=MulticlassClassificationEvaluator(labelCol = "inbound"),
                          numFolds=3)
rf = crossval.fit(train_sdf)

train_pred = rf.transform(train_sdf)
test_pred = rf.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.inbound))
train_metrics = MulticlassMetrics(train_label)
cm_train = np.array(train_metrics.confusionMatrix().toArray())

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.inbound))
test_metrics = MulticlassMetrics(test_label)
cm_test = np.array(test_metrics.confusionMatrix().toArray())

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training Accuracy: ", cm_train.trace() / cm_train.sum())
print("Test Accuracy: ", cm_test.trace() / cm_test.sum())
gc.collect()

Training Accuracy:  0.6714725668763668
Test Accuracy:  0.6714024976159784





1685

Remarks

In the case of classification, we will evaluate the accuracy of each model without a separate anchor, as there is no analogue of predicting the overall mean for classification that does not just measure the relative frequency of the classes.

On a per-tweet basis, it seems that the regularized Logistic Regression and Random Forest perform reasonably well, but not excellent by any means. Both have test accuracies around 67%, which shows that they are not excellent classifiers but will pick company vs user correctly around 2/3 of the time.

Model 2

This is a very similar classification model to the first classification model, except it instead aggregates by author and looks at the average sentiment and ratios of words over all of the author’s tweets.

df_model = df_tweets[["author_id", "inbound", "sentiment", "pos", "neu"]]
df_model["inbound"] = df_model["inbound"].astype(float)
# already verified that taking max of inbound doesn't have any strange consequences, companies are still 0 and users are still 1
df_model = df_model.groupby("author_id").aggregate({'sentiment': 'mean', "pos": "mean", "neu" : "mean", "inbound" : "max"})
print(df_model["inbound"].mean())

features = df_model[["sentiment", "pos", "neu"]]
y = df_model[["inbound"]]
columns_to_use = ["sentiment", "pos", "neu"]
gc.collect()

C:\Users\takhi\AppData\Local\Temp\ipykernel_11608\3387183752.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_model["inbound"] = df_model["inbound"].astype(float)


0.9998456728378479

We can see from the above mean that when we group by the author, there is a very large imbalance in the number of company accounts and other users. This was not an issue previously, as companies accounted for around 45% of total tweets as seen by the mean in the previous model. We try to account for this by creating a weight column in the dataset which is (1 - relative proportion of class) so that the underrepresented class is weighted higher, and then assigning the weightCol field to this column in the logistic regression.

companyWeight = df_model["inbound"].mean()
df_model["weight"] = df_model["inbound"].apply(lambda x: 1 - companyWeight if x else companyWeight)
df_model

	sentiment	pos	neu	inbound	weight
author_id
10026	-0.123567	0.152667	0.639667	1.0	0.000154
100363	-0.659000	0.000000	0.769000	1.0	0.000154
10103	0.492600	0.224000	0.776000	1.0	0.000154
10221	0.000000	0.000000	1.000000	1.0	0.000154
10286	0.000000	0.000000	1.000000	1.0	0.000154
...	...	...	...	...	...
mediatemplehelp	0.354189	0.176534	0.777951	0.0	0.999846
nationalrailenq	0.019662	0.079801	0.831947	0.0	0.999846
sainsburys	0.252030	0.155401	0.787667	0.0	0.999846
sizehelpteam	0.426158	0.203766	0.776856	0.0	0.999846
sprintcare	0.369232	0.178120	0.792602	0.0	0.999846

699812 rows × 5 columns

model_sdf = spark.createDataFrame(df_model)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:474: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():
C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():

assembler = VectorAssembler(inputCols = columns_to_use, outputCol = "features")
pipe = Pipeline(stages = [assembler])
model_sdf = pipe.fit(model_sdf).transform(model_sdf)

train_sdf, test_sdf = model_sdf.randomSplit([0.8, 0.2], seed = random_seed)

Elastic Net

elastic = LogisticRegression(maxIter=10, featuresCol = "features", labelCol = "inbound", weightCol = "weight")
paramGrid = ParamGridBuilder() \
    .addGrid(elastic.regParam, [0, 0.333, 0.667, 1]) \
    .addGrid(elastic.elasticNetParam, [0, 0.333, 0.667, 1]) \
    .build()
crossval = CrossValidator(estimator=elastic,
                          estimatorParamMaps=paramGrid,
                          evaluator=MulticlassClassificationEvaluator(labelCol = "inbound"),
                          numFolds=3)
elastic = crossval.fit(train_sdf)

train_pred = elastic.transform(train_sdf)
test_pred = elastic.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.inbound))
train_metrics = MulticlassMetrics(train_label)
cm_train = np.array(train_metrics.confusionMatrix().toArray())

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.inbound))
test_metrics = MulticlassMetrics(test_label)
cm_test = np.array(test_metrics.confusionMatrix().toArray())

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training Accuracy: ", cm_train.trace() / cm_train.sum())
print("Test Accuracy: ", cm_test.trace() / cm_test.sum())
gc.collect()

Training Accuracy:  0.7206931756928009
Test Accuracy:  0.719706198495163





1923

print(elastic.bestModel.coefficients)
columns_to_use

[-5.807266295644642,5.351497214323913,1.354908711161738]





['sentiment', 'pos', 'neu']

Random Forest

rf = RandomForestClassifier(featuresCol = "features",
                            labelCol = "inbound", seed = random_seed, weightCol = "weight")
paramGrid = ParamGridBuilder() \
    .addGrid(rf.maxDepth, [4, 5, 6]) \
    .addGrid(rf.numTrees, [10, 20, 30]) \
    .build()
crossval = CrossValidator(estimator=rf,
                          estimatorParamMaps=paramGrid,
                          evaluator=MulticlassClassificationEvaluator(labelCol = "inbound"),
                          numFolds=3)
rf = crossval.fit(train_sdf)

train_pred = rf.transform(train_sdf)
test_pred = rf.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.inbound))
train_metrics = MulticlassMetrics(train_label)
cm_train = np.array(train_metrics.confusionMatrix().toArray())

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.inbound))
test_metrics = MulticlassMetrics(test_label)
cm_test = np.array(test_metrics.confusionMatrix().toArray())

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training Accuracy: ", cm_train.trace() / cm_train.sum())
print("Test Accuracy: ", cm_test.trace() / cm_test.sum())
gc.collect()

Training Accuracy:  0.8354234268967019
Test Accuracy:  0.8347760659261914





2030

Remarks

After aggregating by author, the models appear to have diverged significantly for the first time in the project. The Logistic Elastic Net does reasonably well again, with a 72% test accuracy better than the model’s performance on the individual tweets. However, the Random Forest performs much better, with an impressive test accuracy of 83%.

This result has some interesting implications, as it suggests that machine learning models are significantly better at identifying different types of entities on aggregations of larger corpuses of data. It also suggests random forests perform particularly well on these aggregations, so they may work well in other applications, like the particularly topical example of identifying bots on Twitter. For future reference, it would be interesting to train a Boosting model on this data as boosting is known to perform well in classification.

Model 3

This model tries to predict the sentiment of the last user tweet based on the first two company tweets in response. Rather than predicting a sentiment value, however, we classify that value into positive, negative, and neutral based on thresholds from the VADER GitHub and attempt to predict the class. We would have liked to have used more of the companies tweets (e.g. four or five), but as we can see from the summary statistics in the code block below, very few tweet groups even have 2 company tweets so requiring 4 or more would severely limit the sample size of the data and also reduce the external validity of the model as most tweet chains would not have 4 company tweets to predict on.

It is also important to note that we run 3 models in this case, as the LogisticRegression() object from Spark does not support multi-class classification, so we convert the dataset slightly to be a binary classification of positive or not instead. Therefore, the predictive accuracy of the two models are only comparable for this model. However, we also run a separate Random Forest for multi-class classification to see the predictive power of sentiment in this case.

df_tweets[df_tweets["inbound"] == False].groupby("tweet_group")["created_at"].count().reset_index()["created_at"].describe()

count    793905.000000
mean          1.570093
std           1.844123
min           1.000000
25%           1.000000
50%           1.000000
75%           2.000000
max         399.000000
Name: created_at, dtype: float64

df_model = df_tweets[["created_at", "inbound", "sentiment", "pos", "neg", "tweet_group"]]
df_user = df_model[df_model["inbound"] == True]
#getting the users last tweet
df_user = df_user.sort_values("created_at", ascending = False).groupby("tweet_group").head(1)
#classifying sentiment as positive, negative, or neutral based on thresholds from , About the Scoring
df_user["sent_class"] = df_user["sentiment"].apply(lambda x: 2.0 if x >= 0.05 else 1.0 if x > -0.05 else 0.0)

#getting first four tweets and joining into single DF
df_comp = df_tweets[df_tweets["inbound"] == False]
#tweet 1
df_comp1 = df_comp.sort_values("created_at", ascending = True).groupby("tweet_group").head(1)
#tweet 2
df_comp2 = df_comp.sort_values("created_at", ascending = True).groupby("tweet_group").head(2) \
    .sort_values("created_at", ascending = False).groupby("tweet_group").head(1)
df_model = pd.merge(df_comp1[["sentiment", "pos", "neg", "tweet_group"]],
                    df_comp2[["sentiment", "pos", "neg", "tweet_group"]], on = "tweet_group")
df_model = pd.merge(df_model, df_user[["sent_class", "tweet_group"]], on = "tweet_group")

features = df_model[["sentiment_x", "pos_x", "neg_x", "sentiment_y", "pos_y", "neg_y"]]
y = df_model[["sent_class"]]
columns_to_use = ["sentiment_x", "pos_x", "neg_x", "sentiment_y", "pos_y", "neg_y"]

df_model["sent_class_bin"] = df_model["sent_class"].apply(lambda x: 1.0 if x == 2.0 else 0.0)

print(df_model["sent_class"].value_counts())
print(df_model["sent_class_bin"].value_counts())

2.0    318963
0.0    243785
1.0    231094
Name: sent_class, dtype: int64
0.0    474879
1.0    318963
Name: sent_class_bin, dtype: int64

The classes appear to be fairly balanced, so we don’t use any weights for the regressions. Even in the binary case, the imbalance should not degrade the model very significantly.

model_sdf = spark.createDataFrame(df_model)

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:474: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():
C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\pandas\conversion.py:486: FutureWarning: iteritems is deprecated and will be removed in a future version. Use .items instead.
  for column, series in pdf.iteritems():

assembler = VectorAssembler(inputCols = columns_to_use, outputCol = "features")
pipe = Pipeline(stages = [assembler])
model_sdf = pipe.fit(model_sdf).transform(model_sdf)

train_sdf, test_sdf = model_sdf.randomSplit([0.8, 0.2], seed = random_seed)

del (df_model)
del (df_user)
del (df_comp)
del (df_comp1)
del (df_comp2)

Elastic Net

elastic = LogisticRegression(maxIter=10, featuresCol = "features", labelCol = "sent_class_bin")
paramGrid = ParamGridBuilder() \
    .addGrid(elastic.regParam, [0, 0.333, 0.667, 1]) \
    .addGrid(elastic.elasticNetParam, [0, 0.333, 0.667, 1]) \
    .build()
crossval = CrossValidator(estimator=elastic,
                          estimatorParamMaps=paramGrid,
                          evaluator=MulticlassClassificationEvaluator(labelCol = "sent_class_bin"),
                          numFolds=3)
elastic = crossval.fit(train_sdf)

train_pred = elastic.transform(train_sdf)
test_pred = elastic.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sent_class_bin))
train_metrics = MulticlassMetrics(train_label)
cm_train = np.array(train_metrics.confusionMatrix().toArray())

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sent_class_bin))
test_metrics = MulticlassMetrics(test_label)
cm_test = np.array(test_metrics.confusionMatrix().toArray())

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training Accuracy: ", cm_train.trace() / cm_train.sum())
print("Test Accuracy: ", cm_test.trace() / cm_test.sum())
gc.collect()

Training Accuracy:  0.6189349615379167
Test Accuracy:  0.6217193028932579





11913325

print(elastic.bestModel.coefficients)
columns_to_use

[-0.047259260321494204,0.14549436148852637,-0.8277941078978391,-0.3739309669818177,2.2802072169840932,-0.6475419210978054]





['sentiment_x', 'pos_x', 'neg_x', 'sentiment_y', 'pos_y', 'neg_y']

Random Forest

rf = RandomForestClassifier(featuresCol="features",
                            labelCol="sent_class_bin", seed=random_seed)
paramGrid = ParamGridBuilder() \
    .addGrid(rf.maxDepth, [4, 5, 6]) \
    .addGrid(rf.numTrees, [10, 20, 30]) \
    .build()
crossval = CrossValidator(estimator=rf,
                          estimatorParamMaps=paramGrid,
                          evaluator=MulticlassClassificationEvaluator(labelCol="sent_class_bin"),
                          numFolds=3)
rf = crossval.fit(train_sdf)

train_pred = rf.transform(train_sdf)
test_pred = rf.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sent_class_bin))
train_metrics = MulticlassMetrics(train_label)
cm_train = np.array(train_metrics.confusionMatrix().toArray())

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sent_class_bin))
test_metrics = MulticlassMetrics(test_label)
cm_test = np.array(test_metrics.confusionMatrix().toArray())

print("Training Accuracy: ", cm_train.trace() / cm_train.sum())
print("Test Accuracy: ", cm_test.trace() / cm_test.sum())
gc.collect()

Training Accuracy:  0.6221633524896073
Test Accuracy:  0.6239269337269224





3746

Random Forest Multi-Class

rf = RandomForestClassifier(featuresCol="features",
                            labelCol="sent_class", seed=random_seed)
paramGrid = ParamGridBuilder() \
    .addGrid(rf.maxDepth, [4, 5, 6]) \
    .addGrid(rf.numTrees, [10, 20, 30]) \
    .build()
crossval = CrossValidator(estimator=rf,
                          estimatorParamMaps=paramGrid,
                          evaluator=MulticlassClassificationEvaluator(labelCol="sent_class"),
                          numFolds=3)
rf = crossval.fit(train_sdf)

train_pred = rf.transform(train_sdf)
test_pred = rf.transform(test_sdf)

train_label = train_pred.rdd.map(lambda x: (x.prediction, x.sent_class))
train_metrics = MulticlassMetrics(train_label)
cm_train = np.array(train_metrics.confusionMatrix().toArray())

test_label = test_pred.rdd.map(lambda x: (x.prediction, x.sent_class))
test_metrics = MulticlassMetrics(test_label)
cm_test = np.array(test_metrics.confusionMatrix().toArray())

C:\Users\takhi\OneDrive - PennO365\Documents\CIS 545\venv2\lib\site-packages\pyspark\sql\context.py:157: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
  warnings.warn(

print("Training Accuracy: ", cm_train.trace() / cm_train.sum())
print("Test Accuracy: ", cm_test.trace() / cm_test.sum())
gc.collect()

Training Accuracy:  0.42176543087449886
Test Accuracy:  0.42065459407976485





21

Remarks

The overall results of this model specification are not particularly promising. The Elastic Net and Random Forest predict a positive/not positive last tweet from the user correctly more often than not, but their test accuracy is an uninspiring 62%. This suggests that a user’s satisfaction may be slightly influence by the sentiment of the company they interact with, but there is either a very large amount of noise or a different predictor (perhaps one related to their specific issue) that is omitted which is important for this prediction.

The multi-class model, somewhat predictably, does worse than the binary predictions. This makes sense as features do not appear to be very strong predictors, and the overall prediction task is made more difficult by having more classes. Similarly, there may be omitted features that allow for a better prediction.

Description of Challenges Faced / Obstacles Faced

One of the main challenges we faced was figuring out how to incorporate tweet chains/trees into our dataframe. As all we were initially provided with were tweets and the IDs of tweets that replied to the tweet or were the parent tweet, it was initially challenging to figure out how to create the tweet groups. We then realized we could treat each tweet as a node, and connections via reply as edges in a graph, and run BFS. We initially started with an iterative approach through the dataframe, but through trial and error found that doing self joins worked best for our solution due to a very significant reduction in runtime. This was critical for our analysis, as looking at the conversation-level data was a central part of our predictive models.

Additionally, the sheer size of the dataset proved to be a very large obstacle, as the Colab’s hosted runtime has a very limited RAM and CPU and running models and aggregations on the dataset often tested that RAM. This was made more difficult by the fact that exceeding the RAM capabilities meant having to run cleaning commands in the notebook again, which was very time-consuming (also a result of the size). For example, tokenizing and calculating the sentiment for each tweet would take more than 20 minutes in total, and this was before we could even run EDA or models. In fact, even just loading the cleaned dataset (saving resources from running the cleaning, a strategy I tried to just get the output of each model) and then running the first Spark classification model creates crashes in Colab, likely because it is one of the few models that uses every row of the dataset. However, we were able to get a breakthrough in running the models in PyCharm by manually downloading Spark and identifying the appropriate filepaths for Spark and a Java JDK, which let us adapt the PySpark initalization code for local use and eventually let us run the models in PyCharm, avoiding the RAM and CPU limitations of Colab.

Potential Next Steps / Future Direction

One potential next step is to incorporate emojis into our sentiment analysis. Emojis, like text, have the potential to be categorized into positive and negative sentiments, and we think it would be interesting to look into whether tweets with emojis have any significant differences when compared to non-emoji tweets.

Additionally, analysis on word frequency could be interesting, especially considering the unique medium of Twitter. Perhaps a much wider dataset with the frequency of some n most common words could predict sentiment better. More data on the individuals in the tweets could also shed some light for prediction—perhaps users with more followers were more likely to have a neutral or positive wording because of that following, or perhaps it gave them confidence to be more negative. Users with more followers may also have faster response times and be satisfied with their last tweet more often if companies treat them with more importance.

It would also be interesting to compare the NLTK VADER sentiment with the sentiment analysis derived from other techniques, like the recent neural network XLNet, which used autoregressive techniques and an incredibly high amount of GPU time to achieve high performance in natural language processing. Perhaps these separate measures have more specificity in their data, or are able to more accurately identify sentiment within the limited space of a tweet. Overall, though, the VADER analysis seemed to be sufficient for the analysis (particularly given the predictive strength in classification of company vs non-company) as the EDA showed largely what we expected—more positive tweets from companies while users were more neutral across the board. We did expect some more negative sentiment on occasion, which may be an angle for improvement, as the VADER analysis may not be particularly adept at identifying short-form internet negativity.

Overall, our analysis and prediction models found both interesting conclusions and shortcomings with the data. Many things were either not predicted particularly well or barely improved upon predicting with the overall mean every single time. However, certain areas showed promising results: a company’s first response being more positive was generally associated with a more positive second tweet from the user, and the sentiment of a user’s tweet (and especially tweets over time) were quite good predictors of whether a tweet was from an individual or a corporate entity. Further analysis of these conclusions, perhaps with an updated dataset or different sentiment analysis, could provide useful conclusions for corporate Twitters and automated classification of users.

Machine Learning | Akhilesh Tumu

Learning-Based Sketching

Twitter Analytics Project

Introduction/Background

Data Cleaning / Data Wrangling

Adding columns for tweet tree identification and tree depth

Tweet text columns (tokenization, sentiment analysis)

Exploratory Data Analysis

Company vs. Non-Company Tweet Sentiments

Average Sentiments of Company Tweets

Sentiments of First Non-Company Tweets

Difference in Sentiment between Company and User First Tweets By Company

Sentiment of User’s Last Tweet

Change in Sentiment from First Tweet to Last Tweet for Non-Companies

Modeling: Regression

PySpark initialization

Model 1

Elastic Net

Random Forest

Remarks

Model 2

Elastic Net

Random Forest

Remarks

Model 3

Elastic Net

Random Forest

Remarks

Model 4

Elastic Net

Random Forest

Remarks

Modeling: Classification

Model 1

Elastic Net

Random Forest

Remarks

Model 2

Elastic Net

Random Forest

Remarks

Model 3

Elastic Net

Random Forest

Random Forest Multi-Class

Remarks

Description of Challenges Faced / Obstacles Faced

Potential Next Steps / Future Direction