Measuring Job Loss during the Pandemic Recession in Real Time with Twitter Data

Finance and Economics Discussion Series Federal Reserve Board, Washington, D.C. ISSN 1936-2854 (Print) ISSN 2767-3898 (Online) Measuring Job Loss during the Pandemic Recession in Real Time with Twitter Data Anbar Aizenman, Connor M. Brennan, Tomaz Cajner, Cynthia Doniger, Jacob Williams 2023-035 Please cite this paper as: Aizenman, Anbar, Connor M. Brennan, Tomaz Cajner, Cynthia Doniger, and Jacob Williams (2023). “Measuring Job Loss during the Pandemic Recession in Real Time with Twitter Data,” Finance and Economics Discussion Series 2023-035. Washington: Board of Governors of the Federal Reserve System, https://doi.org/10.17016/FEDS.2023.035. NOTE: Staff working papers in the Finance and Economics Discussion Series (FEDS) are preliminary materials circulated to stimulate discussion and critical comment. The analysis and conclusions set forth are those of the authors and do not indicate concurrence by other members of the research staff or the Board of Governors. References in publications to the Finance and Economics Discussion Series (other than acknowledgement) should be cleared with the author(s) to protect the tentative character of these papers.

Measuring Job Loss during the Pandemic Recession in Real Time with Twitter Data* Anbar Aizenman Connor M. Brennan Anbar.Aizenman@gmail.com Connor.M.Brennan@frb.gov Tomaz Cajner Cynthia Doniger Tomaz.Cajner@frb.gov Cynthia.L.Doniger@frb.gov Jacob Williams Jacob.M.Williams@frb.gov May 22, 2023 Abstract WepresentanindicatorofjoblossderivedfromTwitterdata,basedonafine-tuned neuralnetworkwithtransferlearningtoclassifyifatweetisjob-lossrelatedornot. WeshowthatourTwitter-basedmeasureofjoblossiswell-correlatedwithandpredictive of other measures of unemployment available in the official statistics and withtheaddedbenefitsofreal-timeavailabilityanddailyfrequency. Thesefindings are especially strong for the period of the Pandemic Recession, when our Twitter indicator continues to track job loss well but where other real-time measures like unemploymentinsuranceclaimsprovidedanimperfectsignalofjobloss. Additionally, wefindthatourTwitterjoblossindicatorprovidesincrementalinformationin predictingofficialunemploymentflowsinagivenmonthbeyondwhatweeklyunemploymentinsuranceclaimsoffer. Keywords: JobLoss,NaturalLanguageProcessing,NeuralNetworks. JELClassification: J63. *WethankElizabethVrankovichforhelpinobtainingTwitterdataandAndersonMonkenfor providing useful insight regarding machine learning implementation. The analysis and conclusionssetfortharethoseoftheauthorsanddonotindicateconcurrencebyothermembersofthe researchstaffortheBoardofGovernors.

1 Introduction Official statistics on unemployment and job displacement lag behind the actual state of the labor market. The Current Population Survey (CPS), which provides detailed information on job loss in the United States as well as underlies the official statistics on employment, is typically released during the first week after the end of each month, and thus lags about four weeks behind economic conditions,andonlyshowsasnapshotofwhenthesurveywasconducted. Inaddition, statistics may be distorted by sudden catastrophic events, such as the shutdowns at the onset of the COVID-19 pandemic. For example, Ward and Edwards (2020) and Rothbaum and Bee (2021) document that shutdowns non-randomly altered patterns of non-response to official government surveys, such as the CPS. State unemployment insurance (UI) claims are released weekly with little lag and at a weekly frequency. However, UI claims likely provided a distorted picture of job losses during the Pandemic Recession because of processing delays, errant claim duplication,andfraud(Cajner,Figura,Price,RatnerandWeingarden,2020). Meanwhile, knowledge about the state of the labor market in as close to real time as possible is valuable to policy makers and markets. For example, officialunemploymentstatisticsarelegislatedastriggersin“automaticstabalization” policies, such as UI expansions. As such, prescience regarding the likely realization of these statistics can help policy makers ensure that appropriate resources are in place. As another example, Boyd, Hu and Jagannathan (2005) document movementsinfinancialmarketsassociatedwithsurprisesintheseofficialreleases. Inthispaper,weconstructlowlatencyandhigh-frequencyproxiesforjobloss derived from Twitter data. We then evaluate the performance of these measures over the period from January 1, 2015 to March 18, 2023, which includes the Pandemic Recession. Twitter is a popular social media platform that allows users to post short messages known as “tweets”. We hypothesize that tweets discussing job loss and unemployment can provide timely information on the current state ofthelabormarket. The approach is well precedented. Specifically, we build on the methodology 1

of Antenucci, Cafarella, Levenstein, Ré and Shapiro (2014), who used job-related phrases in tweets to create indexes of labor market flows. In particular, they use job-related phrases in Tweets to create indexes of job loss, job search, and job posting and find that the job loss index can track and predict initial claims for unemployment insurance better than official data and can capture the effects of economic shocks in real time. Similarly, Proserpio, Counts and Jain (2016) use keyword searches refined with a dictionary-based approach to develop a behavioral macroeconomic model that predicts levels of the U.S. unemployment. They found that their psychological well-being measures were leading indicators, predicting economic indices weeks in advance with higher accuracy than traditional forecasttechniques. However, we show that the signal contained in tweet data can be improved by refining tweets to those germane to actual job losses of the tweeting individuals and their social network. Considering tweets as job-loss related based on containing keywords alone (such as “lost job”, “laid off”, or “pink slip”) is not sufficient as they may contain posts unrelated or not reflective of real job losses, such as jokes or the job losses of celebrities. Our method contributes to the literature by using machine learning to filter out this noise. We use natural language processing (NLP) techniques to identify and analyze tweets related to job loss from 2015 to 2023, covering the period before the Pandemic Recession, the recession, and its aftermath. Our Twitter-based measures of job loss demonstrate a highdegreeofcorrelationwithofficialstatisticsfromtheCPS,aswellasUIclaims and JOLTS layoffs and could be constructed in near to real time. We then test if our Twitter-based measure of job loss can provide additional explanatory power in predicting current-month employment-to-unemployment flows in CPS above whatinitialclaims—thetypicalreal-timeindicator—provide. Our main contributions are as follows: First, we corroborate the feasibility and validity of using Twitter data to measure job loss. Second, we provide new insights into the dynamics, heterogeneity, and real-time traceability of job loss by gender. Third, we explore the potential applications and limitations of our approachforfutureresearchandpolicy. 2

2 Data With the access to the historical Twitter data through the Twitter Enterprise Full- Archive Search API, we obtained a sample of tweets based on search terms included in a tweet’s text. In particular, our sample is based on the following two search criteria, which are intentionally kept simple: i) “laid off” or “layoff”, and ii) “lost job”. For the first criterion we match on the exact keywords, while for the second we match tweets where “lost” and “job” are no more than six tokens apart (e.g., “lost my job”). Our data are daily and go from January 1, 2015 to March 18, 2023.1 We restrict the sample to tweets from the U.S., that is, tweets that tagged a place in the U.S. or tweets from a user who listed the U.S. as where they reside in their Twitter profile. We exclude retweets, because we are primarily interested in people reporting their own personal job loss. Table 1 shows the count of tweets containing the job-loss related keywords (Table 2 shows the exact TwitterAPIqueriesused). Intotal,wehave2.1milliontweets. Table1: FrequencyofDifferentTwitterQueries(January1,2015toMarch18,2023) QueryTerm QueriedTweets laidofforlayoff 1,220,164 laidoff 901,878 layoff 318,286 lostjob 919,369 Source: Twitterandauthors’calculations. Wefurtherrefineourinitialquerybyfilteringouttweetspostedbyuserswith afollower-to-followingratioeithergreaterthan10orlowerthan0.1. Therationale is that we would like to focus on tweets of “typical” individuals and thus we aim toremovetweetslikelypostedbylargecorporationsandbots. Afterapplyingthis restriction,oursamplesizeisreducedfrom2.1millionto1.7million. 1WhileTwitterdatahaveafewmoreyearsofearlierhistory, wefoundthattheperiodbefore 2015hasamorelimitednumberoftweetsavailableandisthuslesssuitableforouranalysis. 3

Table2: TwitterAPIQueries Group Query ("laidoff"ORlayoff) Layoff (profile_country:usORplace_country:us) -is:retweet ("lostjob"~6) Lostjobproximity (profile_country:usORplace_country:us) -is:retweet 3 Analysis of Raw Job-Loss Twitter Data We proceed with the analysis of the raw job-loss Twitter data. In Figure 1, we plot the weekly number of tweets for our two search criteria, together with official statistics on initial claims for unemployment insurance (UI claims) and labor market flows from employment-to-unemployment (EU flow) as measured in the Current Population Survey (CPS) data. For a more direct comparison, we use non-seasonallyadjusteddata.2 As can be seen in Figure 1, UI claims spiked roughly by a factor of 30, rising from levels around 200,000 in February 2020 to an unprecedented 6.2 million in the week ending April 4, 2020. Interestingly, the number of tweets containing the phrase "laid off" rose by almost exactly the same relative amount, while the numberoftweetswith"lostjob"rosesomewhatless. AsTwitterdataareavailable inreal-time,theyprovidedsometimeadvantagewhencomparedtoUIclaims. In particular, the UI claims data for the week ending March 21 — which showed an unprecedented tenfold increase to 3.3 million — were published on March 26. In contrast, the rapid increase in job loss was evident in Twitter data already as of Monday, March 16, providing a time advantage of 10 days. Moreover, UI claims were overstating true job losses, especially in the period from April 2020 to about 2The large swings observed during the COVID pandemic introduced several challenges for seasonally adjusting data, especially for methods that rely on multiplicative adjustment. Additionally,typicalseasonalpatternswereswampedbytheobservedmovementsduringtheCOVID pandemic. 4

April 2021, as evidenced by CPS EU flow being substantially below UI claims. Interestingly, both Twitter-based job-loss measures followed CPS EU flow data relatively well during that period, suggesting they provide useful information aboutjobloss. Figure1: Job-LossTweets,UIClaims,andCPSEUFlow(Feb. 2020=100) 3000 3000 2000 2000 1000 1000 500 500 100 100 Jan. 1, 2020 Jan. 1, 2021 Jan. 1, 2022 Jan. 1, 2023 Tweets: laid off Tweets: lost job UI claims CPS EU flow Note: Non-seasonallyadjusteddataplotted. Ratioscaleused. Job-losstweetsareweeklysumof tweets,indexedtotheiraverageinFebruary2020beingequalto100. Source: Twitter,BLS,CPS,andauthors’calculations. 3.1 Gender Analysis Next, we construct Twitter-based job loss measures by gender. To do so, we first download counts of names by gender and year going back to 1900 for the U.S. 5

from the Social Security Administration website.3 The data contain all names except for those with fewer than five occurrences in any given year. We sum names bygenderfrom1900to2021andtoexcludenamesthatareusedforbothgenders, we only keep names that have a 95 percent or greater occurrence in a single gender following the method from Mislove, Lehmann, Ahn, Onnela and Rosenquist (2011). To clean the Twitter name field, we remove titles from the name that may be mistaken for first names as well as replace characters often used in place of letters in names.4 We then extract the first word from the name field and match it tothe95-percentthresholdnamegenderlistfromabove. Overall,wewereableto assigngendertoabout50percentoftweetsinoursample. One notable empirical pattern observed during the pandemic recession was that the increase in the unemployment rate was larger for women than men, in sharp contrast with the typical pattern observed during recessions from 1980 to 2010. In particular, while the unemployment rate for both men and women equaled 3.5 percent in January 2020, it increased in April 2020 to 16.2 percent for women and 13.5 percent for men. By December 2020, the unemployment rate for both genders was again equal at 6.7 percent. In order to examine whether the same pattern exists also in Twitter-based job loss data, Figure 2 plots job-loss tweets by gender (because the sample is now smaller given that we are able to assign gender to only about 50 percent of tweets, we join tweets for both of our search terms together). Interestingly, job loss tweets from women jumped more thanthosefrommenintheearlystagesofpandemic,consistentwithCPSEUflow databygender. Additionally,joblosstweetsforbothgenderswereroughlyequal from early 2021 onward, consistent with the convergence in CPS EU flow data as well(andtheconvergenceinpublishedunemploymentrates). 3Seehttps://www.ssa.gov/oact/babynames/names.zip. 4The excluded words include: king, prince, the, dr, sir, mr, ms, mrs, lil, rev, fr, father, queen, princess, lord, mr., ms., mrs., brother, sister, little, doc, sir, and professor. The characters and substitutionsinclude: $:s,@:a,4:a,!:l,8:b,0:o,3:e,/:l,and|:l. 6

Figure2: Job-LossTweetsandCPSEUFlowbyGender(Feb. 2020=100) 3000 3000 2000 2000 1000 1000 500 500 100 100 Jan. 1, 2020 Jan. 1, 2021 Jan. 1, 2022 Tweets: Men Tweets: Women CPS EU flow: Men CPS EU flow: Women Note: Non-seasonallyadjusteddataplotted. Ratioscaleused. Joblosstweetsareweeklysumof tweets,indexedtotheiraverageinFebruary2020beingequalto100. Source: Twitter,CPS,andauthors’calculations. 4 Using Machine Learning to Obtain More Precise Job Loss Twitter Measure So far we have been using raw counts of job-loss tweets. While these raw counts basedonrelativelysimplesearchtermsdoaprettygoodjobatmeasuringjobloss in real time as argued in the previous section, the Twitter queries we gathered might not yield exclusively tweets that are capturing job loss. To obtain a more precise job-loss Twitter measure, we proceed by using a machine learning model forNLPcalled“BERT”orBidirectionalEncoderRepresentationsfromTransform- 7

ers. BERT is a pre-trained neural network developed by researchers at Google in 2018anddescribedinDevlin,Chang,LeeandToutanova(2018). BERT’skeytechnicalinnovationisapplyingbidirectionaltrainingtolanguagemodelling: instead ofreadingtextsequentially(left-to-rightorright-to-left),BERTreadstheentiresequence of wordsat once, allowing the modelto learn the contextof a word based on all of its surroundings. Understanding the context of a word is crucial for determining whether a tweet is actually related to job loss or not despite containing ajob-lossrelatedphrase. Forexample,considerthetweet: “Noonelosttheirjob”. Contextisneededtorecognizethat“noone”isimportantwheninterpreting“lost theirjob”. Otherwise,the modelwouldincorrectly classifythetweet asrelated to anactualjoblosswhenitisnot. Inadditiontoitsbidirectionaltraining,BERTalso comes pre-trained on all of Wikipedia (2.5 billion words) and Book Corpus (800 millionwords),givingitafairunderstandingofsentencestructureandwriting. Wefine-tunetheBERTmodelwithanadditionaloutputlayerthatwillclassify tweetsasrelatedtoanactualjoblossornot. Wecreatetrainingandvalidationdata by labeling 6,011 randomly selected tweets from our query. In addition to binary labeling whether tweets generally relate to job loss or not, we also mark whether a tweet is present-tense, meaning relating to a job loss that just happened, and whetheratweetrelatestoacelebrityratherthanaeverydayworker(oftenpeople tweetaboutfamousindividualslosingtheirjobandwewouldliketoexcludesuch tweets). Tweetsarelabeleda“real”joblossiftheyarebothrelatedtojoblossand present tense but not related to a famous person, for a total of 37.5 percent of our labeled data. We then split 70 percent of our data to a training set, 15 percent to a testset,and15percenttoavalidationset. Figure 3 shows both the model’s decreasing loss over successive rounds of training and a confusion matrix of our model’s classifications on our validation set. Over each epoch of training, weretain a given epoch oftraining if the validation loss of that epoch is better than the validation loss of the previously retained epoch. Early stopping of the training is executed after 15 consecutive epochs of non-decreasing validation loss relative to the last retained epoch of training. In total, we train 118 epochs into our outer layer, and the model is able to correctly 8

predict approximately two-thirds of the time. Table 3 presents the model’s classification report. With accuracy of approximately 70 percent, our model is considered acceptable by most machine learning standards. Given that our groupings are a bit unbalanced, our reasonable F1-scores give us confidence in our model’s performance. Figure3: ModelTrainingandEvaluation Table3: ModelClassificationReport TrueLabel Precision Recall F1Score Support 0 0.760 0.687 0.722 521 1 0.622 0.703 0.660 381 MacroAverage 0.691 0.695 0.691 902 WeightedAverage 0.702 0.694 0.696 902 Accuracy 0.694 Figure 4 shows raw job-loss tweet counts together with the data based on the BERT classification algorithm. Note that in absolute numbers (not shown), the BERT-classified job-loss tweet series are always below raw job-loss tweets counts(byaboutonethird). However,Figure4plotsthedataindexedtoFebruary 2020 being equal to 100. The results show that the BERT-classified series actually jumped more during the Pandemic Recession than the raw data. On the other hand, during the period of late 2022 and early 2023, the BERT-classified series is below the raw data. Note that a spike in Twitter-based job loss measure during 9

thatperiodtoalargeextentreflectslayoffsinthetechsector. TheBERTalgorithm correctly predicts that many of those tweets are not about individual people reporting their own job loss, but rather about many tweets discussing relatively isolated layoff events in the tech sector. However, even the BERT-classified series seems somewhat elevated to information we have from UI claims and CPS data, suggesting that further improvements in making the Twitter-based job-loss measuremoreprecisethroughafurtherfine-tunedBERTmodelarepossible. Figure4: KeywordSearchvs. BERT-ClassifiedTweets(Feb. 2020=100) 1500 1500 1000 1000 500 500 100 100 Jan. 1, 2020 Jan. 1, 2021 Jan. 1, 2022 Jan. 1, 2023 Keyword search BERT classification Note: Non-seasonallyadjusteddataplotted. Ratioscaleused. Valueareindexedtotheiraverage inFebruary2020beingequalto100. Source: Twitterandauthors’calculations. Table4showshowwellourTwitterjob-lossindicatorcorrelatestoofficialmeasuresofunemploymentflowscomparedtocountsoftweetscontainingkeywords 10

for job loss. Over our whole sample, we note that our BERT-classified tweets are markedly more correlated to the CPS, JOLTS, and UI claims data. This suggests that our indicator more stably tracks job loss over a wider horizon. We also note thatthesecorrelationsjumpduringtheCOVIDpandemic,suggestingthatTwitter may provide a useful alternative measure of job losses during crises, when other datalikeUIclaimsmayhaveissues. Table4: CorrelationsofTwitterjob-lossindicatorstoofficialjob-lossindicators BERTClassifiedTweets TweetsbyJobLossKeywords WholeSample COVID WholeSample COVID CPS 0.943 0.986 0.886 0.983 JOLTS 0.859 0.898 0.814 0.893 UIclaims 0.948 0.971 0.906 0.980 Notes: CPS refers to the Employed to Unemployed flow from the Current Population Survey, from the week containing the twelfth of the month from last month to the week containingthetwelfthofthemonthinthecurrentmonth. JOLTSreferstomonthlylayoffs inJOLTS.UIclaimsreferstounemploymentinsurancenumbersreleasedweekly. COVID referstocorrelationsovertheperiodJanuary2020toMarch2020whileWholeSamplerefers tocorrelationsovertheperiodJanuary2015toMarch2023. Dataareofmonthlyfrequency. Source: Twitter,BLS,CPS,andauthors’calculations. 5 Forecasting EU Flows With Job-Loss Twitter Data We estimate four regression models with the monthly data from January 2015 to March 2023 to examine the relationship between employed-to-unemployed CPS flows and our Twitter job-loss measure. The predictors include initial claims, tweetsbyjob-losskeywords,thelagofemployed-to-unemployedCPSflows,and tweets classified by BERT as job-loss related. The first model uses only UI claims data and lagged EU flows. The second model includes all predictors except the BERT-classifiedtweets. Thethirdmodelincludesallpredictors. Thefourthmodel includestheBERT-classifiedtweetsbutexcludesthetweetsbyjob-losskeywords. Resultsare presentedinTable5. We findthatthe tweetsbyjob-losskeywords are 11

insignificant in the second model, suggesting that they contain too much noise from unrelated topics and are dominated by initial claims as a predictor. However, in the third model, both the BERT classified tweets and the tweets by jobloss keywords are significant with the coefficient of BERT-classified tweets being positive whereas the coefficient for job-loss keywords is negative, indicating that the BERT-classification extracts the job-loss signal from the noisy tweets and improves the forecasting power of CPS flows. In contrast, in the fourth model, only the BERT-classified tweets are significant and positive, confirming their informationalvalueinpredictingemployed-to-unemployedCPSflows. Table5: EntireSample(January2015toMarch2023) (1) (2) (3) (4) Excl. Twitter Excl. BERT-classified All Excl. keywords ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ UIclaims 0.602 0.578 0.495 0.549 (19.81) (15.88) (11.75) (14.45) ∗∗∗ ∗∗∗ ∗∗∗ ∗∗∗ LagofEUflows -0.328 -0.327 -0.323 -0.325 (-4.61) (-4.77) (-6.69) (-5.10) ∗∗∗ Keywordtweets 0.0369 -0.514 (1.03) (-2.92) ∗∗∗ ∗ BERT-classified 0.606 0.0722 (3.29) (1.91) Constant -0.00266 -0.00367 -0.00570 -0.00470 (-0.15) (-0.21) (-0.34) (-0.27) Observations 97 97 97 97 R2 0.715 0.717 0.743 0.721 AIC -65.88 -64.54 -71.86 -66.14 tstatisticsinparentheses ∗ p <0.10,∗∗ p <0.05,∗∗∗ p <0.01 12

6 Conclusions ThispaperdevelopsameasureofjoblossusingTwitterdata,basedonamachine learning algorithm called BERT that classifies Tweets as job-loss related or not. We have shown that this measure can track job loss and predict EU flows at high frequencyandinrealtime,andcancapturetheeffectsofeconomicshockssuchas theCOVID-19pandemic. Our paper contributes to the literature on using social media data to measure and analyze labor market, and provides a valuable tool for policy makers and researchers who need timely and accurate indicators of job loss. However, there areseverallimitationsandchallengesthatneedtobeaddressedinfutureresearch. Forexample: • Howtoimprovetheidentificationandextractionofjob-relatedphrasesfrom tweets,usingnaturallanguageprocessingandmachinelearningtechniques • How to extend the analysis to other countries and regions, taking into accountthedifferencesinlanguage,culture,andlabormarketinstitutions • How to incorporate other sources of social media data, such as Facebook, LinkedIn,orReddit,tocreateamorecomprehensivepictureoflabormarket dynamics. References ANTENUCCI, D., CAFARELLA, M., LEVENSTEIN, M., RÉ, C. and SHAPIRO, M. D. (2014). Using Social Media to Measure Labor Market Flows. NBER Working Paper 20010,UniversityofMichigan. BOYD, J. H., HU, J. and JAGANNATHAN, R. (2005).Thestockmarket’sreactionto unemployment news: Why bad news is usually good for stocks. The Journal of Finance,60(2),649–672. 13

CAJNER, T.,FIGURA, A.,PRICE, B.M.,RATNER, D.andWEINGARDEN, A.(2020). ReconcilingUnemploymentClaimswithJobLossesintheFirstMonthsoftheCovid-19 Crisis.Fedsworkingpaper,FederalReserveBoardofGovernors. DEVLIN, J., CHANG, M.-W., LEE, K. and TOUTANOVA, K. (2018). Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprintarXiv:1810.04805. MISLOVE, A., LEHMANN, S., AHN, Y.-Y., ONNELA, J.-P. and ROSENQUIST, J. (2011). Understanding the Demographics of Twitter Users. Fifth International AAAIConferenceonWeblogsandSocialMedia,5(1). PROSERPIO, D., COUNTS, S. and JAIN, A. (2016). The psychology of job loss: using social media data to characterize and predict unemployment. In WEB- Sci’16: Proceedings of the 8th ACM Conference on Web Science, WEBSCI, Seattle, WA:ACM,pp.223–232. ROTHBAUM, J. and BEE, A. (2021). Coronavirus Infects Surveys, Too: Survey NonresponseBiasandtheCoronavirusPandemic.Tech.rep.,USCensus. WARD, J.andEDWARDS, K. A.(2020).StatisticsintheTimeofCoronavirus: COVID- 19-RelatedNonresponseintheCPSHouseholdSurvey.Tech.rep.,Rand. 14