Downloading Tweets Using Keyword Searches

Before starting with Mozdeh, you will need a set of relevant Twitter queries. The process to build a useful set of queries is:

1. Brainstorm a set of potentially relevant queries: Create a list of queries that you believe should match relevant tweets. The queries can be keywords or phrases that describe your topic and, as far as possible, do not also describe irrelevant topics. Enclose phrases in quotes and save the queries in a plain text file. It is particularly important to use a plain text file (e.g., in Windows Notepad) because if you have quotes in a word processor document then it will change the type of quotes from straight to smart, which the Twitter Search will not recognise.
2. Check each query for irrelevant content and remove if almost all content is irrelevant, or refine the query if there is some irrelevant content. Test each query but submitting it to Twitter Search (https://twitter.com/search-home) and checking the results for false matches. Queries can be modified by adding extra terms specific to the topic or by subtracting terms matching irrelevant topics. For example, if the initial search was tiger for the big cat then the refinement might be tiger cat or even tiger –woods, both of which should get a higher percentage of relevant queries than the initial query.
3. Brainstorm for additional queries to add to the set
4. Repeat from 2 until the set of queries seems to be stable and satisfactory.

Once this is complete, the queries can be used to gather the data for analysis, as described below.

Try the list of search operators here too - and beware that the web search results may not work in the same way as Mozdeh search results. For example, searching for a username returns some of the users tweets in the results online but seems not to in Mozdeh - the from:user command is needed instead.

See Twitter for a complete set of instructions for allowable queries.

Gather matching tweets with Mozdeh

This YouTube video illustrates downloading and installing Mozdeh, collecting tweets, searching, filtering and keyword analysing tweets.

Now Mozdeh can be used with your queries. First, Mozdeh must be installed on the computer that will run the analysis. To do this, download the appropriate version from here. Start Mozdeh and follow the instructions about selecting a folder in which to store your data. Mozdeh will ask to create a folder called rss_data on your computer. This folder will be initially empty but you will eventually populate this with new subfolders, one for each Twitter project.

Mozdeh can help with piloting the original queries. Start Mozdeh, enter a name for the pilot test (e.g., Tiger pilot1) and click the Start New Project button.

Then enter the queries in the Data Collection screen. Make sure that queries matching too many irrelevant tweets are rejected, and that queries matching some irrelevant tweets are refined to eliminate most irrelevant tweets. For example, a study to investigate how the words kiss and hug are used in Twitter to express affection started with these two terms as queries (below left). After pilot testing the queries, terms were subtracted from them to remove lots of unwanted matches (below right). For example, subtracting ass removed many instances of the phrase "kiss my ass", removing 104.1 excluded references to the Kiss 104.1 radio station and subtracting bora removed lots of spam related to a TV meme at the time of checking.

->

Once the queries are entered, click the Search Twitter Once button. Or, if this is a full-scale study, click the Search Twitter Continually button and then click again to stop it at the end of your data collection period.

If running a pilot test (Search Twitter Once) you get the chance to change the maximum number of tweets per query, but beware that increasing the maximum slows the data collection.

You will be asked for your "bearer token". If you don't have one already, sign up here as a Twitter developer and follow the instruction until you get the page below, showing your bearer token, as below.

Enter your bearer token into Mozdeh and click OK. The Mozdeh searches are slow so you will need to wait a few minutes at least.

When it has finished, it will ask a series of questions – please click OK or give the suggested answer to these questions and then you should get the main search screen. Now follow from the second or third step in the instructions in order to analyse the tweets.

Please see also: Thelwall, M. (2015). Evaluating the comprehensiveness of Twitter Search API results: A four step method. Cybermetrics,18-19, p1.

Made by the Statistical Cybermetrics and Research Evaluation Group at the University of Wolverhampton during the CREEN and CyberEmotions EU projects.