⌙ Featured , Google Apps Script , Google Sheets , Twitter

Twitter Archiving Google Spreadsheet TAGS v5

Martin Hawksey

Update: TAGS v6.0 is here! You can jump straight to the new site. If you are using version 5.x here are instructions on how to upgrade (there is an old Sheets version of TAGS which is recommended for use with TAGSExplorer).

For a couple of years now to support my research in Twitter community analysis/visualisation I’ve been developing my Twitter Archiving Google Spreadsheet (TAGS). To allow other to explore the possibilities of data generated by Twitter I’ve released copies of this template to the community.
In September 2012 Twitter announced the release of a new version of their API (the spreadsheet uses this to request data from Twitter). Around the same time Twitter also announced that the old version of their API would be switched off in March 2013. This has required some modification of TAGS to work with the new API. The biggest change for TAGS is that all requests now need authenticated access.
So here it is:

*** Twitter Archive Google Spreadsheet – TAGS v5.1 ***
[If the first link doesn’t work try Opening this Spreadsheet and File > Make a copy]

Important: Uldis Bojars (@CaptSolo) has highlighted an issue with TAGS V5.0 which meant it would not page all the results. This has been fixed with the release of v5.1. If you are running version 5.0 of TAGS it’s highly recommended you update the code.

Note! Search API isn’t 100% reliable:

We find that the search API over-represents the more central users and does not offer an accurate picture of peripheral activity; we also find that the bias is greater for the network of mentions – González-Bailón, Sandra, et al. “Assessing the bias in communication networks sampled from twitter.” Available at SSRN 2185134 (2012).

Instructions for setting up TAGSv5

Instructions are included on the Readme/Settings sheet of the template. If you are having problems it’s worth checking Instructions for Tags v5.0 written by Stacy Blasiola (@Blasiola) or this modified version by Karen Smith & Shanifa Nasser made for Open Data Day Toronto available as CC-BY-SA.

What will happen to my existing TAGS sheets that aren’t version 5.0?

When Twitter turn off the old API (test outages this March) all authenticated and unauthenticated search requests will stop working.

How do I upgrade existing versions of TAGS spreadsheets (v3.x to v4.0) to keep collecting beyond March 2013?

As I can’t push an update to existing copies of TAGS you’ll have to manually update by opening your spreadsheet, then opening Tools > Script editor… and replacing the section of code that starts function getTweets() { and finishes 134 lines later (possiblly with the line function twDate(aDate){ ) with the code here. [And yes I know that’s a pain in the ass but best I could do] … or you can just start a new archive using TAGSv5.0
More additional tips and info when I get a chance

chevron_left
chevron_right

Join the conversation

comment 352 comments

  • dkernohan

    Thanks Martin – good job 🙂

  • Sam

    Excellent – Thanks Martin.

  • LAK13: Recipes in capturing and analyzing data – Twitter Jisc CETIS MASHe

    […] As well as not being free the raw data isn’t also always available. My solution has been to develop a Google Spreadsheet to archive searches from Twitter (TAGS). This is just one of many other solutions like pulling data directly using R and Tableau the main […]

  • Sharon

    Thanks for this Martin. But I’m getting the “Script function onOpen could not be found” error message when I try to use the spreadsheet. I’ve tried making a new copy but with no better results. Any ideas? Should I try again later?

  • Sharon

    Wait – it is working, but I was pressing the wrong button. I’ve got two buttons – one is labelled Enable Custom Menu – that gave the error. There is a blank button next to it that worked (?!).
    But now I’m getting an authentication error: OAuth Error (line 376, file “Google(c) Code”).

  • peelie

    hi
    was trying to use TAGS explorer but getting error message “Error in query: Access denied Access denied”
    thanks

  • Jonas

    Hi Martin,
    thank you for your tremendous work and the new version of your TAGs sheet!
    Similar to Sharon I also got the “OAuth Error (line 376, file “Google(c) Code”)” error. This error occurs when I try authenticateTwitter in the Script Editor.
    Thanks
    Jonas

    • Martin Hawksey

      OAuth Error (line 376, file “Google(c) Code”)

      Sounds like your key/secret are entered incorrectly or a mistake in the callback url entered in dev.twitter.com

      • Jonas

        Thanks for your reply! It was indeed the missing callback url. If anyone’s interested, the solution to this error is the callback url “https://spreadsheets.google.com/macros”.
        thanks again!

  • Ann Priestley

    Like Peelie I get errors with the 2 Web view links at the foot of the dashboard. I’ve also tried entering the URL/key at http://hawksey.info/tagsexplorer/- it just sits there looking at me!

    • Martin Hawksey

      For those views you need to make sure the spreadsheet is File > published to the web

      • Ann Priestley

        Doh…now I feel really stoopid! Thanks for the speedy reply.

  • ReadyWriting

    How many Tweets does this archive? I can’t get it to archive past about 80, instead of the 1500 like the last spreadsheet.
    Thanks!

    • Martin Hawksey

      I imagine you are hitting the Twitter Search API 7 days limit (anything older isn’t available). I’ve got v5.0 running on #edcmooc and it’s beginning to creak at 15k tweets

  • Stacy Blasiola

    Hi Martin,
    Thanks again for setting this up.
    I am running into issues exporting the spreadsheet. I’ve tried all of the options, but they all return the same error: ERROR: Export to .csv failed.
    Any ideas why this is happening?
    Thanks!
    Stacy

    • FredericGSole

      It happens when the number of tweets is greater than 10k, more or less. What you can do is to copy the file, and delete half the tweets in the original, and the other half in the copy, and export both…
      Frederic

  • Sharing Time! | Text 2 All

    […] then you should be following Martin Hawksey. He is responsible for coming up with the scripts to archive tweets in Google Spreadsheets and the cool visualization tool. There’s lots of other great stuff for analyzing tweets and […]

  • Ashley Patrick

    Hi,
    Thanks for this. I may be a little confused, but do is the newest version of TAGS (ie v5) sufficient to work or do we have to change the script in like it says?
    Thanks
    Ashley

    • Martin Hawksey

      Hi Ashley – I’ve clarified the text slightly. v5.0 requires no update to the script.

  • Shelley Rodrigo

    THANKS a bunch for this work! Do you have a location for contributing to the work fund?! I definitely am happy to donate a bit to the cause.

  • Cynthia

    Just want to thank you very much for putting this up and making the public search interface available

  • Chris Fiorello

    Hi Martin,
    I’ve gotten through everything smoothly, but when I try to perform a search, the “Running script collectTweets” box just remains on-screen indefinitely and nothing ever appears in the archive. Any advice on what I’ve done wrong?
    Thanks so much

  • David Schick

    So, I followed all the steps, set everything up correctly… but I have a question about how much data is pulled. I am noticing an inconsistency in the number of tweets by a particular person via a particular hashtag. Also, the google doc “dashboard” lists a little over 1500 tweets and shows that the first tweet was made 2 days ago (Feb. 28). And the analyzer only shows from March 1st…
    And I just noticed that the google archive is missing a good number of Feb. 28 tweets… (also, is it possible to change the time zone?)
    Anyway, sorry to complain, want to say that even though I’m having problems this is amazing.

    • Martin Hawksey

      The twitter API is limited by the last 1500 tweets and/or 7 days. So if you made a search on the 1st March it’ll pull the last 1500 tweets upto a 7day limit (which ever one it hits first). Do you think that explains your problem? Not sure about timezone, you might be able to go into Tools Script editor and change something in File > Project settings(?) but I’ve low confidence that will work (it may need to be tweaked in the code)
      Martin

      • David Schick

        Thanks for the response…
        Curious, is there a way I can embed the archive or explored on my webpage?

  • Julie

    Dear Martin,
    I was so excited to find this and I have followed the instructions carefully and I am desperate to make use of it and share it with others. Two surprise things:
    1) at min 3.25 in your video the authenticate twitter step – I did that, but I got no popup to authenticate on twitter – I did get the ” the yellow “running function pop up at min 3.28, but no “authorization required at min 3.28 or the subsequent twitter authentication window.
    As a “hack”, I did see on the developer page that I could create my own token right from the Twitter Developers site – “Use the access token string as your “oauth_token” and the access token secret as your “oauth_token_secret” to sign requests with your own Twitter account. Do not share your oauth_token_secret with anyone.” and I Verified the following was in my twitter profile: – Julie Archive by
    A way to archive my tweets
    Permissions: read-only
    Approved: Sunday, March 3, 2013 9:34:02 p.m.
    -If this is not correct this may explain why my spreadsheet isn’t working (see #2 below).
    2. At mins 4.10-4.15 I ran a search for the hashtag #ihateschool – i went to menu TAGS, RUN NOW! and it’s just hanging at “Running a script collectTweets Dismiss”
    And Nothing is going into the archive window.
    Any help you could offer…?
    Thanks,
    Julie

  • Martin

    Hi Martin, any idea why the Twitter application returns “Not a valid URL format” when I enter the spreadsheet url set out above?

    • Martin Hawksey

      Hi Martin – First thing I’d check is there is no space before or after the url. If it’s still not working let me know
      Martin

  • Mary Canady

    Great resource, thanks!! Do the advanced functions such as get followers work? I did try and it looked like it was working but a new tab was not created with the data.

    • Martin Hawksey

      Hi Mary – the sheets data are written to are hidden be default. Click on the icon bottom left with 4 thick black lines to unhide them
      Martin

  • Kelli Burns

    I have used your TAGS spreadsheets before. I followed all the directions on your video and have the new Twitter key and secret. When I try to authenticate twitter, it flashes quickly with “running function authenticate twitter” but then nothing happens after that.

  • Kelli Burns

    Got it! Forgot to put in the callback URL.

  • Scott

    First of all, thank you for this tool. It is probably the most useful twitter tool I have ever come across.
    Second, is it possible to use the advanced search operators ‘near:’ and ‘within:’ ?
    I have tried this in several different combinations with single, double and no quotes and cannot seem to get it to work.
    Thank you.

    • Martin Hawksey

      Hi Scott – Unfortunately the Search API doesn’t support near: and within: It is possible to add a geocode parameter manually in the source code. If you open Tools > Script editor around line 120ish you’ll see a number of params are set eg params.count = advParams.count || 100; you can add another line with something like params.geocode = “37.781157,-122.398720,1mi”; setting your own latitude, longitude and radius (see documentation for details)

      • Scott

        That worked perfectly, thank you!
        A great tool just got even better.

      • Adam

        Can you provide any more detail on this geocode solution. I edited the code as suggested and continually come back with no results. Any help is appreciated.

      • Matthew Shepherd

        The geocode filter was exactly what I was looking for. Thanks for such a great tool and for the support 🙂

  • Nick Bowman

    Martin, this is a *great* application – as a social media user and researcher it’s great to have a customized search that I can access through Google Drive. However, I just wanted to confirm that the -7 day limit was a Twitter API issue, yes? We’re trying to pull hashtags from catastrophic events (such as the High Park Fire in Colorado, #highparkfire) but it seems that we can’t get those, as they would have been from June 2012.
    We’re able to use the “regular” Twitter search to get access to them (https://twitter.com/search/realtime?q=%23highparkfire&src=typd) but it seems that there is no way to pull these into an Excel sheet or file such as Tag v5.0?

    • Martin Hawksey

      You are correct the api is limited to last 7 days. Twitter recently started rolling out the inclusion of older tweets in it’s search results but this data is currently not accessible to anyone else. Datasift have a a historics sou for older data. The data isn’t free but very cheap

      • Nick Bowman

        Thanks so much for the reply, and for the great program. I’m having some trouble getting the visualizations to show up in the Google Doc (lots of “#name” errors) but the TAGSVisualizer is pretty slick.
        I’ll follow this project often and recommend it to others – and you’ve provided our lab with an invaluable Tweet archiving tool that we will be sure to credit you with in any publications. =)

  • Alan O'Rourke

    Thank you Martin for doing this. Brilliant resource.
    I have tried setting it up twice and when i run it i get the following message “Exceeded maximum execution time” after a while, using different search terms.
    Any ideas?
    Alan

  • Su Butcher (@SuButcher)

    Hi Martin,
    I’ve set up one of these archives and it went fine, but today I’ve had another go with V5.0 and I’m getting an error message.
    After Twitter API Authentication is completed (I made a new app, but I’ve also tried it with the old app), I authorise the API, put in the search term (a hashtag) and click Run Now.
    I then get asked for ‘Twitter API Authentication required’, and a red erro message, “TypeError: Cannot read property “length” from undefined.”
    Any ideas what I’ve done wrong?
    Many thanks, Su

    • Martin Hawksey

      Hi Su – when you run the script authenticateTwitter does it redirect you to Twitter where you approve the connection?

      • Su Butcher (@SuButcher)

        To be more accurate, it says ‘Running Function authenticateTwitter’ and then that disappears and nothing.

        • Martin Hawksey

          Are your Twitter API keys/secret set?

          • Su Butcher (@Subutcher)

            Yes, made a new application as per instructions.

          • Martin Hawksey

            The Twitter API settings have the correct callback url (in the callback field)?

        • Kelly

          Hi, I am experiencing the same issue. Have been using the tool for about a month now and suddenly it has stopped working and is giving me the error “Twitter API authentication required” — my access token and secret are set, and I have triple-checked the callback URL.
          Then today I have also started receiving the error “TypeError: Cannot read property “length” from undefined.”
          Any thoughts?
          Thanks so much!
          Kelly

          • Su Butcher (@SuButcher)

            Hi Martin,
            Yes all the settings were fine, i went through them again.
            Then this morning I clicked Twitter API Authentication and ‘OK’ again, and chose ‘Run Now!’ and the script just worked. No idea what I did, it just worked fine this time.
            Thanks for your patience!

  • Su Butcher (@SuButcher)

    Yes it did, though I’ve just had another go via the script editor and it just sits there when I ask it to authenticate twitter.

  • Julie

    This is happening to me too. I just set it up to try to run it again and I got the same thing: TypeError: Cannot read property “length” from undefined

  • Julie

    Hey – All, I got this to work finally! I was having trouble with the “run thing” issue as well – then i just went to the link provided by Martin here – https://docs.google.com/document/d/1N838xH6NeRa6_kZQVkB9tBCliJo_8drvPuM8Ds6tsHg/edit
    And I followed those instructions exactly. I didn’t get the results I was expecting so then I went back to Tools, Script Editor and I ran the collectTweets function and now it is working.

  • Twitter Archiving Google Spreadsheet TAGS v5 | eLearning tools | Scoop.it

    […] For a couple of years now to support my research in Twitter community analysis/visualisation I’ve been developing my Twitter Archiving Google Spreadsheet (TAGS). To allow other to explore the possibilities of data generated by Twitter I’ve released copies of this template to the community.  […]

  • Kevin

    Hi
    A previous version worked for edges and a simple network analysis.
    I tried to fill “edges sheet” but it didn’t work out. I know NodeXL does social network analysis, but it does not update archive.
    Thanks much for your excellent work!

    • Bruno

      I’m also having some trouble with the “getCommunityGraph” function for filling the edges – it does fill the edges but i can’t make out what exactly is being displayed, whether they are “following” vertices or retweet vertices.
      Sometimes i come up with a “#Value” in the cell, but can’t figure what the error is.
      Didn’t understand how the “getCommunityGraphfromsaveddata” works, as well.
      What I’m actually trying to figure out is the best way to import retweet relations into Gephi, any advice would be much appreciated!
      Besides that, i too must pay homage to the incredible work you’ve done! You rule man! Thank you veeery much!

  • Chux

    A big thank you for this great tool. Please, can the archived data be exported into Gephi (or any other network analysis tool) for analysis? Once again, THANKS!

  • Hossein

    Hi Martin
    I have a problem with authentication, and I can see that is not only my problem. I did everything exactly as described, but when I go to Tools > Script Editor, and run authenticate Twitter, nothing pops up to get me to the Twitter authentication page. Hence, the app does not work. Any idea what is wrong?
    Thanks
    Hossein

  • mickski_007

    Hi Martin,
    firstly thank you for sharing your work, I have been using using the various incarnations of TAGS spreadsheet (v.3 – v5.05) extensively in my day-to-day analysis. Lately, I have been looking at the code closely and have a question. I am currently searching 24 @keywords and limiting the number of tweets too 100. As soon as I add the 25th @keyword I get the “Execution Timeout Exceeded”. Changing number of Tweets doesn’t make a difference. After a bit of research I came across the following link;
    http://stackoverflow.com/questions/7854573/exceeded-maximum-execution-time-in-google-apps-script
    And wanted your advice as to whether in principle/theory it would be possible to modify the collectTweets function to run in a similar way to the runMe() function in the link above. A timed batch process of var SEARCH_TERM and do it in chunks.
    Hope this makes sense, and thanks again for your contribution.

    • Martin Hawksey

      Hi, quick Q are you using a single spreadsheet for you 20+ keywords? If so the issue is probably the url sent to Twitter (which will include your search term) is too long.
      Thanks,
      Martin

      • mickski_007

        Morning,
        Yes, you are correct, the keywords are in a single spreadsheet (below keywords take 4secs to collect&getTweets). Here is my working keywords set, = 439 Characters;
        @Universal_WE OR @mci_group OR @PMMOurWorld OR @Zibrant OR @GPJ_EMEA OR @GeorgePJohnson OR @jackmorton OR @TROexperiential OR @CarlsonWagonlit OR @Banks_Sadler OR @aileen_reuter OR @BCDTravel OR @rpmltd OR @VenuesEventMgt OR @wandoevents OR @logistikgroup OR @cievents OR @FirstProtocol OR @WRGCreative OR @ubm OR @Clarion_Gaming OR @ImaginationLabs OR @aymtm OR @BSIEvents OR @cascade_alan OR @crownlondon OR @drpgroup OR @FirstEventLeeds
        Is there a better way of analysing multiple keywords? Should I look to break this down into multiple TAGS files and try to analyse across multiple files?
        Thanks,
        Michael.

        • Martin Hawksey

          I’m not aware of an easy way to get around the query length other than breaking it up over spreadsheets. It would be possible to add some extra code to do this (unfortunately I don’t have time to do it)

          • mickski_007

            I’ll keep it to Top 20 for now, thanks for getting back to me.
            I monitor some fairly high traffic hashtags that required me to add a couple of functions that deleted tweets based on numRows & numDays. Hope its ok to include them here in case anyone wants to create a new script file;

            function deleteOldDays() {
  // THIS FUNCION DELETES ROWS OLDER THAN 7 DAYS
  // I SET AS A TIMED TRIGGER > DAILY
  var ss = SpreadsheetApp.openById("SPREADSHEET_ID"); // Number between key= & # in current spreadsheet
  var sheet = ss.getSheetByName("Archive");
  var datarange = sheet.getDataRange();
  var lastrow = datarange.getLastRow();
  var currentDate = new Date();
  var oneweekago = new Date();
  // CHANGE THIS NUMBER TO INCREASE OR DECREASE DAYS
  oneweekago.setDate(currentDate.getDate() - 7);
 for (i=lastrow;i>=2;i--) {
    var tempdate = sheet.getRange(i, 1).getValue();
    if(tempdate < oneweekago)
    {
      sheet.deleteRow(i);
    }
  }
}
function deleteOldRows() {
  var ss = SpreadsheetApp.openById("YOUR-SPREADSHEET-ID"); //Remember to replace this with your ID
  var sheet = ss.getSheetByName("Archive");
  var rowsToKeep = 12000; //CHANGE TO YOUR DESIRED NUMBER OF ROWS TO KEEP.
  var rows = sheet.getLastRow();
  var numToDelete = rows - rowsToKeep  -1;
  sheet.deleteRows(2, numToDelete);
}

  • Meghann

    Hello Martin. Thanks for what you have create.
    I would create two files for different keywords. Do I just need to make two copies in putting different keywords inside it and it works or there is another manipulation? I’m afraid that the two files with the same settings and authentication are not distinct. I need to do two separate searches with different keywords. It is for this that I want two separate files.
    (Sorry for my english, I’m french)
    Thank you !

    • Martin Hawksey

      Hi, you can use the same file and in the search term use OR. For example, cats OR dogs

  • Bill

    Hi Martin,
    I’m getting the following error on V5:
    Line 157 Cannot read property “followers_count” from undefined.TypeError
    I am able to test the collection successfully.
    Any ideas why that might be happening?
    Thanks,
    Bill

    • Martin Hawksey

      Is this using a copy of the template or trying to update an existing spreadsheet?

      • Bill

        A copy of the template. I created it by clicking on the *** Twitter Archive Google Spreadsheet – TAGS v5.0 *** link.
        Thanks,
        Bill

  • Victoria

    Hi Martin,
    I’m getting the following error on V5 after running the authenticateTwitter function on the script editor:
    Request failed for returned code 404. Server response: {“errors”:[{“message”:”Sorry, that page does not exist”,”code”:34}]} (línea 376, archivo “Google(c) Code”)
    I have already authorised permission with my twitter account.
    Do you know why that might be happening?
    By the way, I have another problem with the conversation visualization in TAGSExplorer. I have already publish the Google Spreadsheet in the Web but when I try my URL in the TAGSExplorer, the web is loading for ever with the message “Loading data”. I think that there aren’t too many tweets to be loading so slowly… (only 246), so I suppose it may be another thing… Any idea to solve this?
    Thank you very much and good job!
    Victoria

  • Pasquale Franzese

    Hi, thank you for TAGS, you may save me weeks of work!
    I’ve followed the instructions (got special menus, authorized the app), but when I click on “run now!” it just says “Running script collectTweets Dismiss” and stays like this, i can still do things in the sheet, but it doesn’t give me any tweet in the archieve.
    What can I do to make it work?

    • Martin Hawksey

      Hi, As far as I can see this is to do with Google infrastructure and have run out of optimisation solutions. For some reason on the initial collection I find better performance if I open tools > script editor and then Run > collectTweets. Once an archive is running they generally update without any fuss. Seems to be an issue around when the spreadsheet is first copied.

      • Pasquale Franzese

        thank you, I just opened the script editor and ran the collectTweets from there, a popup asked the app authorization and after it started working normally

  • Alan O'Rourke

    I get the same as Pasquale.

  • UgalyiDoc

    Hi there, can I check if this only pulls in organic tweets and not promoted tweets?

  • Nedyalko Terziev

    Great job, @Martin and thank you for sharing it with the world!
    Couldn’t find an answer to this, so asking here- does TAGS support a combination of logical operators for the search term. Specifically, I am looking for something like: a AND (x OR y OR z).
    Cheers,
    Ned

    • Martin Hawksey

      Hi Nedyalko – yes you can use combinations of operators. One thing to watch is your query string doesn’t get too long (the maximum is 1,000 characters – this includes some parameters sent by the script so it’s a little less than this)
      Martin

  • Humanidades digitales: la conversación en Twitter sobre el Congreso | Por vista de ojos

    […] Por medio del TAGS Explorer de Martin Hawksey, podemos representar de manera gráfica todos los tuits que hacen mención al hashtag #HD_griso (o a cualquier otro término de búsqueda). Para ello era necesario desarrollar una pequeña aplicación con Twitter Archiving Google Spreadsheet TAGS v5 siguiendo las instrucciones que aparecen en http://mashe.hawksey.info/2013/02/twitter-archive-tagsv5/. […]

  • n3k1 ma1l

    I use your script, but I wanted to create new one and it seems that something is wrong. I have only “running script” but no data is pulled from TW. Do you have any idea?

    • Martin Hawksey

      not authenticating is where most trip up

      • PF Anderson

        I’m having problems with this, too. I’ve authenticated several times, but cannot get the custom menu to install. It doesn’t matter how many times I click on the button, give authorization, and click “accept”, it does exactly the same thing, and I just end up in a loop. Baffled.

        • Martin Hawksey

          hmm – have you tried taking a fresh copy of the template?

          • Julian

            I had the same problem with the message “running script” and no data pulled from the spreadsheet. In order to solve it I tried to reload the spreadsheet, then went to Script Editor and authenticated again. Then closed the editor and from the spreadsheet I re-ran the script to get the data, and this time it worked like a charm!

  • Technicity: Web Scraping

    […] Working on my technicity idea that focuses on fuel poverty. I have done a little twitter analysis having followed Martin Hawksey great tutorial on how to use his analysis tools. […]

  • GIZIPP

    Thanks for this awesome tools! 😀
    Tweet mining is so cool too…

  • Brendan H.

    Hey Martin,
    I just have a question about rate limitations. I know that it is possible to search up to 15,000 tweets or tweets from the past 7 days (which ever limitation is reached first), but would you be able to tell me if any further rate limitations exist specific to this software? For example, I know that I was having a problem at one point where I made too many searches in one 24-hour period, so my searches were unable to run any further. Is there an exact rate limitation in this regard? Maybe the 15K/7-day rate limitation applies in this scenario? Thank you!
    All the Best,
    Brendan

    • Martin Hawksey

      Hi Brendan – the api docs state 180 calls per hour. The script fetches 100 tweets per call. TAGS Advanced menu has a ‘Test Rate’ menu option for you to monitor remain calls

      • Brendan H.

        Ah, perfect! Exactly what I was looking for; thank you!

  • Tanis Thiessen

    Hi Martin – I’ve been experimenting with the TAGS v5.0 script to explore how to archive our #cdnedchat sessions. I have had trouble getting the archive to show times during the turnover from end-of-day to next (23:59:00 June 3rd to 00:59:00 June 4th) in the search.
    I’ve tried changing the settings up (2 days, 1 day, 3 days) but that particular hour is just not showing up in the search. As it happens, that’s the exact hour when our chat happens online.
    Any way for me to fix this? What do I need to adjust in the readme/settings?
    Thanks!

    • Tanis Thiessen

      Okay, I went back to Tools, Script Editor and I ran the collectTweets function and now it is working – all tweets are showing up. Just have to figure out how to get the analytics to work with the “paged” archives. Slowly but surely! Thanks for such an amazing resource!

      • Martin Hawksey

        Hi Tanis – glad you got it working. The paged collection is a feature from the very early versions of this template and I haven’t really been looking after it. My recommendation would be to use the single archive version. If you want to give users easy navigation per chat the MLAA developer TAG Viewer is a great option. Here’s an example on how the TAGS sheet for #cdnedchat is rendered. Alternatively there is this searchable interface I’ve created
        Martin

        • Tanis Thiessen

          Brilliant! Thanks for the suggestions regarding archiving multiple chats in one archive – love these resources you have created!

  • Micha K.

    Thanks for this great tool. Really a brilliant thing. I also got a question. I just ran a sweep but was wondering why it doesn’t seem to go very far back in time. Is there a way to make it sweep tweets from the past few days? Right now it only goes back for about a couple of hours even though I know that there is much more. Any ideas?
    Thanks again.
    Micha

    • Martin Hawksey

      Version 5 was developed specifically to address the api changes

  • Μ.Μ.

    Since yesterday the script is not working. There is a problem with the variable followers, that I am suppose that is due to the retirement of the API v1. I have been using your script for several months now, and I wanna thank you for that.

  • M.M.

    Martin this is the error that I get:
    Apps Script
    Line 157 Cannot read property “followers_count” from undefined.TypeError
    I tried to set up a new gsheet but I get the same error. It was reported that twitter has made some changes on how to get the followers, but I didn’t have so far the opportunity to search for more details.
    Thank you again, for your work.

  • Tim Stevens

    Thanks for Tagsv5.0! It is of immense importance to me. I am PhD at the Wageningen University & Research centre (the Netherlands) and I am doing a research on the influence of twitter-acitivty on the governance of sustainability in agro-food systems. I’ve made five documents for archiving tags, but now I encounter two problems: 1) only my first two are working (1 shared, 1 private), the others do not archive data, and 2) Tagsexplorer isn’t working for any: it says ‘loading data’ (for 8 hours).. I hope you can give me any direction for solving these problems. Keep up the great work!

  • Andres

    I am facing same error as M.M. Script is not working. Plus, I get a redbox saying something about the length property.
    Thanks for your work

  • Tim Stevens

    Thanks for this great tool. I am a PhD researcher and this is of great value to me!
    If you have problems with TAGSExplorer: it might be because you’re using Internet Explorer, try Google Chrome instead. It worked for me 🙂
    I can’t make new archives. I’ve two running properly, collecting about 40 tweets a day for a few weeks now. When I make a new one and click ‘run now’, it says
    Apps script:
    Twitter API Configuration Required
    And then, when I close that window, it says in red:
    TypeError: Cannot read property “length” from undefined
    I’m sure I’ve followed all steps, including the Twitter API authentication procedure (you can use your key and secret multiple times right?) and authenticate Twitter via Tools, script editor (you need to do that for all docs?). I haven’t archived many data yet, so I guess it’s not the amount of data I am pulling in. Is there a limitation for the amount of archives to have running, or the times you do the Twitter API authorization configuration, or just for the amount of data pulling in?
    I hope someone can help me with this somehow. Many many thanks

    • Tim Stevens

      Problem Solved: When I did the Twitter API authentication via Tools > script editor > run > authenticate twitter it worked. I guess I did something different before..

      • itsdanprice

        Thanks! You saved me a good amount of head scratching!
        This tool is wonderful! I’ve popped it on my blog.

  • Tim Stevens

    Any suggestions for getting historical data in the same format? I need very little and basic data starting from November 2011 with the same search term as I used with TAGS. Anybody already done that? I would love to hear any suggestion. Thanks a lot.

  • Andy Mitchell

    Hi, I’ve just made a copy of version 5.0, but when I click on the ‘enable custom menu’ button a window opens and says, “Oops, Script function onOpen could not be found”
    Is this a problem with the software or is it my google drive causing it to do that?

    • Martin Hawksey

      Hi Andy,
      Google sometimes needs a couple of seconds to catch up with the template copy to attach the script. The advice I’ve seen elsewhere is to keep hitting the damn button until t works 😉

  • Twitter API Version 1.1 Switchover » Nicola Osborne

    […] The other change to accessibility and flexibility of data is less obvious but also frustrating for those who like to do a quick and low tech mash up or prototype. Whilst Twitter have been using OAuth for some time this update ensures that effectively no Twitter tool, app or widget can use anything other than OAuth – rendering the RESTful API a rather different beast to use. With that last change in mind users of Martin Hawksey’s excellent Twitter Archiving Google Spreadsheet may want to make sure they have upgraded to Version 5. […]

  • Misha

    Thank you very much for this! I hope this is not a silly question but only the first search term I enter shows up in the archive. Is it possible to enter more than one search term or do I have to do separate sheets for them? Also, my understanding is that Twitter allows for searching further back than 7 days. Is this a possibility here? Thank you very much again!

  • Su Butcher (@SuButcher)

    Hi Martin,
    Is there an option for language filter in the archive? I’ve notice the French are using the same hashtag as us. If I could filter out the french language tweets…

    • Martin Hawksey

      Hi Su – requires some code tweaking. In the Tools > Script Editor change the lines around number 87 from
      if (sinceid){
      var data = getTweets(SEARCH_TERM, {"sinceid": sinceid}); // get results from twitter sinceid
      } else {
      var data = getTweets(SEARCH_TERM); // get results from twitter
      }
      to
      if (sinceid){
      var data = getTweets(SEARCH_TERM, {"sinceid": sinceid, "lang": "en"}); // get results from twitter sinceid
      } else {
      var data = getTweets(SEARCH_TERM, {"lang": "en"}); // get results from twitter
      }

      • Su Butcher (@SuButcher)

        Hi Martin, that’s great – thanks for letting me have the code. I’ve made the changes and can’t see any more French tweets. I’ve deleted the ones from the weekend.
        I’m liveblogging the Government’s Construction Summit tomorrow – will be also promoting TagsExplorer and TagsArchive to help non-tweeters see what’s going on.

  • Pepijn

    Hi, very awesome script and interface! I’ve ran into a problem. I set up a couple of scrapes for my thesis research. All working fine, but one isn’t working and states dates from 30/12/1899 till 30/12/1899. What could this be?

  • Bill

    Hello Martin, when i run the time-driven triggers to collect tweets .. is there a way to filter the new ones from the old ones ? Basically .. i want to be able to capture all tweets on a file between 1pm – 3pm and another separate file from 3pm – 5pm and a separate file from 5pm – 10pm & so on and so forth . .. is this possible to do ? Or does TAGS just update the tweets to the spreadsheet ?
    ALSO, is there a limit to how many tweets it looks for ?
    Lastly, on the spreadsheet .. the times it says the tweets have been posted are incorrect as to when i open my tweet deck. How do i change the time zone so i see the correct time same as i see on tweet deck ??
    The time is off by like 7 hours or something like that ..
    Thanks,
    Bill

      • Jeffrey Keefer

        This is very helpful!
        One question; how can I change the time zone so the times are accurate for EDT (the time for New York City)?
        Perhaps an ideal would be if there were a way to have the date in one column (in US format, which would be 7/12/13 for July 12, 2013) and the time (in EDT) in another column? Would this be something easy to adjust?
        Thanks for this wonderful script!

        • Martin Hawksey

          As a couple of people have asked. One way to get different timezone date display is in the Spreadsheet go to File > Spreadsheet settings and select date locale and timezone and then in Tools > Script editor open File > Project properties and change the timezone there as well. This should convert any existing and all new date/time data in the time column. The created_at column is unchanged as this is the metadata passed back from Twitter and required for tools that visualise a TAGS archive

  • Cristina

    Hi Martin,
    Thanks for the amazing tool. I have one question though. How can I collect tweets based on geolocation? I want to collect some data from Bucharest and Brasov, Romania, but not by using hashtags. Is it something that I need to modify?
    Thanks,
    Cristina

    • Martin Hawksey

      check the comments thread this has been asked and answered

  • Day 10 of #STEM10DoT: Past and Future | STEMDigital

    […] who knows a lot more about it than I do! If you want to know more about how these were created, there are instructions online and further instructions on making a visualisation such as the ones linked to […]

  • Clive

    Brilliant tool – thank you, Martin. I had been using “The Archivist” to archive #globalclassroom twitter chats but, now that it’s subscription only, your script is the perfect replacement!

  • Matthew Shepherd

    Fantastic tool, thanks Martin! I’m trying to figure out a way to strip line breaks out of the tweets when they are imported to the Google Docs spreadsheet.
    I’m exporting this data via a web query to Excel and line breaks in Tweets cause the text to broken in to separate cells.

    • Martin Hawksey

      Do you have an example web query I can try?

      • Matthew Shepherd

        Thanks for your guidance Martin. I have managed to strip line breaks from the Tweet text by adding some code to the TAGS.gs script. Around line 187 (after //end of while loop), at the end of the gettweets function I added the following code:
        // remove line breaks
        for (var i = 0; i < data.length; i++) {
        data[i]["text"] = data[i]["text"].replace(/\n/g, " ");
        }
        Perhaps this might be a useful option to include in future releases?

        • Martin Hawksey

          Thanks for this. Given the workflow Richard has highlighted this makes sense as an option.

  • Micha K.

    Hi Martin,
    This question has been asked before by Bill but I dont think found the answer to it. Is there a way just to scrape the tweets in a particular time frame. That is for instance from last Sunday 10pm to 11pm EST US time or Monday 2am to 3am UTC? When I started my search last night at 11.20pm I only was able to get the last 5 minutes that is from 10.55pm and a lot of “clutter” that followed after. Now I wonder if there is still a way for me to get the tweets from yesterday 10pm to 11pm? Your help is greatly appreciated.
    Thanks.
    Micha

    • Martin Hawksey

      Hi Micha – unfortunately not down to the hour the best the Twitter API can give is a day range (from last 7 days). On the readme/settings sheet there is an option to get yesterday, -2 days, -3 days etc. To use this you need to make sure there aren’t newer tweets in the archive than the date range you are trying to get. e.g. if you change the setting to yesterday and you’ve already got tweets from today in the archive when you hit TAGS > Run Now! it won’t insert the older tweets.

  • Rebecca Shore

    I have followed the directions to the T and still can’t seem to bring any tweets back. On the Dashboard it says “Cannot read property ‘title’ of undefined”. I’m not sure what that means. I would really love to use.
    Let me know if I’m missing something.
    Thanks!

  • Rameez Ramzan

    I have read your Twitter Archive post on moz.com, I admire your post and happy to read and implement in my twitter strategy. I hope you do more like this.I know that you put the effort on this. We are happy to use this sheet 🙂

  • Jason Stum

    Thanks for making this tool publicly available, it’s much appreciated.
    I am however having trouble getting everything to work correctly. I’ve followed the setup instructions yet each query I try to run gives me a “exceeded maximum execution time” error.
    I’ve authenticated twitter from the Script Editor and plugged in my Twitter Oath Key & Secret.
    Am I missing something? Any help you can provide would be wonderful. Thanks!

    • Jason Stum

      Scratch that, seems to be working now. Thanks again!

  • Twitter Archiving Google Spreadsheet TAGS v5 | ...

    […] Une solution permettant de créer votre propre Google app pour archiver vos tweets et vos hashtags en utilisant le Cloud façon Google (Google Docs + Google Drive)  […]

  • Derek Bruff

    Thanks for this great tool, Martin. I haven’t figured out how to get anything in the “geo_coordinates” column. Is there a trick here? I’ve turned on locations for my tweets, but that doesn’t seem to populate that column. Any advice?

  • betty

    Thanks for this awesome tool, Martin! Do you know a possibility to get older tweets, like from march? Operators ‘since’and ‘until’ don’t work, and I know there’s the time period of 7 days but is it possible to search further back? Thank you so much!

    • Martin Hawksey

      limitiation of Twitter API is last 7 days

  • Building the RMOOC Site, Syndication and All - CogDogBlog

    […] every project that uses a twitter hash tag I use Twitter Archiving Google Spreadsheet TAGS v5 developed by Martin Hawksey. This not only creates an archive of all tweets, we get a summary of […]

  • Post-#SAA13 roundup (We want your input!) | The Schedule

    […] future Twitter accessibility, was able to collect most of the #saa13 tweets with the help of the TAGS Google Spreadsheet, meaning that you can go back and revisit the tweets weeks after they scroll off Twitter search. Go […]

  • Julian

    Here is another comment in case you want to use TagsExplorer with your own spreadsheet. I tried to instructions but I was unable to see the visualization (spreadsheet was not loaded) until I published it to the web (even if it was shared). I think this step should be added to the list of steps. Thanks, this is an awesome tool.

  • Herberto

    Hi, Martin
    Thanks for this great tool. I use SNA in my PhD project and, when it comes to extract data from Twitter, NodeXL seemed to do the trick… until the moment I found out about TAGS magic. So, I took it for a test drive, but the SNA script is not working for me. Every time I try to run some functions of TAGSAdv.ps (from the script editor) I get an error:
    1- GetCommunityGraph: “TypeError: Cannot read property “length” from undefined. (line 43, file “TAGSAdv”)”
    2- replaceIDsWithNames: “The coordinates or dimensions of the range are invalid. (line 98, file “TAGSAdv”)”
    3- GetCommunityGraphFromSavedData: The script runs but nothing happens!
    4- displayIDSinCell: “Invalid JSON string. (line 150, file “TAGSAdv”)”
    Everything else seems to be working fine (and I could export the relevant data to UCINET, for example), but without the SNA dimension TAGS loses its allure to me…
    Keep up the good work and thanks for your help!

  • Nicola Osborne

    Martin,
    We’re having an odd problem on one of our TAGS sheets. We haven’t received any updated since Tuesday night, including a few direct replies which are usually very reliably caught.
    I have tried running again manually, reauthenticating, etc. but no error is showing and no additional tweets are being collected.
    Have there been any issues with the API/changes made since the conversation tweak Twitter made this week? Any/all advice very much appreciated 😉
    Thanks, Nicola.

    • Martin Hawksey

      Hi Nicola – when you look in dev.twitter.com is your app still okay? Things I would try: Delete last tweet row in archive and try again; another thing you could try is open Tools > Script editor, Run > collectTweets and then View > Logs

      • Nicola Osborne

        Thanks Martin.
        So, no indication of issues in dev.twitter.com. App still seems fine.
        Deleting the last line and rerunning resulted in that line being reinstated. But none of the more recent tweets being added. The missing tweets are both from our account and replying to it and would normally be caught by the TAGS sheet (see anything post 27th Aug here: https://twitter.com/search?q=digimap4schools&src=typd).
        Running collectTweets from the Script editor appears to be fine and the Log is blank.
        Sorry, this seems to be one stubborn issue :-O
        – Nicola.

          • Nicola Osborne

            Ah, thanks for this. I shall have a full read. So I take it that TAGS uses the Search rather than the Streaming API? That’s useful to know anyway 😉
            In that case I think we’ll probably handle the omitted tweets as we have a couple of times before – by manually adding in the key date/tweet ID/content/user info etc.
            Huge thanks for all your help! 🙂
            – Nicola.

  • Herberto

    Hi again, Martin
    Sorry for insisting, but the problems persist (errors while running some functions) and I cannot render any SNA related data. I have a complete verteces sheet, but blank edges and no metrics. What am I doing wrong here?
    Thanks!

  • Reporting on the First Week of Headless ds106 | ds106 tricks

    […] I was just thinking that it would be useful to track some weekly stats, because there is nothing I enjoy better than the smell of analytics in the morning. No it does seem useful to chart some data, so I hope to update a spreadsheet to show by week the number of blogs listed, the total number of posts, and a weekly summary of twitter activity (the latter coming from our use of Martin Hawksey’s tracking spreadsheets). […]

  • The Wayback Machine Scraper - Josh Kellett

    […] There’s a ton of agile tools already out there — from robust Twitter analytics docs (TAGS is my favourite), SERP analysis tools, content idea generators, and others — and most of them […]

  • Ed Reggi

    I absolutely love TAGS and have been using it over the past 18 months to scrape many of the hashtags for a local public television station. Our budget is limited and your software has become an amazing asset to keeping up with the times.
    We are running into a recent problem every since the latest update of TAGS back in the spring. We are now experiencing duplicates in our archive. The archive can have tweets from say 9-17-2013 to 9-10-2013 and then it will start back at 9-17-2013 again. What would be the cause of this hiccup within the archive?
    Thanks,
    Reggi

    • Martin Hawksey

      Hi – I’ve also noticed recently problems with duplicates with the new API. Technically this shouldn’t happen, but I wonder if as part of the new API they’ve gone for speed over accuracy. In newer versions of TAGS there is a delete duplicate option from the TAGS menu to help clean up the archive

      • Jonas

        Hi Martin,
        just wanted to chime in, because this option doesn’t work in my archive.
        The error message I get (just deleted the whole archive and created a new one with the same result) is: TypeError: Cannot find function trim in object 380014964331007900
        Do you have an idea how to fix that?
        Thanks!

  • #edchatde – Runde 2: Archiv | MatthiasHeil.de

    […] von #edchatde war “Kollaborative Arbeiten initiieren, gestalten und bewerten”. – TAGS 5.0 von Martin Hawksey (herzlichen Dank für den Hinweis an den wunderbaren Jörg Lohrer!) hat knapp […]

  • Joe

    This is brilliant! Thank you! Any chance you will be doing one for Face Book?
    Thanks again!

  • #edchatde – Runde 2: Archiv | openreli

    […] von #edchatde war “Kollaborative Arbeiten initiieren, gestalten und bewerten”. – TAGS 5.0 von Martin Hawksey (herzlichen Dank für den Hinweis an den wunderbaren Jörg Lohrer!) hat […]

  • Pepijn

    Hi Martin,
    I’ve ran into a strange thing in the tool. I want to collect all tweets to and from ‘@Connexxion’. To:Connexxion is working perfectly, but from:Connexxion gives me “Twitter said: null”. Actually, every day there are several tweets from this account so this shouldn’t be a problem.
    Thanks in advance!

    • John Mattos

      I have the same problem. I have created 5 spreadsheets now, all with this same result, creating them as I always have.

    • Leonardo Flores

      I’m having the same problem. I want to collects tweets from @IAM_SHAKESPEARE and keep getting “null” to any from:IAM_SHAKESPEARE requests
      How can I fix this?

  • John Mattos

    I seem to get this with every spreadsheet copy I make now…
    Cannot read property ‘title’ of undefined
    – Twitter app is set up
    – App consumer and secret are set up properly
    – Ran twitter authenticate (And I dot the popup and agreed)

    • Martin Hawksey

      What’s your search query?

      • John Mattos

        all it is is “#ORUHangout”

  • John Mattos

    Also, thank you so much for your effort in this. I REALLY appreciate you taking the time to create and maintain this as well as engage with the users. You rock.

  • Poleak

    Hi Martin,
    Thank you so much for your great app. It helps me a lot in my research. 🙂
    However, could you please tell me how can I change the time (actually i am in France)?
    Thank you again,

  • Asher Rospigliosi

    Hi Martin, this looks a great tool, but I have filed to get it working. I get an error in the summary tab, in the right box, highlighted in red “sc(..)is undefined”
    I have added the authentication codes form Twitter and run the authenticate script, but was not sure if I needed to use the access tokens then offered by twitter. Do I need to use them in some way?
    Thanks
    asher

  • Sam Martin

    Hi,
    I have been trying unsuccessfully to get TAGS 5.0 to work on and off for about 48 hours now, even going so far as to make new copies, refresh existing sheets, repeat authentication, different twitter authentication keys/accounts and also use basic search terms (e.g. ‘food’) and small quota requests.
    But each time, even if I just try to do a ‘Test Collection’, the TAGS sheet hangs at the notification box ‘Running script. Dismiss’.
    My TAGS 5.0: https://docs.google.com/spreadsheet/ccc?key=0Al135Uzr-xDZdHdLbHRsWXVDTFhEZjN5ejZCUDd2UUE&usp=sharing
    Can anyone let me know if their TAGS 5.0 has worked in the last 7 days? Just so I know where I’m going wrong.
    Thanks,
    Sam

  • Asher Rospigliosi

    Hi Sam, sounds like the same problem I had. I too tried multiple installs/copies with fresh authenticaitons
    Martin, sorry if I /we are making an obvious mistake.
    asher

  • NoGoYo

    Hi Martin,
    Great Sheet! Excellent work!
    I’m trying to look for search terms that have a single quote-mark in them. e.g. here’s the set of terms I want to search and archive
    (#DevilsDue OR “Devil’s Due”)
    The scripts never complete running with this query.
    When I search for #DevilsDue I get resuts.
    When I search for Devil’s Due I get no results, so I assume the ‘ is the culprit.
    Any ideas what may be gong on here?
    Thanks again – great work!
    Nogoyo

    • Martin Hawksey

      There is an open ticket in Google Apps Script for UrlFetchApp not preserve RFC 3986 escaping which is used in the Twitter API 1.1. I’ve added a note to the search term box in the template to note this

      • Nogoyo

        Thanks Martin. Great work.

  • Asher Rospigliosi

    Leaving the script untended overnight I get “Exceeded maximum execution time” error, but no tweets.
    asher

    • Bill

      Hi,
      I created a video tutorial for my students for setting it up, including some steps that are missing from other instructions, but I’ve found are necessary for avoiding the kind of errors you are getting. You can see it here:
      http://www.youtube.com/watch?feature=player_embedded&v=ssnrJTNrX0k
      Ignore all the course-specific stuff, of course.
      Good luck!
      Bill

      • Sam Martin

        Hi,
        Asher, Bill – thank-you so much for pursuing this. By your joint help, I managed to get the script working (by watching Bill’s very useful video, I found I had forgotten to add the Callback URL).
        Everything is working perfectly now, and like Asher, I want to thank-you Martin for creating this script, and the supportive community around it – really excellent work. And thanks Bill for your great additional resource.
        Best,
        Sam

  • Asher Rospigliosi

    @Bill, thanks, your tutorial really helped. I now have a working copy, I had missed an element of the authentication. Your video is a great support, much appreciated.
    asher

  • Asher Rospigliosi

    @ Martin.
    Martin, thank you for the work you have put into this fabulous tool. I have been trying to find an effective way for gathering tweets for quite some while. Your tool is exactly what I wanted. I also love the community of support you have around it. Nice one!
    asher

  • Archives | Manitoba Ed Chat

    […] Our #mbedchat archives are powered by Martin Hawksey’s TAGS v5.0. […]

  • Jay

    Excellent! @Martin – Thank you very much for the tool. This is very helpful for my research.
    (PS. One observation – There are numerous duplicate tweets in the Archive sheet. I hope this will get sorted out soon.)

  • Twitter Archiving | Adam J Stark

    […] Hawksey.  With the power of the Internet I am now deeply indebted to this individual.  This post from the site provides a link to a Twitter Archival tool using Google Spreadsheets.  Avoiding the […]

  • Twitter Archiving Google Spreadsheet TAGS v5 MA...

    […] For a couple of years I've been sharing a Google Sheet template for archiving searches from Twitter. In September 2012 Twitter announced the release of a new version of their API (the spreadsheet uses this to request data from Twitter).  […]

  • Tom Roper

    I’ve used these TAGS scripts since V3, and jolly good they are too. Here’s a puzzle though, I set up one to collect a hashtag, #cilip2014. I know there’s a seven day limit imposed by Twitter, but for some reason it doesn’t collect anything earlier than one time stamped Thu Oct 24 18:37:17 +0000 2013….this was actually half way through the event whose tweets I was trying to archive, so I know there are earlier ones that day. I’ve tried tweaking the collect date from default to -7 days, and running again, in case that would force it to find the earlier tweets, but still it fails

      • ellen forsyth (@ellenforsyth)

        Martin – thanks for TAGS – it is brilliant, but I have hit a snag. It has been collecting #rwpchat since December (and doing the job beautifully), but this week has not collected new tweets since Monday (and there are a couple of hundred new tweets). I keep getting the message “Exceeded maximum execution time”. Any suggestions about how I can get around this? Thanks in advance.
        Ellen

        • Martin Hawksey

          Hi Ellen – I recently had similar problems and made a minor adjustment to the code. You should try opening Tools > Script editor.. and replace the code on the first page with the code here https://gist.github.com/mhawksey/7230553 (New copies of the template include this modification)

  • Alex

    Thank you for this fabulous tool. My only problem is re-using one spreadsheet for more than one keyword. The first time I set up a sheet with a new keyword, everything is good and the archive etc populates perfectly. However I am very excited about its potential and wish to run lots of different queries!
    If I then try to do a new keyword by overwriting the text in the readme/settings page, and again press Run, it says the query is running but the data in archive and the other sheets does not update.
    Do I need to create a new spreadsheet for every new query, going through the setting up tags dropdown and authenticate process from scratch? Or is there some way of completely refreshing the data?

    • Martin Hawksey

      Hi yes currently you need to setup a new spreadsheet for each query. When the script runs it looks for the last tweet in the archive and uses this to only get newer results so if you change the query it will only collect tweets newer that the tweet from the old query (if you see what I mean). It’s not ideal but I’m working on an improved solution

  • The Origins and Power of the #Tableau Twitter Database | 3danim8's Blog

    […] There are a lot of great ideas being generated by people all over the globe with respect to Tableau.  Many people write about their work on blogs and on Twitter.  People are using various hashtags to represent events or ideas and Twitter is a great way to spread the message about their work.  About six months ago, I learned of a technique to capture Twitter data based on any given search term including hashtags and have the information automatically updated and stored in a database.  Termed the Twitter Archiving Google Spreadsheet (TAGS), this work was completed by Martin Hawksey, who is unbelievably brilliant in his work and is openly willing to share his techniques. You can click here to get instructions on how to do this, or click here to learn more about Martin and TAGS version 5.1. […]

  • John Benfield

    Stunning good app Martin – many thanks for creating it.
    It is all working perfect apart from one thing – when I publsi h the Dashboard to the web I get a Cannot read property ‘title’ of undefined error on the Top Tweeters’ graph (although I can see it on the GDrive instance).
    Many thanks in advance.
    John

    • Martin Hawksey

      Hi John – Found similar myself. If you go into the chart edit mode and then come of it it seems to fix it (guessing bug)

      • John Benfield

        Perfect – many thanks

  • Jo M

    Hi! I’ve been trying to set up, but Twitter authentification fails (Url Invalid) using the Callback URL provided. Any solutions about this?
    Thanks!

    • michealclesham

      The same thing has happened me. there are no spaces in the field

  • Day Ten of #ARU10DoT | #ARU10DoT Ten Days of Twitter

    […] lot more about it than I do! If you want to know more about how these visualisations were created, there are instructions online and further instructions on making a visualisation such as the ones linked to […]

  • Sandeep

    Hi Martin,
    Awesome tool and very helpful in my research. Have a query if you could resolve…
    Google spreadsheet reaches its size limit frequently and need to be manually emptied out and download every few hrs. Any better way for long-term tasks?

  • leonard makombe

    I have used TAGS v5 and am very happy with it. Keep us updated should there be new developpments

  • Toni Brugués

    Hi Martin,
    I’m interested to map tweets so I’m wondering if it’s possible to collect only the geocoded tweets.
    thank you.

      • Toni Brugués

        I did it before but it shows all the tweets within the radius, the geotagged ones and the non geotagged ones but having a location in the profile within the radius (the ones that the geo_coordinates cell appears empty).
        e.g. I’m looking for tweets from Barcelona area (41.40,2.15,100km). There are Indonesian Barcelona supporters using Barcelona (or even Camp Nou) as a location. Even they’re tweeting from Indonesia their tweets appear in my search.
        So I’m only interested in the 1st ones, I don’t know if it’s possible though.

  • MASHe Review 2013: Google Apps Script and Google Sheets MASHe

    […] conveniently takes us to February and the top post of the year with the release of version 5 of Twitter Archiving Google Spreadsheet TAGS. TAGS has been a long standing project, version 5 emerging in anticipation to changes to the […]

  • Michael Tsai - Blog - Archiving Favorite Tweets

    […] imported the most recent favorites and did not make them available on the Web, only for download. Twitter Archiving Google Spreadsheet wanted access to my whole Google Drive, and the output is, well, a […]

  • Siva

    Can some one please help to access Tweets from R software

  • Rob

    I am seeing the following error;
    collectTweets TypeError: Cannot call method “getSheetByName” of null. (line 62, file “TAGS”)
    How can I remedy this?

  • #MLA14: A First Look | Far Away, Yet Close

    […] have been using Martin Hawksey‘s Twitter Archiving Google Spreadsheet TAGS (now in it’s fifth version) for a few years now, and it’s what I used to start […]

  • #MLA14: A First Look | Ernesto Priego

    […] have been using Martin Hawksey‘s Twitter Archiving Google Spreadsheet TAGS (now in it’s fifth version) for a few years now, and it’s what I used to start […]

  • Making a Twitter Hashtag Contributor Map using TAGS | MASHe

    […] for doing this increasingly difficult. There is a solution for doing using a modification of my Twitter Archiving Google Spreadsheet (TAGS) project (the guerrilla approach so to speak). The result for #rhizo14 is here (only viewable in non-mobile […]

  • #rhizo14 – Rhizomatisches Lernen | elektrisches lernen

    […] kommt.  Schon allein die Visualisierung dieser Lernbewegungen fasziniert mich. Uns so habe mir das Werkzeug von Martin Hawksey geschnappt und als erstes Lernprodukt (und vorbereitend  für #openreli, den Online-Kurs für […]

  • Twitter Archiving Google Spreadsheet TAGS v5 | ...

    […] For a couple of years I've been sharing a Google Sheet template for archiving searches from Twitter. In September 2012 Twitter announced the release of a new version of their API (the spreadsheet u…  […]

  • Building the Future of Learning Site - CogDogBlog

    […] every project that uses a twitter hash tag I use Twitter Archiving Google Spreadsheet TAGS v5 developed by Martin Hawksey. This not only creates an archive of all tweets, we get a summary of […]

  • Adam H

    Error: “Twitter said: null”
    Thanks for writing this script! I’m getting the above message when doing a test collection. If I run the script with a search term, I simply get a sheet with no results. Any ideas?
    I’m hoping to use the script to compile a list of twitter users (say, bloggers with an interest in pottery). I’m hoping that using the search term “blogger AND pottery” would return the list I see in Twitter’s search.

    • jane

      Had the same problem, here’s what worked for me:
      1. Created the spreadsheet again (clicked on the second link given above, then File> “Save Copy”
      2. Followed instructions 1 and 2
      3. Clicked on Tools > Script Editor > Run > Authenticate Twitter. A Window opened “You are about to authenticate your first App” – clicked on button (this last third step I had done before, but the window hadn’t opened – may I hadn’t been logged in to Twitter)

  • #MLA14 Twitter Archive Added to Academic Commons

    […] were both archiving #MLA14 tweets using Martin Hawksey‘s Twitter Archiving Google Spreadsheet TAGS (“a quick way to collect tweets, make publicly available and collaborate exploring the […]

  • Doc Mike

    There’s one thing missing from all instructions.
    I set up an instance of TAGS 5.1, followed all instructions, set up a search for a hashtag and did a one-time run, and viola! I got a bunch of tweets. Fantastic!
    The next day, I wanted to run it again and… wasn’t sure exactly what I needed to do. I just went to the TAGS menu and clicked the option to RUN. The script ran, but I got no tweets. Then I tried doing various steps over again, but never got anything else to run. I got no error messages, but just never got anything in the archive.
    So which, if any, of the steps in the setup are required to do again before another run?
    If there are no steps that need to be duplicated, I suspect that maybe my problem is related to the fact that I cut all the data out of the archive tab and moved it somewhere else. I assumed that TAGS could run again and just repopulate the blank archive tab, but one of Martin’s posts suggested that it looks for the last tweet to figure out where to restart. I wonder if there is a way to remedy that? Like maybe, copy all the data out, but leave the last tweet?
    -Michael

  • A look at Twitter and #OERS14 | CConley.ca

    […] the keynote speakers and workshop presenters on twitter (#OERS14).  As with previous years, I used TAGS v5.1, a spreadsheet utility developed in Google Drive by Martin Hawksey, to collect and archive the […]

  • Report on Teaching a Twitter Chat Class | A Wee Blether

    […] In short: I struggled to find a strong (free) one. That said, I was pretty thrilled to get the Google Docs TAGS working. I have no idea what to do with a spreadsheet of nearly 700 class-related tweets, but it’s cool […]

  • Ben Colmery

    Hi Martin,
    Did you remove the ability to archive favorited sheets in 5.1? I’m not seeing it, unless I’m missing something.
    I’ve been running 5.0 since you added that functionality. I don’t want to lose it, as it is crucial for me.
    Hoping I didn’t overlook something in the comments addressing this issue.
    I’ve also noticed in 5.0 (and IFTTT when I used to use it) that if I retweet a tweet, and then favorite an earlier tweet, 5.0 doesn’t pull the favorited tweet. It’s like the retweet creates a time barrier. Maybe just a quirk of Twitter’s API? Anyway, I thought I’d raise it.
    Love the sheet. Hoping I can upgrade. And would like to spread the word to a class I’m teaching, but favorites are key to that.
    Thanks!

  • A few small discoveries.

    […] TAGS: With thanks to Martin Hawksey, I successfully managed to put together this visualization on the interaction on Twitter […]

  • Perry

    Hi, I am new to this awesome API and am doing a cool project here at the university. Question. is there a limit to the number of handles I can list in a query? I have a list of about 100 twitter individuals that I would love to follow and download and study. Thank you for your help and advice.

    • Martin Hawksey

      yes there is a limit to the query length (I can’t remember exactly but the answer is in this thread somewhere)

  • Vectoring The #Thoughtvectors - CogDogBlog

    […] and of course am automatic thing was setting up a Hawkseyian Twitter Tags worksheet to archive #thoughtvector tweets. Right now, its mostly Tim and Jon saying […]

  • Twitter in the Classroom: Early African History | jill.e.kelly

    […] them a class hash tag #HIST2391. And I planned to use Martin Hawksey (EdTech Explorer)’s Twitter Archiving Google Doc Spreadsheet to keep track of it all. (That lasted only a couple of weeks…. I think more on account of my […]

  • Brian G.

    Hi Martin,
    I responded to another partially-relevant comment elsewhere, but thought it might be too buried for you to see.
    First off, thanks so much for this tool!
    I’m trying to collect only those tweets that have geocodes. I have found elsewhere how to collect only those tweets associated with locations within a predetermined distance from a coordinate point (params.geocode = “-x,y,zmi”). But I need to specifically filter for tweets with longitudes and latitudes. How would I go about filtering specifically for them?
    Thanks again for your hard work and for any advice you can provide.

    • Martin Hawksey

      Doesn’t appear to be anything in the API to prevent place association. Is the data good enough for post processing?

      • Brian G.

        Yes it is and I have already written in a place association. What I am trying to do now is filter for only those tweets with geo.coordinates[0] and geo.coordinates[1] data. Do you have any ideas on how to do this?

  • Amsel

    Hi Martin,
    thank you for this great tool!
    Is there any possibility to check the connections between the archived users?
    I would like to know how many people from the list (sheet: summary) a user is following.
    I didn’t find a solution during my web-research, but I think it might be possible through this api: https://dev.twitter.com/docs/api/1.1/get/friendships/show
    Do you have any idea how to solve it? (sorry, i’m no coder – so i’m not aware if it’s a big deal.)

  • Twitter archiving tool | MARS Lab

    […] Hawksey has developed a useful tool for archiving Twitter searches.  Twitter Archiving Google Spreadsheet (TAGS), provides an effective means to capture tweets up to 7 days old and works especially well when […]

  • carla2198

    Thank you for sharing this, I’m VERY grateful for your work! Right now, I’m getting an error on the sheets I’m using saying “TypeError: Cannot call method “getSheetByName” of null. (line 62, file “TAGS”)” I’m not sure what that means and how to fix it. I’m not very well versed in this, but have found this to be INCREDIBLY helpful when it was working, now I’m having to do it by hand, because of these errors. Fixing this would save me a LOT of time. Thank you in advance.

  • Making use of scraped Twitter analytics - Taras Young

    […] (which at £6 a month doesn’t really break the bank) there are free and open source ways to analyse your account’s performance and interactions. However, those ways have to already have been running on your account. What if you wanted to […]

  • Learning Analytics in Education and a data dogma?

    […] data collection as part of my personal quantified self and data collection for quantifying others. TAGS is a prime example of where I initially wanted to collect data to understand the shape of the […]

  • Ricardo Salinas

    How i can make this in mysql and php? because google drive work very slow for me when i try to search with this script, can anyone help me?
    Regards.

  • Tim

    I love this sheet and you for creating it. SO helpful.
    I’ve had great luck with it so far, but for some reason today it’s only returning 100 Tweets when I search for a specific term (a hashtag that I know has more than 100 tweets). Do you know why it’s limiting the archive tweets to only 100? I left it at the default 1500 results on the settings page.
    Thank you for your help!

      • Tim

        Yup – 5.1. Thanks!

  • tim

    fantastic sheet, thanks so much for creating it. I managed to fill it up rather quickly (gluttonous clearly) – google have announced larger spreadsheets are now available but I wasn’t sure how to transfer the data into a new sheet?
    again, very grateful for you work!

  • Tim

    Still getting only 100 results. Anyone else have this problem?

    • Dave

      Also only getting 100 results, even on lists of accounts that have previously generated 1000+. No clues as to why, unfortunately.

    • Mogens

      Yup! same problem here. Did you find a way to fix your problem? We made two archives work in february, but this week, using v51, we’re only able to collect 100 tweets in retrospect – and after that, it’s only adding newer tweets in hourly updates. Tried paged searches for e.g. -7d, -6d, and older data are positively retrievable. But still, no initial or continuous inclusion of those. Strange.

      • Tim

        Sadly, I never heard back and had to go with a paid version.

  • Como fazer mapeamento de posição social no Twitter |

    […] que estão discutindo um tópico específico usando uma hashtag específica. Um grande exemplo é o Explorador de TAGS de Martin Hawksey, que pode ser usado para arquivar e visualizar uma conversa do Twitter através […]

  • Für die Ewigkeit? | Test

    […] aktuellen Authentifizierungsform von Twitter zusammenarbeiten kann, gefunden: TAGS in Version 5 (Link). Hierbei werden alle Tweets (soweit in die Vergangenheit, wie die Suche es erlaubt) in einem […]

  • Für die Ewigkeit? | Linguence

    […] aktuellen Authentifizierungsform von Twitter zusammenarbeiten kann, gefunden: TAGS in Version 5 (Link). Hierbei werden alle Tweets (soweit in die Vergangenheit, wie die Suche es erlaubt) in einem […]

  • Cornelius

    Hi Martin,
    thank you very much for sharing your work, works fine for me so far. I was wondering if it is possible to set the starting point for the data collection even below -1 day. That would be a great help for me dealing with discussions, which generate thousands of tweets a day. The perfect scenario would be that the run now command just starts gathering data from the moment of its first execution. Is that somehow possible?
    Thanks
    Cornelius

  • lmakombe

    I have archived six different documents on the same hash tag, is it possible to merge them without losing the metrics such as number of unique users etc

  • eyesonice

    Dear Martin,
    Is there any way to retrieve the number of times a tweet has been favorited. I would like to retrieve tweets, and 1/2 days later, using just its id, retrieve the number of times (and by who/date-hour) it has been favorited during these 2 days.
    Thanks again for your great tool and help!
    Frederic

  • Karla

    Hi Martin,
    Thanks for setting this up!
    I am currently having a few technical difficulties. First, I am getting a error message when clicking on the web views link. I did file>publish to web and I am still getting the error message.
    I am also having a problem using the mozcontemplate – http://builtvisible.com/wp-content/uploads/2013/07/template.xlsx (SEO Tools is installed)
    Once I open the template and enable editing I get 5 different error messages.
    Error Messages received:
    1.Unshort URL error
    2: Twitter count error
    3.Google Plus Utils error
    4. Xpath On URL error
    Once I clear the errors messages, I import data from web (using the link in the tags google doc-file>publish to web section). The data is populated in the twitter archive sheet. The data in the charts sheet do not refresh and the analysis sheet once information is dragged down, everything to the right of the extracted URL does not populate.
    I am at a lost and looking forward to your advise.
    Thanks,
    Karla

  • Under the Hood of Thoughtvectors.net - CogDogBlog

    […] and we’ve been lighting up the #thoughtvectors hashtag for weeks. We are running the Magically Delicious Twitter Tags Spreadsheet by [Sir?] Martin Hawksey, and I have them all embedded into local pages on the site– the most insanely useful one to […]

  • Kathryn James

    Many thanks for this Martin, It is really simple to follow and I am hoping to use it for a conference this week. However I am having a small issue with the fact it will collect the #enconf14 but will not pick up @enconf if I put them in the box as
    #enconf14 AND from:@enconf
    I realise you are not a Q&A service! But could you let me know if I am doing something obviously wrong? I have googled but can’t seem to find anything.
    Many thanks
    Kathryn

    • Daniel

      Kathryn, try the search without the @ symbol.

  • Ellis

    Hi Martin, I set everything up smoothly. However when entering my selected tags into the search bar and moving onto the archives page, nothing is popping up. It does say ‘running script’ at the top however. Does it take a while for data to pop up if a commonly used tag? Or has something gone wrong here? Thanks!

    • Sam Martin (@DigitalCoeliac)

      This happens to me from time to time. Sometimes, authorisation does not always work with the Twitter API, and you have to try a few times to make sure you see both pop-up boxes asking you to authenticate in the main TAGS>Twitter Api authetication, AND the Tools>Script Editor menus. What I advise is to:
      1) Close and reopen the sheet (to kill any scripts that might have crashed),
      2) Once the sheet is reopended, go back to Tools> Script Editor, and Run>authenticate Twitter again (make sure the white box connecting to the Twitter API opens and asks you to allow the script app to use your Twitter account.
      3) Then go back to the sheet and run the collection process from the TAGS>Run now menu, and it should work after a few moments.

  • Marian

    Hi Martin, thanks for sharing your FANTASTIC tool! It’s a great help for my PhD as I am working on very similar graphs, but they are not interactive yet (http://www.md2c.nl/twitter-community-overview/). However, I could not get the getCommunityGraph() working: I got vertices, but no edges. Any idea?

  • Networking City

    Hello, Martin. Thank you for your great work. I really like to use it.
    But I cannot use the function of collecting tweets on specific time within 7days.
    I set 1) Tags> Update archive every hour 2) Tool>Script editors> project’s triggers>specific date and time, but it was not operating.
    Would you let me know the right way to use this function?
    Many thanks.
    Seungho

  • Know your Twitter Community and Prioritize

    […] who to follow or to have a conversation with on Twitter. Thanks to scripts from Amit Agarwal and Martin Hawskey, it’s now easy to get more insights about your community. Inspired by their work, I created a […]

  • MD2C Know your Twitter Community and Prioritize - MD2C

    […] to have a conversation with is sometimes challenging. Thanks to scripts from Amit Agarwal and Martin Hawskey, it’s now easy to get more insights about your community. Inspired by their work, I created a […]

  • Feed WordPress 101: Some Feed Magic - CogDogBlog

    […] for any project where tweets are happening, especially with a hash tag, is to always set up a Twitter Tags Archive in Google Docs as created my Martin Hawksey. His post provides instructions and a video to guide […]

  • MD2 Consultancy

    Great tool! You might want to get also the https:// links counted by tweaking your formula a little: =ARRAYFORMULA(SUM(IF(ISERROR(FIND(“http://”,LOWER(‘Archive’!C:C))),0,1);IF(ISERROR(FIND(“https://”,LOWER(‘Archive’!C:C))),0,1)))

  • philippallerPhilipp

    Hi,
    First, thanks for your great work. It has been very beneficial to my reasearch. I however noticed that the columns “in_reply_to_user_id_str”, “in_reply_to_screen_name” and “geo_coordinates” remain empty in my spreadsheet. Is there a reason or something I’m doing wrong? Thanks in advance.
    Cheers, philipp

    • Daniel Eshuis (@deshuis)

      Hi Philipp,
      “in_reply_to_user_id_str” and “in_reply_to_screen_name” will only be populated if the tweet was a reply to someone.
      Similarly with “geo_coordinates”, this will only be populated if the user adds their location to the tweet.
      Daniel

  • ldnvoteswatch2014 (@LondonVotes)

    Hello,
    I am new to this twitter scraping spreadsheet. When I test the collection or run the script, it throws the following error:
    “Message details
    TypeError: Cannot read property “length” from undefined.”
    Any idea(s) on how to solve this?

    • ldnvoteswatch2014 (@LondonVotes)

      SOLVED – PEBKAC – incorrectly performed the following:
      3. When your Key and Secret are saved you need to open Tools > Script Editor … and run the ‘authenticateTwitter’ function

  • ldnvoteswatch2014 (@LondonVotes)

    Having trouble with triggering:
    getTweets>Time-driven>Hour timer>Every hour
    Execution failure notifications
    via email at [email protected]>immediately
    No notifications have been emailed, and no updates have occurred without manually using “Run Now!”. Any ideas?

    • ldnvoteswatch2014 (@LondonVotes)

      Ok, so a failure notification email was eventually sent. I guess setting this to “immediately” is not working.
      The email says:
      Summary:
      Error Message Count
      Authorization is required to perform that action. 11
      Followed by a list of 11 instances were getTweets tried to trigger but failed because of “authorization required to perform that action”
      I do not know what authorization I need to give this, as I’ve got it working for “Run Now!”. Any ideas?

      • Martin Hawksey

        hi – when you open tools > script editor and run authenticateTwitter does it popup an authorisation window?

        • ldnvoteswatch2014 (@LondonVotes)

          Well, it did open a popup window for authorisation in the past. Now it does not, though.
          Instead I now get a script warning:
          ScriptProperties API is deprecated.
          File: TAGS Line: 27
          The API has been marked as deprecated which means that the feature should be avoided and may be removed in the future. Consider using an alternative solution.
          Line 27 being the following from authenticateTwitter:
          ScriptProperties.setProperty(‘active’, SpreadsheetApp.getActiveSpreadsheet().getId());

  • suzankoseoglu

    Hi, I’m trying to find a way to download an entire hashtag (#thoughtvectors) for research. Is the number of tweets I can archive using TAGS v5 limited to the last seven days? Thank you!

    • Martin Hawksey

      yes – but someone else might be archiving it using tags (worth pinging @cogdog)

      • suzankoseoglu

        Thanks a lot, I’m exploring few other tools, but will connect @cogdog if I can’t find a good tool

  • soozyn

    Having same issue as @LondonVotes. You say it still works tho? This is my first time using. I hit “Run Now!” and it still says “Running Script” after about 20 minutes. How long does it normally take?

  • » Digital Frontiers 2014 Anna E. Kijas

    […] Europeana Sounds, PennSound, and others) could be discovered by analyzing Twitter. She used the Twitter Archiving Google Spreadsheet (TAGS) to collect data and analyze how many people were discussing these digital sound libraries, […]

  • belinda

    I love this script, but the loading of the datavisualization takes too long en mostly it times out. I tried with a fresh archive (500 tweets), but no change.

  • #LAK13: Recipes in capturing and analyzing data – Twitter | MASHe

    […] As well as not being free the raw data isn’t also always available. My solution has been to develop a Google Spreadsheet to archive searches from Twitter (TAGS). This is just one of many other solutions like pulling data directly using R and Tableau the main […]

  • Leida Dos Santos (@Anfileida)

    Dear Martin,
    Thank for setting up this. I am having trouble running the script. It doesn’t seem to work. I don’t get the same window as you, and the list on archive.:( Am I doing something wrong?

Comments are closed.

css.php