Update: TAGS v6.0 is here! You can jump straight to the new site. If you are using version 5.x here are instructions on how to upgrade (there is an old Sheets version of TAGS which is recommended for use with TAGSExplorer).
For a couple of years now to support my research in Twitter community analysis/visualisation I’ve been developing my Twitter Archiving Google Spreadsheet (TAGS). To allow other to explore the possibilities of data generated by Twitter I’ve released copies of this template to the community.
In September 2012 Twitter announced the release of a new version of their API (the spreadsheet uses this to request data from Twitter). Around the same time Twitter also announced that the old version of their API would be switched off in March 2013. This has required some modification of TAGS to work with the new API. The biggest change for TAGS is that all requests now need authenticated access.
So here it is:
*** Twitter Archive Google Spreadsheet – TAGS v5.1 ***
[If the first link doesn’t work try Opening this Spreadsheet and File > Make a copy]
Important: Uldis Bojars (@CaptSolo) has highlighted an issue with TAGS V5.0 which meant it would not page all the results. This has been fixed with the release of v5.1. If you are running version 5.0 of TAGS it’s highly recommended you update the code.
We find that the search API over-represents the more central users and does not offer an accurate picture of peripheral activity; we also find that the bias is greater for the network of mentions – González-Bailón, Sandra, et al. “Assessing the bias in communication networks sampled from twitter.” Available at SSRN 2185134 (2012).
Instructions for setting up TAGSv5
Instructions are included on the Readme/Settings sheet of the template. If you are having problems it’s worth checking Instructions for Tags v5.0 written by Stacy Blasiola (@Blasiola) or this modified version by Karen Smith & Shanifa Nasser made for Open Data Day Toronto available as CC-BY-SA.
What will happen to my existing TAGS sheets that aren’t version 5.0?
When Twitter turn off the old API (test outages this March) all authenticated and unauthenticated search requests will stop working.
How do I upgrade existing versions of TAGS spreadsheets (v3.x to v4.0) to keep collecting beyond March 2013?
As I can’t push an update to existing copies of TAGS you’ll have to manually update by opening your spreadsheet, then opening Tools > Script editor… and replacing the section of code that starts
function getTweets() {
and finishes 134 lines later (possiblly with the line
function twDate(aDate){
) with the code here. [And yes I know that’s a pain in the ass but best I could do] … or you can just start a new archive using TAGSv5.0
More additional tips and info when I get a chance
Permalink
Permalink
Thanks Martin – good job :-)
Permalink
Excellent – Thanks Martin.
Permalink
Permalink
Thanks for this Martin. But I’m getting the “Script function onOpen could not be found” error message when I try to use the spreadsheet. I’ve tried making a new copy but with no better results. Any ideas? Should I try again later?
Permalink
Wait – it is working, but I was pressing the wrong button. I’ve got two buttons – one is labelled Enable Custom Menu – that gave the error. There is a blank button next to it that worked (?!).
But now I’m getting an authentication error: OAuth Error (line 376, file “Google(c) Code”).
Permalink
hi
was trying to use TAGS explorer but getting error message “Error in query: Access denied Access denied”
thanks
Permalink
Hi Martin,
thank you for your tremendous work and the new version of your TAGs sheet!
Similar to Sharon I also got the “OAuth Error (line 376, file “Google(c) Code”)” error. This error occurs when I try authenticateTwitter in the Script Editor.
Thanks
Jonas
Permalink
Sounds like your key/secret are entered incorrectly or a mistake in the callback url entered in dev.twitter.com
Permalink
Thanks for your reply! It was indeed the missing callback url. If anyone’s interested, the solution to this error is the callback url “https://spreadsheets.google.com/macros”.
thanks again!
Permalink
Like Peelie I get errors with the 2 Web view links at the foot of the dashboard. I’ve also tried entering the URL/key at http://hawksey.info/tagsexplorer/- it just sits there looking at me!
Permalink
For those views you need to make sure the spreadsheet is File > published to the web
Permalink
Doh…now I feel really stoopid! Thanks for the speedy reply.
Permalink
How many Tweets does this archive? I can’t get it to archive past about 80, instead of the 1500 like the last spreadsheet.
Thanks!
Permalink
I imagine you are hitting the Twitter Search API 7 days limit (anything older isn’t available). I’ve got v5.0 running on #edcmooc and it’s beginning to creak at 15k tweets
Permalink
Hi Martin,
Thanks again for setting this up.
I am running into issues exporting the spreadsheet. I’ve tried all of the options, but they all return the same error: ERROR: Export to .csv failed.
Any ideas why this is happening?
Thanks!
Stacy
Permalink
It happens when the number of tweets is greater than 10k, more or less. What you can do is to copy the file, and delete half the tweets in the original, and the other half in the copy, and export both…
Frederic
Permalink
Permalink
Hi,
Thanks for this. I may be a little confused, but do is the newest version of TAGS (ie v5) sufficient to work or do we have to change the script in like it says?
Thanks
Ashley
Permalink
Hi Ashley – I’ve clarified the text slightly. v5.0 requires no update to the script.
Permalink
THANKS a bunch for this work! Do you have a location for contributing to the work fund?! I definitely am happy to donate a bit to the cause.
Permalink
Just want to thank you very much for putting this up and making the public search interface available
Permalink
Permalink
Hi Martin,
I’ve gotten through everything smoothly, but when I try to perform a search, the “Running script collectTweets” box just remains on-screen indefinitely and nothing ever appears in the archive. Any advice on what I’ve done wrong?
Thanks so much
Permalink
So, I followed all the steps, set everything up correctly… but I have a question about how much data is pulled. I am noticing an inconsistency in the number of tweets by a particular person via a particular hashtag. Also, the google doc “dashboard” lists a little over 1500 tweets and shows that the first tweet was made 2 days ago (Feb. 28). And the analyzer only shows from March 1st…
And I just noticed that the google archive is missing a good number of Feb. 28 tweets… (also, is it possible to change the time zone?)
Anyway, sorry to complain, want to say that even though I’m having problems this is amazing.
Permalink
The twitter API is limited by the last 1500 tweets and/or 7 days. So if you made a search on the 1st March it’ll pull the last 1500 tweets upto a 7day limit (which ever one it hits first). Do you think that explains your problem? Not sure about timezone, you might be able to go into Tools Script editor and change something in File > Project settings(?) but I’ve low confidence that will work (it may need to be tweaked in the code)
Martin
Permalink
Thanks for the response…
Curious, is there a way I can embed the archive or explored on my webpage?
Permalink
Dear Martin,
I was so excited to find this and I have followed the instructions carefully and I am desperate to make use of it and share it with others. Two surprise things:
1) at min 3.25 in your video the authenticate twitter step – I did that, but I got no popup to authenticate on twitter – I did get the ” the yellow “running function pop up at min 3.28, but no “authorization required at min 3.28 or the subsequent twitter authentication window.
As a “hack”, I did see on the developer page that I could create my own token right from the Twitter Developers site – “Use the access token string as your “oauth_token” and the access token secret as your “oauth_token_secret” to sign requests with your own Twitter account. Do not share your oauth_token_secret with anyone.” and I Verified the following was in my twitter profile: – Julie Archive by
A way to archive my tweets
Permissions: read-only
Approved: Sunday, March 3, 2013 9:34:02 p.m.
-If this is not correct this may explain why my spreadsheet isn’t working (see #2 below).
2. At mins 4.10-4.15 I ran a search for the hashtag #ihateschool – i went to menu TAGS, RUN NOW! and it’s just hanging at “Running a script collectTweets Dismiss”
And Nothing is going into the archive window.
Any help you could offer…?
Thanks,
Julie
Permalink
Hi Martin, any idea why the Twitter application returns “Not a valid URL format” when I enter the spreadsheet url set out above?
Permalink
Hi Martin – First thing I’d check is there is no space before or after the url. If it’s still not working let me know
Martin
Permalink
Great resource, thanks!! Do the advanced functions such as get followers work? I did try and it looked like it was working but a new tab was not created with the data.
Permalink
Hi Mary – the sheets data are written to are hidden be default. Click on the icon bottom left with 4 thick black lines to unhide them
Martin
Permalink
I have used your TAGS spreadsheets before. I followed all the directions on your video and have the new Twitter key and secret. When I try to authenticate twitter, it flashes quickly with “running function authenticate twitter” but then nothing happens after that.
Permalink
Got it! Forgot to put in the callback URL.
Permalink
First of all, thank you for this tool. It is probably the most useful twitter tool I have ever come across.
Second, is it possible to use the advanced search operators ‘near:’ and ‘within:’ ?
I have tried this in several different combinations with single, double and no quotes and cannot seem to get it to work.
Thank you.
Permalink
Hi Scott – Unfortunately the Search API doesn’t support near: and within: It is possible to add a geocode parameter manually in the source code. If you open Tools > Script editor around line 120ish you’ll see a number of params are set eg params.count = advParams.count || 100; you can add another line with something like params.geocode = “37.781157,-122.398720,1mi”; setting your own latitude, longitude and radius (see documentation for details)
Permalink
That worked perfectly, thank you!
A great tool just got even better.
Permalink
Can you provide any more detail on this geocode solution. I edited the code as suggested and continually come back with no results. Any help is appreciated.
Permalink
The geocode filter was exactly what I was looking for. Thanks for such a great tool and for the support :)
Permalink
Martin, this is a *great* application – as a social media user and researcher it’s great to have a customized search that I can access through Google Drive. However, I just wanted to confirm that the -7 day limit was a Twitter API issue, yes? We’re trying to pull hashtags from catastrophic events (such as the High Park Fire in Colorado, #highparkfire) but it seems that we can’t get those, as they would have been from June 2012.
We’re able to use the “regular” Twitter search to get access to them (https://twitter.com/search/realtime?q=%23highparkfire&src=typd) but it seems that there is no way to pull these into an Excel sheet or file such as Tag v5.0?
Permalink
You are correct the api is limited to last 7 days. Twitter recently started rolling out the inclusion of older tweets in it’s search results but this data is currently not accessible to anyone else. Datasift have a a historics sou for older data. The data isn’t free but very cheap
Permalink
Thanks so much for the reply, and for the great program. I’m having some trouble getting the visualizations to show up in the Google Doc (lots of “#name” errors) but the TAGSVisualizer is pretty slick.
I’ll follow this project often and recommend it to others – and you’ve provided our lab with an invaluable Tweet archiving tool that we will be sure to credit you with in any publications. =)
Permalink
Thank you Martin for doing this. Brilliant resource.
I have tried setting it up twice and when i run it i get the following message “Exceeded maximum execution time” after a while, using different search terms.
Any ideas?
Alan
Permalink
Hi Martin,
I’ve set up one of these archives and it went fine, but today I’ve had another go with V5.0 and I’m getting an error message.
After Twitter API Authentication is completed (I made a new app, but I’ve also tried it with the old app), I authorise the API, put in the search term (a hashtag) and click Run Now.
I then get asked for ‘Twitter API Authentication required’, and a red erro message, “TypeError: Cannot read property “length” from undefined.”
Any ideas what I’ve done wrong?
Many thanks, Su
Permalink
Hi Su – when you run the script authenticateTwitter does it redirect you to Twitter where you approve the connection?
Permalink
To be more accurate, it says ‘Running Function authenticateTwitter’ and then that disappears and nothing.
Permalink
Are your Twitter API keys/secret set?
Permalink
Yes, made a new application as per instructions.
Permalink
The Twitter API settings have the correct callback url (in the callback field)?
Permalink
Hi, I am experiencing the same issue. Have been using the tool for about a month now and suddenly it has stopped working and is giving me the error “Twitter API authentication required” — my access token and secret are set, and I have triple-checked the callback URL.
Then today I have also started receiving the error “TypeError: Cannot read property “length” from undefined.”
Any thoughts?
Thanks so much!
Kelly
Permalink
Hi Martin,
Yes all the settings were fine, i went through them again.
Then this morning I clicked Twitter API Authentication and ‘OK’ again, and chose ‘Run Now!’ and the script just worked. No idea what I did, it just worked fine this time.
Thanks for your patience!
Permalink
Yes it did, though I’ve just had another go via the script editor and it just sits there when I ask it to authenticate twitter.
Permalink
This is happening to me too. I just set it up to try to run it again and I got the same thing: TypeError: Cannot read property “length” from undefined
Permalink
Hey – All, I got this to work finally! I was having trouble with the “run thing” issue as well – then i just went to the link provided by Martin here – https://docs.google.com/document/d/1N838xH6NeRa6_kZQVkB9tBCliJo_8drvPuM8Ds6tsHg/edit
And I followed those instructions exactly. I didn’t get the results I was expecting so then I went back to Tools, Script Editor and I ran the collectTweets function and now it is working.
Permalink
Permalink
Hi
A previous version worked for edges and a simple network analysis.
I tried to fill “edges sheet” but it didn’t work out. I know NodeXL does social network analysis, but it does not update archive.
Thanks much for your excellent work!
Permalink
I’m also having some trouble with the “getCommunityGraph” function for filling the edges – it does fill the edges but i can’t make out what exactly is being displayed, whether they are “following” vertices or retweet vertices.
Sometimes i come up with a “#Value” in the cell, but can’t figure what the error is.
Didn’t understand how the “getCommunityGraphfromsaveddata” works, as well.
What I’m actually trying to figure out is the best way to import retweet relations into Gephi, any advice would be much appreciated!
Besides that, i too must pay homage to the incredible work you’ve done! You rule man! Thank you veeery much!
Permalink
Permalink
Permalink
Permalink
A big thank you for this great tool. Please, can the archived data be exported into Gephi (or any other network analysis tool) for analysis? Once again, THANKS!
Permalink
Permalink
Hi Martin
I have a problem with authentication, and I can see that is not only my problem. I did everything exactly as described, but when I go to Tools > Script Editor, and run authenticate Twitter, nothing pops up to get me to the Twitter authentication page. Hence, the app does not work. Any idea what is wrong?
Thanks
Hossein
Permalink
Permalink
Hi Martin,
firstly thank you for sharing your work, I have been using using the various incarnations of TAGS spreadsheet (v.3 – v5.05) extensively in my day-to-day analysis. Lately, I have been looking at the code closely and have a question. I am currently searching 24 @keywords and limiting the number of tweets too 100. As soon as I add the 25th @keyword I get the “Execution Timeout Exceeded”. Changing number of Tweets doesn’t make a difference. After a bit of research I came across the following link;
http://stackoverflow.com/questions/7854573/exceeded-maximum-execution-time-in-google-apps-script
And wanted your advice as to whether in principle/theory it would be possible to modify the collectTweets function to run in a similar way to the runMe() function in the link above. A timed batch process of var SEARCH_TERM and do it in chunks.
Hope this makes sense, and thanks again for your contribution.
Permalink
Hi, quick Q are you using a single spreadsheet for you 20+ keywords? If so the issue is probably the url sent to Twitter (which will include your search term) is too long.
Thanks,
Martin
Permalink
Morning,
Yes, you are correct, the keywords are in a single spreadsheet (below keywords take 4secs to collect&getTweets). Here is my working keywords set, = 439 Characters;
@Universal_WE OR @mci_group OR @PMMOurWorld OR @Zibrant OR @GPJ_EMEA OR @GeorgePJohnson OR @jackmorton OR @TROexperiential OR @CarlsonWagonlit OR @Banks_Sadler OR @aileen_reuter OR @BCDTravel OR @rpmltd OR @VenuesEventMgt OR @wandoevents OR @logistikgroup OR @cievents OR @FirstProtocol OR @WRGCreative OR @ubm OR @Clarion_Gaming OR @ImaginationLabs OR @aymtm OR @BSIEvents OR @cascade_alan OR @crownlondon OR @drpgroup OR @FirstEventLeeds
Is there a better way of analysing multiple keywords? Should I look to break this down into multiple TAGS files and try to analyse across multiple files?
Thanks,
Michael.
Permalink
I’m not aware of an easy way to get around the query length other than breaking it up over spreadsheets. It would be possible to add some extra code to do this (unfortunately I don’t have time to do it)
Permalink
I’ll keep it to Top 20 for now, thanks for getting back to me.
I monitor some fairly high traffic hashtags that required me to add a couple of functions that deleted tweets based on numRows & numDays. Hope its ok to include them here in case anyone wants to create a new script file;
Permalink
Hello Martin. Thanks for what you have create.
I would create two files for different keywords. Do I just need to make two copies in putting different keywords inside it and it works or there is another manipulation? I’m afraid that the two files with the same settings and authentication are not distinct. I need to do two separate searches with different keywords. It is for this that I want two separate files.
(Sorry for my english, I’m french)
Thank you !
Permalink
Hi, you can use the same file and in the search term use OR. For example, cats OR dogs
Permalink
Permalink
Hi Martin,
I’m getting the following error on V5:
Line 157 Cannot read property “followers_count” from undefined.TypeError
I am able to test the collection successfully.
Any ideas why that might be happening?
Thanks,
Bill
Permalink
Is this using a copy of the template or trying to update an existing spreadsheet?
Permalink
A copy of the template. I created it by clicking on the *** Twitter Archive Google Spreadsheet – TAGS v5.0 *** link.
Thanks,
Bill
Permalink
Hi Martin,
I’m getting the following error on V5 after running the authenticateTwitter function on the script editor:
Request failed for returned code 404. Server response: {“errors”:[{“message”:”Sorry, that page does not exist”,”code”:34}]} (línea 376, archivo “Google(c) Code”)
I have already authorised permission with my twitter account.
Do you know why that might be happening?
By the way, I have another problem with the conversation visualization in TAGSExplorer. I have already publish the Google Spreadsheet in the Web but when I try my URL in the TAGSExplorer, the web is loading for ever with the message “Loading data”. I think that there aren’t too many tweets to be loading so slowly… (only 246), so I suppose it may be another thing… Any idea to solve this?
Thank you very much and good job!
Victoria
Permalink
Hi, thank you for TAGS, you may save me weeks of work!
I’ve followed the instructions (got special menus, authorized the app), but when I click on “run now!” it just says “Running script collectTweets Dismiss” and stays like this, i can still do things in the sheet, but it doesn’t give me any tweet in the archieve.
What can I do to make it work?
Permalink
Hi, As far as I can see this is to do with Google infrastructure and have run out of optimisation solutions. For some reason on the initial collection I find better performance if I open tools > script editor and then Run > collectTweets. Once an archive is running they generally update without any fuss. Seems to be an issue around when the spreadsheet is first copied.
Permalink
thank you, I just opened the script editor and ran the collectTweets from there, a popup asked the app authorization and after it started working normally
Permalink
I get the same as Pasquale.
Permalink
Hi there, can I check if this only pulls in organic tweets and not promoted tweets?
Permalink
yep only pulls organic
Permalink
Permalink
Permalink
Great job, @Martin and thank you for sharing it with the world!
Couldn’t find an answer to this, so asking here- does TAGS support a combination of logical operators for the search term. Specifically, I am looking for something like: a AND (x OR y OR z).
Cheers,
Ned
Permalink
Hi Nedyalko – yes you can use combinations of operators. One thing to watch is your query string doesn’t get too long (the maximum is 1,000 characters – this includes some parameters sent by the script so it’s a little less than this)
Martin
Permalink
Permalink
I use your script, but I wanted to create new one and it seems that something is wrong. I have only “running script” but no data is pulled from TW. Do you have any idea?
Permalink
not authenticating is where most trip up
Permalink
I’m having problems with this, too. I’ve authenticated several times, but cannot get the custom menu to install. It doesn’t matter how many times I click on the button, give authorization, and click “accept”, it does exactly the same thing, and I just end up in a loop. Baffled.
Permalink
hmm – have you tried taking a fresh copy of the template?
Permalink
I had the same problem with the message “running script” and no data pulled from the spreadsheet. In order to solve it I tried to reload the spreadsheet, then went to Script Editor and authenticated again. Then closed the editor and from the spreadsheet I re-ran the script to get the data, and this time it worked like a charm!
Permalink
Permalink
Permalink
Permalink
Thanks for this awesome tools! :D
Tweet mining is so cool too…
Permalink
Hey Martin,
I just have a question about rate limitations. I know that it is possible to search up to 15,000 tweets or tweets from the past 7 days (which ever limitation is reached first), but would you be able to tell me if any further rate limitations exist specific to this software? For example, I know that I was having a problem at one point where I made too many searches in one 24-hour period, so my searches were unable to run any further. Is there an exact rate limitation in this regard? Maybe the 15K/7-day rate limitation applies in this scenario? Thank you!
All the Best,
Brendan
Permalink
Hi Brendan – the api docs state 180 calls per hour. The script fetches 100 tweets per call. TAGS Advanced menu has a ‘Test Rate’ menu option for you to monitor remain calls
Permalink
Ah, perfect! Exactly what I was looking for; thank you!
Permalink
Hi Martin – I’ve been experimenting with the TAGS v5.0 script to explore how to archive our #cdnedchat sessions. I have had trouble getting the archive to show times during the turnover from end-of-day to next (23:59:00 June 3rd to 00:59:00 June 4th) in the search.
I’ve tried changing the settings up (2 days, 1 day, 3 days) but that particular hour is just not showing up in the search. As it happens, that’s the exact hour when our chat happens online.
Any way for me to fix this? What do I need to adjust in the readme/settings?
Thanks!
Permalink
Okay, I went back to Tools, Script Editor and I ran the collectTweets function and now it is working – all tweets are showing up. Just have to figure out how to get the analytics to work with the “paged” archives. Slowly but surely! Thanks for such an amazing resource!
Permalink
Hi Tanis – glad you got it working. The paged collection is a feature from the very early versions of this template and I haven’t really been looking after it. My recommendation would be to use the single archive version. If you want to give users easy navigation per chat the MLAA developer TAG Viewer is a great option. Here’s an example on how the TAGS sheet for #cdnedchat is rendered. Alternatively there is this searchable interface I’ve created
Martin
Permalink
Brilliant! Thanks for the suggestions regarding archiving multiple chats in one archive – love these resources you have created!
Permalink
Permalink
Thanks for this great tool. Really a brilliant thing. I also got a question. I just ran a sweep but was wondering why it doesn’t seem to go very far back in time. Is there a way to make it sweep tweets from the past few days? Right now it only goes back for about a couple of hours even though I know that there is much more. Any ideas?
Thanks again.
Micha
Permalink
Permalink
One more question…does the API v1 Retirement by Twitter on June 11
(see https://dev.twitter.com/blog/api-v1-retirement-date-extended-to-june-11) impact the script to be working properly?
Thanks again for your good work.
Permalink
Version 5 was developed specifically to address the api changes
Permalink
Since yesterday the script is not working. There is a problem with the variable followers, that I am suppose that is due to the retirement of the API v1. I have been using your script for several months now, and I wanna thank you for that.
Permalink
Martin this is the error that I get:
Apps Script
Line 157 Cannot read property “followers_count” from undefined.TypeError
I tried to set up a new gsheet but I get the same error. It was reported that twitter has made some changes on how to get the followers, but I didn’t have so far the opportunity to search for more details.
Thank you again, for your work.
Permalink
Thanks for Tagsv5.0! It is of immense importance to me. I am PhD at the Wageningen University & Research centre (the Netherlands) and I am doing a research on the influence of twitter-acitivty on the governance of sustainability in agro-food systems. I’ve made five documents for archiving tags, but now I encounter two problems: 1) only my first two are working (1 shared, 1 private), the others do not archive data, and 2) Tagsexplorer isn’t working for any: it says ‘loading data’ (for 8 hours).. I hope you can give me any direction for solving these problems. Keep up the great work!
Permalink
for the Tagsexplorer problem: https://docs.google.com/spreadsheet/pub?key=0Ary8s3DW0yExdGhZNFlQUXh1QmY2ODJsYk9QS0FTc0E&gid=36
Permalink
I am facing same error as M.M. Script is not working. Plus, I get a redbox saying something about the length property.
Thanks for your work
Permalink
Thanks for this great tool. I am a PhD researcher and this is of great value to me!
If you have problems with TAGSExplorer: it might be because you’re using Internet Explorer, try Google Chrome instead. It worked for me :)
I can’t make new archives. I’ve two running properly, collecting about 40 tweets a day for a few weeks now. When I make a new one and click ‘run now’, it says
Apps script:
Twitter API Configuration Required
And then, when I close that window, it says in red:
TypeError: Cannot read property “length” from undefined
I’m sure I’ve followed all steps, including the Twitter API authentication procedure (you can use your key and secret multiple times right?) and authenticate Twitter via Tools, script editor (you need to do that for all docs?). I haven’t archived many data yet, so I guess it’s not the amount of data I am pulling in. Is there a limitation for the amount of archives to have running, or the times you do the Twitter API authorization configuration, or just for the amount of data pulling in?
I hope someone can help me with this somehow. Many many thanks
Permalink
Problem Solved: When I did the Twitter API authentication via Tools > script editor > run > authenticate twitter it worked. I guess I did something different before..
Permalink
Thanks! You saved me a good amount of head scratching!
This tool is wonderful! I’ve popped it on my blog.
Permalink
Any suggestions for getting historical data in the same format? I need very little and basic data starting from November 2011 with the same search term as I used with TAGS. Anybody already done that? I would love to hear any suggestion. Thanks a lot.
Permalink
Hi, I’ve just made a copy of version 5.0, but when I click on the ‘enable custom menu’ button a window opens and says, “Oops, Script function onOpen could not be found”
Is this a problem with the software or is it my google drive causing it to do that?
Permalink
Hi Andy,
Google sometimes needs a couple of seconds to catch up with the template copy to attach the script. The advice I’ve seen elsewhere is to keep hitting the damn button until t works ;)
Permalink
Permalink
Thank you very much for this! I hope this is not a silly question but only the first search term I enter shows up in the archive. Is it possible to enter more than one search term or do I have to do separate sheets for them? Also, my understanding is that Twitter allows for searching further back than 7 days. Is this a possibility here? Thank you very much again!
Permalink
Hi Martin,
Is there an option for language filter in the archive? I’ve notice the French are using the same hashtag as us. If I could filter out the french language tweets…
Permalink
Hi Su – requires some code tweaking. In the Tools > Script Editor change the lines around number 87 from
if (sinceid){
var data = getTweets(SEARCH_TERM, {"sinceid": sinceid}); // get results from twitter sinceid
} else {
var data = getTweets(SEARCH_TERM); // get results from twitter
}
to
if (sinceid){
var data = getTweets(SEARCH_TERM, {"sinceid": sinceid, "lang": "en"}); // get results from twitter sinceid
} else {
var data = getTweets(SEARCH_TERM, {"lang": "en"}); // get results from twitter
}
Permalink
Hi Martin, that’s great – thanks for letting me have the code. I’ve made the changes and can’t see any more French tweets. I’ve deleted the ones from the weekend.
I’m liveblogging the Government’s Construction Summit tomorrow – will be also promoting TagsExplorer and TagsArchive to help non-tweeters see what’s going on.
Permalink
Permalink
Permalink
Hi, very awesome script and interface! I’ve ran into a problem. I set up a couple of scrapes for my thesis research. All working fine, but one isn’t working and states dates from 30/12/1899 till 30/12/1899. What could this be?
Permalink
Hello Martin, when i run the time-driven triggers to collect tweets .. is there a way to filter the new ones from the old ones ? Basically .. i want to be able to capture all tweets on a file between 1pm – 3pm and another separate file from 3pm – 5pm and a separate file from 5pm – 10pm & so on and so forth . .. is this possible to do ? Or does TAGS just update the tweets to the spreadsheet ?
ALSO, is there a limit to how many tweets it looks for ?
Lastly, on the spreadsheet .. the times it says the tweets have been posted are incorrect as to when i open my tweet deck. How do i change the time zone so i see the correct time same as i see on tweet deck ??
The time is off by like 7 hours or something like that ..
Thanks,
Bill
Permalink
Hi Bill,
Google spreadsheets has some nice filter/query options. If you read http://mashe.hawksey.info/2013/02/lak13-recipes-in-capturing-and-analyzing-data-twitter/ there is a section on this topic with some examples.
Regarding the timezone twitter returns the data as GMT. To date I haven’t included the option to manipulate timezone differences but I’m beginning to ponder the next version of TAGS so it may feature
Martin
Permalink
This is very helpful!
One question; how can I change the time zone so the times are accurate for EDT (the time for New York City)?
Perhaps an ideal would be if there were a way to have the date in one column (in US format, which would be 7/12/13 for July 12, 2013) and the time (in EDT) in another column? Would this be something easy to adjust?
Thanks for this wonderful script!
Permalink
As a couple of people have asked. One way to get different timezone date display is in the Spreadsheet go to File > Spreadsheet settings and select date locale and timezone and then in Tools > Script editor open File > Project properties and change the timezone there as well. This should convert any existing and all new date/time data in the time column. The created_at column is unchanged as this is the metadata passed back from Twitter and required for tools that visualise a TAGS archive
Permalink
Permalink
Permalink
Permalink
Hi Martin,
Thanks for the amazing tool. I have one question though. How can I collect tweets based on geolocation? I want to collect some data from Bucharest and Brasov, Romania, but not by using hashtags. Is it something that I need to modify?
Thanks,
Cristina
Permalink
check the comments thread this has been asked and answered
Permalink
Permalink
Permalink
Brilliant tool – thank you, Martin. I had been using “The Archivist” to archive #globalclassroom twitter chats but, now that it’s subscription only, your script is the perfect replacement!
Permalink
Fantastic tool, thanks Martin! I’m trying to figure out a way to strip line breaks out of the tweets when they are imported to the Google Docs spreadsheet.
I’m exporting this data via a web query to Excel and line breaks in Tweets cause the text to broken in to separate cells.
Permalink
Do you have an example web query I can try?
Permalink
Thanks for your guidance Martin. I have managed to strip line breaks from the Tweet text by adding some code to the TAGS.gs script. Around line 187 (after //end of while loop), at the end of the gettweets function I added the following code:
// remove line breaks
for (var i = 0; i < data.length; i++) {
data[i]["text"] = data[i]["text"].replace(/\n/g, " ");
}
Perhaps this might be a useful option to include in future releases?
Permalink
Thanks for this. Given the workflow Richard has highlighted this makes sense as an option.
Permalink
Hi Martin,
This question has been asked before by Bill but I dont think found the answer to it. Is there a way just to scrape the tweets in a particular time frame. That is for instance from last Sunday 10pm to 11pm EST US time or Monday 2am to 3am UTC? When I started my search last night at 11.20pm I only was able to get the last 5 minutes that is from 10.55pm and a lot of “clutter” that followed after. Now I wonder if there is still a way for me to get the tweets from yesterday 10pm to 11pm? Your help is greatly appreciated.
Thanks.
Micha
Permalink
Hi Micha – unfortunately not down to the hour the best the Twitter API can give is a day range (from last 7 days). On the readme/settings sheet there is an option to get yesterday, -2 days, -3 days etc. To use this you need to make sure there aren’t newer tweets in the archive than the date range you are trying to get. e.g. if you change the setting to yesterday and you’ve already got tweets from today in the archive when you hit TAGS > Run Now! it won’t insert the older tweets.
Permalink
I have followed the directions to the T and still can’t seem to bring any tweets back. On the Dashboard it says “Cannot read property ‘title’ of undefined”. I’m not sure what that means. I would really love to use.
Let me know if I’m missing something.
Thanks!
Permalink
Permalink
I have read your Twitter Archive post on moz.com, I admire your post and happy to read and implement in my twitter strategy. I hope you do more like this.I know that you put the effort on this. We are happy to use this sheet :)
Permalink
Thanks for making this tool publicly available, it’s much appreciated.
I am however having trouble getting everything to work correctly. I’ve followed the setup instructions yet each query I try to run gives me a “exceeded maximum execution time” error.
I’ve authenticated twitter from the Script Editor and plugged in my Twitter Oath Key & Secret.
Am I missing something? Any help you can provide would be wonderful. Thanks!
Permalink
Scratch that, seems to be working now. Thanks again!
Permalink
Permalink
Permalink
Thanks for this great tool, Martin. I haven’t figured out how to get anything in the “geo_coordinates” column. Is there a trick here? I’ve turned on locations for my tweets, but that doesn’t seem to populate that column. Any advice?
Permalink
Thanks for this awesome tool, Martin! Do you know a possibility to get older tweets, like from march? Operators ‘since’and ‘until’ don’t work, and I know there’s the time period of 7 days but is it possible to search further back? Thank you so much!
Permalink
limitiation of Twitter API is last 7 days
Permalink
Permalink
Permalink
Permalink
Here is another comment in case you want to use TagsExplorer with your own spreadsheet. I tried to instructions but I was unable to see the visualization (spreadsheet was not loaded) until I published it to the web (even if it was shared). I think this step should be added to the list of steps. Thanks, this is an awesome tool.
Permalink
Hi, Martin
Thanks for this great tool. I use SNA in my PhD project and, when it comes to extract data from Twitter, NodeXL seemed to do the trick… until the moment I found out about TAGS magic. So, I took it for a test drive, but the SNA script is not working for me. Every time I try to run some functions of TAGSAdv.ps (from the script editor) I get an error:
1- GetCommunityGraph: “TypeError: Cannot read property “length” from undefined. (line 43, file “TAGSAdv”)”
2- replaceIDsWithNames: “The coordinates or dimensions of the range are invalid. (line 98, file “TAGSAdv”)”
3- GetCommunityGraphFromSavedData: The script runs but nothing happens!
4- displayIDSinCell: “Invalid JSON string. (line 150, file “TAGSAdv”)”
Everything else seems to be working fine (and I could export the relevant data to UCINET, for example), but without the SNA dimension TAGS loses its allure to me…
Keep up the good work and thanks for your help!
Permalink
Martin,
We’re having an odd problem on one of our TAGS sheets. We haven’t received any updated since Tuesday night, including a few direct replies which are usually very reliably caught.
I have tried running again manually, reauthenticating, etc. but no error is showing and no additional tweets are being collected.
Have there been any issues with the API/changes made since the conversation tweak Twitter made this week? Any/all advice very much appreciated ;)
Thanks, Nicola.
Permalink
Hi Nicola – when you look in dev.twitter.com is your app still okay? Things I would try: Delete last tweet row in archive and try again; another thing you could try is open Tools > Script editor, Run > collectTweets and then View > Logs
Permalink
Thanks Martin.
So, no indication of issues in dev.twitter.com. App still seems fine.
Deleting the last line and rerunning resulted in that line being reinstated. But none of the more recent tweets being added. The missing tweets are both from our account and replying to it and would normally be caught by the TAGS sheet (see anything post 27th Aug here: https://twitter.com/search?q=digimap4schools&src=typd).
Running collectTweets from the Script editor appears to be fine and the Log is blank.
Sorry, this seems to be one stubborn issue :-O
– Nicola.
Permalink
hmm the Search API isn’t 100% reliable so it might be because of this ‘Assessing the Bias in Communication Networks Sampled from Twitter’ http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2185134
Permalink
Ah, thanks for this. I shall have a full read. So I take it that TAGS uses the Search rather than the Streaming API? That’s useful to know anyway ;)
In that case I think we’ll probably handle the omitted tweets as we have a couple of times before – by manually adding in the key date/tweet ID/content/user info etc.
Huge thanks for all your help! :)
– Nicola.
Permalink
Hi again, Martin
Sorry for insisting, but the problems persist (errors while running some functions) and I cannot render any SNA related data. I have a complete verteces sheet, but blank edges and no metrics. What am I doing wrong here?
Thanks!
Permalink
Permalink
Permalink
Permalink
Permalink
I absolutely love TAGS and have been using it over the past 18 months to scrape many of the hashtags for a local public television station. Our budget is limited and your software has become an amazing asset to keeping up with the times.
We are running into a recent problem every since the latest update of TAGS back in the spring. We are now experiencing duplicates in our archive. The archive can have tweets from say 9-17-2013 to 9-10-2013 and then it will start back at 9-17-2013 again. What would be the cause of this hiccup within the archive?
Thanks,
Reggi
Permalink
Hi – I’ve also noticed recently problems with duplicates with the new API. Technically this shouldn’t happen, but I wonder if as part of the new API they’ve gone for speed over accuracy. In newer versions of TAGS there is a delete duplicate option from the TAGS menu to help clean up the archive
Permalink
Hi Martin,
just wanted to chime in, because this option doesn’t work in my archive.
The error message I get (just deleted the whole archive and created a new one with the same result) is: TypeError: Cannot find function trim in object 380014964331007900
Do you have an idea how to fix that?
Thanks!
Permalink
Permalink
This is brilliant! Thank you! Any chance you will be doing one for Face Book?
Thanks again!
Permalink
Permalink
Permalink
Hi Martin,
I’ve ran into a strange thing in the tool. I want to collect all tweets to and from ‘@Connexxion’. To:Connexxion is working perfectly, but from:Connexxion gives me “Twitter said: null”. Actually, every day there are several tweets from this account so this shouldn’t be a problem.
Thanks in advance!
Permalink
I have the same problem. I have created 5 spreadsheets now, all with this same result, creating them as I always have.
Permalink
I’m having the same problem. I want to collects tweets from @IAM_SHAKESPEARE and keep getting “null” to any from:IAM_SHAKESPEARE requests
How can I fix this?
Permalink
Permalink
I seem to get this with every spreadsheet copy I make now…
Cannot read property ‘title’ of undefined
– Twitter app is set up
– App consumer and secret are set up properly
– Ran twitter authenticate (And I dot the popup and agreed)
Permalink
What’s your search query?
Permalink
all it is is “#ORUHangout”
Permalink
Also, thank you so much for your effort in this. I REALLY appreciate you taking the time to create and maintain this as well as engage with the users. You rock.
Permalink
Hi Martin,
Thank you so much for your great app. It helps me a lot in my research. :-)
However, could you please tell me how can I change the time (actually i am in France)?
Thank you again,
Permalink
Permalink
Hi Martin, this looks a great tool, but I have filed to get it working. I get an error in the summary tab, in the right box, highlighted in red “sc(..)is undefined”
I have added the authentication codes form Twitter and run the authenticate script, but was not sure if I needed to use the access tokens then offered by twitter. Do I need to use them in some way?
Thanks
asher
Permalink
Hi,
I have been trying unsuccessfully to get TAGS 5.0 to work on and off for about 48 hours now, even going so far as to make new copies, refresh existing sheets, repeat authentication, different twitter authentication keys/accounts and also use basic search terms (e.g. ‘food’) and small quota requests.
But each time, even if I just try to do a ‘Test Collection’, the TAGS sheet hangs at the notification box ‘Running script. Dismiss’.
My TAGS 5.0: https://docs.google.com/spreadsheet/ccc?key=0Al135Uzr-xDZdHdLbHRsWXVDTFhEZjN5ejZCUDd2UUE&usp=sharing
Can anyone let me know if their TAGS 5.0 has worked in the last 7 days? Just so I know where I’m going wrong.
Thanks,
Sam
Permalink
Hi Sam, sounds like the same problem I had. I too tried multiple installs/copies with fresh authenticaitons
Martin, sorry if I /we are making an obvious mistake.
asher
Permalink
Hi Martin,
Great Sheet! Excellent work!
I’m trying to look for search terms that have a single quote-mark in them. e.g. here’s the set of terms I want to search and archive
(#DevilsDue OR “Devil’s Due”)
The scripts never complete running with this query.
When I search for #DevilsDue I get resuts.
When I search for Devil’s Due I get no results, so I assume the ‘ is the culprit.
Any ideas what may be gong on here?
Thanks again – great work!
Nogoyo
Permalink
There is an open ticket in Google Apps Script for UrlFetchApp not preserve RFC 3986 escaping which is used in the Twitter API 1.1. I’ve added a note to the search term box in the template to note this
Permalink
Thanks Martin. Great work.
Permalink
Leaving the script untended overnight I get “Exceeded maximum execution time” error, but no tweets.
asher
Permalink
Hi,
I created a video tutorial for my students for setting it up, including some steps that are missing from other instructions, but I’ve found are necessary for avoiding the kind of errors you are getting. You can see it here:
http://www.youtube.com/watch?feature=player_embedded&v=ssnrJTNrX0k
Ignore all the course-specific stuff, of course.
Good luck!
Bill
Permalink
Hi,
Asher, Bill – thank-you so much for pursuing this. By your joint help, I managed to get the script working (by watching Bill’s very useful video, I found I had forgotten to add the Callback URL).
Everything is working perfectly now, and like Asher, I want to thank-you Martin for creating this script, and the supportive community around it – really excellent work. And thanks Bill for your great additional resource.
Best,
Sam
Permalink
@Bill, thanks, your tutorial really helped. I now have a working copy, I had missed an element of the authentication. Your video is a great support, much appreciated.
asher
Permalink
@ Martin.
Martin, thank you for the work you have put into this fabulous tool. I have been trying to find an effective way for gathering tweets for quite some while. Your tool is exactly what I wanted. I also love the community of support you have around it. Nice one!
asher
Permalink
Permalink
Excellent! @Martin – Thank you very much for the tool. This is very helpful for my research.
(PS. One observation – There are numerous duplicate tweets in the Archive sheet. I hope this will get sorted out soon.)
Permalink
Permalink
Permalink
I’ve used these TAGS scripts since V3, and jolly good they are too. Here’s a puzzle though, I set up one to collect a hashtag, #cilip2014. I know there’s a seven day limit imposed by Twitter, but for some reason it doesn’t collect anything earlier than one time stamped Thu Oct 24 18:37:17 +0000 2013….this was actually half way through the event whose tweets I was trying to archive, so I know there are earlier ones that day. I’ve tried tweaking the collect date from default to -7 days, and running again, in case that would force it to find the earlier tweets, but still it fails
Permalink
Hi Tom,
Turns out there was a bug that been lurking on my todo for a while. Should be fixed now in latest version (v5.1). As a memento here’s the #cilip2014 archive I used for creating version 5.1. Instructions for updating archives or getting v5.1 of the template contained in the post.
Thanks,
Martin
Permalink
Martin – thanks for TAGS – it is brilliant, but I have hit a snag. It has been collecting #rwpchat since December (and doing the job beautifully), but this week has not collected new tweets since Monday (and there are a couple of hundred new tweets). I keep getting the message “Exceeded maximum execution time”. Any suggestions about how I can get around this? Thanks in advance.
Ellen
Permalink
Hi Ellen – I recently had similar problems and made a minor adjustment to the code. You should try opening Tools > Script editor.. and replace the code on the first page with the code here https://gist.github.com/mhawksey/7230553 (New copies of the template include this modification)
Permalink
Thank you very much – it worked!
Permalink
Thank you for this fabulous tool. My only problem is re-using one spreadsheet for more than one keyword. The first time I set up a sheet with a new keyword, everything is good and the archive etc populates perfectly. However I am very excited about its potential and wish to run lots of different queries!
If I then try to do a new keyword by overwriting the text in the readme/settings page, and again press Run, it says the query is running but the data in archive and the other sheets does not update.
Do I need to create a new spreadsheet for every new query, going through the setting up tags dropdown and authenticate process from scratch? Or is there some way of completely refreshing the data?
Permalink
Hi yes currently you need to setup a new spreadsheet for each query. When the script runs it looks for the last tweet in the archive and uses this to only get newer results so if you change the query it will only collect tweets newer that the tweet from the old query (if you see what I mean). It’s not ideal but I’m working on an improved solution
Permalink
Permalink
Permalink
Permalink
Permalink
Stunning good app Martin – many thanks for creating it.
It is all working perfect apart from one thing – when I publsi h the Dashboard to the web I get a Cannot read property ‘title’ of undefined error on the Top Tweeters’ graph (although I can see it on the GDrive instance).
Many thanks in advance.
John
Permalink
Hi John – Found similar myself. If you go into the chart edit mode and then come of it it seems to fix it (guessing bug)
Permalink
Perfect – many thanks
Permalink
Hi! I’ve been trying to set up, but Twitter authentification fails (Url Invalid) using the Callback URL provided. Any solutions about this?
Thanks!
Permalink
The same thing has happened me. there are no spaces in the field
Permalink
Permalink
Hi Martin,
Awesome tool and very helpful in my research. Have a query if you could resolve…
Google spreadsheet reaches its size limit frequently and need to be manually emptied out and download every few hrs. Any better way for long-term tasks?
Permalink
I have used TAGS v5 and am very happy with it. Keep us updated should there be new developpments
Permalink
Permalink
Permalink
Permalink
Hi Martin,
I’m interested to map tweets so I’m wondering if it’s possible to collect only the geocoded tweets.
thank you.
Permalink
See http://mashe.hawksey.info/2013/02/twitter-archive-tagsv5/comment-page-1/#comment-102507
Permalink
I did it before but it shows all the tweets within the radius, the geotagged ones and the non geotagged ones but having a location in the profile within the radius (the ones that the geo_coordinates cell appears empty).
e.g. I’m looking for tweets from Barcelona area (41.40,2.15,100km). There are Indonesian Barcelona supporters using Barcelona (or even Camp Nou) as a location. Even they’re tweeting from Indonesia their tweets appear in my search.
So I’m only interested in the 1st ones, I don’t know if it’s possible though.
Permalink
Permalink
Permalink
Permalink
Permalink
Permalink
Can some one please help to access Tweets from R software
Permalink
Permalink
I am seeing the following error;
collectTweets TypeError: Cannot call method “getSheetByName” of null. (line 62, file “TAGS”)
How can I remedy this?
Permalink
Permalink
Permalink
Permalink
Permalink
Permalink
Permalink
Permalink
Permalink
Permalink
Error: “Twitter said: null”
Thanks for writing this script! I’m getting the above message when doing a test collection. If I run the script with a search term, I simply get a sheet with no results. Any ideas?
I’m hoping to use the script to compile a list of twitter users (say, bloggers with an interest in pottery). I’m hoping that using the search term “blogger AND pottery” would return the list I see in Twitter’s search.
Permalink
Had the same problem, here’s what worked for me:
1. Created the spreadsheet again (clicked on the second link given above, then File> “Save Copy”
2. Followed instructions 1 and 2
3. Clicked on Tools > Script Editor > Run > Authenticate Twitter. A Window opened “You are about to authenticate your first App” – clicked on button (this last third step I had done before, but the window hadn’t opened – may I hadn’t been logged in to Twitter)
Permalink
Permalink
Permalink
There’s one thing missing from all instructions.
I set up an instance of TAGS 5.1, followed all instructions, set up a search for a hashtag and did a one-time run, and viola! I got a bunch of tweets. Fantastic!
The next day, I wanted to run it again and… wasn’t sure exactly what I needed to do. I just went to the TAGS menu and clicked the option to RUN. The script ran, but I got no tweets. Then I tried doing various steps over again, but never got anything else to run. I got no error messages, but just never got anything in the archive.
So which, if any, of the steps in the setup are required to do again before another run?
If there are no steps that need to be duplicated, I suspect that maybe my problem is related to the fact that I cut all the data out of the archive tab and moved it somewhere else. I assumed that TAGS could run again and just repopulate the blank archive tab, but one of Martin’s posts suggested that it looks for the last tweet to figure out where to restart. I wonder if there is a way to remedy that? Like maybe, copy all the data out, but leave the last tweet?
-Michael
Permalink
Permalink
Permalink
Permalink
Permalink
Hi Martin,
Did you remove the ability to archive favorited sheets in 5.1? I’m not seeing it, unless I’m missing something.
I’ve been running 5.0 since you added that functionality. I don’t want to lose it, as it is crucial for me.
Hoping I didn’t overlook something in the comments addressing this issue.
I’ve also noticed in 5.0 (and IFTTT when I used to use it) that if I retweet a tweet, and then favorite an earlier tweet, 5.0 doesn’t pull the favorited tweet. It’s like the retweet creates a time barrier. Maybe just a quirk of Twitter’s API? Anyway, I thought I’d raise it.
Love the sheet. Hoping I can upgrade. And would like to spread the word to a class I’m teaching, but favorites are key to that.
Thanks!
Permalink
Hi Ben – I think it was just an unofficial fork. You should be able to still make a copy from here https://docs.google.com/spreadsheet/ccc?key=0AqGkLMU9sHmLdENLWUc1OW9iTnd3b251UUhOQk8wbHc&usp=drive_web#gid=36 (next version of TAGS should have this baked in ;)
Permalink
Thanks! Baking it into 6 would be great. I’m trying to get some Columbia University students using it in a media innovation class I’m helping lead, based around the favoriting functionality. And, losing that would impact how I’m tracking projects online.
Permalink
Permalink
Permalink
Hi, I am new to this awesome API and am doing a cool project here at the university. Question. is there a limit to the number of handles I can list in a query? I have a list of about 100 twitter individuals that I would love to follow and download and study. Thank you for your help and advice.
Permalink
yes there is a limit to the query length (I can’t remember exactly but the answer is in this thread somewhere)
Permalink
Permalink
Permalink
Permalink
Hi Martin,
I responded to another partially-relevant comment elsewhere, but thought it might be too buried for you to see.
First off, thanks so much for this tool!
I’m trying to collect only those tweets that have geocodes. I have found elsewhere how to collect only those tweets associated with locations within a predetermined distance from a coordinate point (params.geocode = “-x,y,zmi”). But I need to specifically filter for tweets with longitudes and latitudes. How would I go about filtering specifically for them?
Thanks again for your hard work and for any advice you can provide.
Permalink
Doesn’t appear to be anything in the API to prevent place association. Is the data good enough for post processing?
Permalink
Yes it is and I have already written in a place association. What I am trying to do now is filter for only those tweets with geo.coordinates[0] and geo.coordinates[1] data. Do you have any ideas on how to do this?
Permalink
Permalink
Permalink
Permalink
Hi Martin,
thank you for this great tool!
Is there any possibility to check the connections between the archived users?
I would like to know how many people from the list (sheet: summary) a user is following.
I didn’t find a solution during my web-research, but I think it might be possible through this api: https://dev.twitter.com/docs/api/1.1/get/friendships/show
Do you have any idea how to solve it? (sorry, i’m no coder – so i’m not aware if it’s a big deal.)
Permalink
Hi Amsel – this used to be a feature of TAGS but has wilted because of restrictions on the Twitter API (it’s still a menu option but not sure if it still works). For non-coders my advice would be to look at NodeXL http://mashe.hawksey.info/2011/09/twitter-network-analysis-and-visualisation-ii-nodexl/
Permalink
Permalink
Thank you for sharing this, I’m VERY grateful for your work! Right now, I’m getting an error on the sheets I’m using saying “TypeError: Cannot call method “getSheetByName” of null. (line 62, file “TAGS”)” I’m not sure what that means and how to fix it. I’m not very well versed in this, but have found this to be INCREDIBLY helpful when it was working, now I’m having to do it by hand, because of these errors. Fixing this would save me a LOT of time. Thank you in advance.
Permalink
Permalink
Permalink
Permalink
Permalink
How i can make this in mysql and php? because google drive work very slow for me when i try to search with this script, can anyone help me?
Regards.
Permalink
You might want to check out other project like this one https://github.com/ginatrapani/ThinkUp
Permalink
I test it but its oriented to self search, not for tracking hashtag or external data, like your script :(
Permalink
Martin, do you know another system like this? like yours but mysql php?
Permalink
Ricardo, this is a very good opensource mysql, PhP tool: https://github.com/digitalmethodsinitiative/dmi-tcat/wiki
Permalink
Ill gonna tested, let u know! thanks m8 :)
Permalink
This is also great for PhP mySQL capture: https://github.com/digitalmethodsinitiative/dmi-tcat/wiki
Permalink
Already test dmi-tcat and need more details than that, any other idea? i see u copy link twice is that correct?
Permalink
The only other tools I know are is Discovertext.com (though you have to pay after 30 days), and Zappier – you can use Google Docs with this, but again, you have to pay after a certain quota: https://zapier.com/zapbook
Permalink
I love this sheet and you for creating it. SO helpful.
I’ve had great luck with it so far, but for some reason today it’s only returning 100 Tweets when I search for a specific term (a hashtag that I know has more than 100 tweets). Do you know why it’s limiting the archive tweets to only 100? I left it at the default 1500 results on the settings page.
Thank you for your help!
Permalink
Is this version 5.1?
Permalink
Yup – 5.1. Thanks!
Permalink
fantastic sheet, thanks so much for creating it. I managed to fill it up rather quickly (gluttonous clearly) – google have announced larger spreadsheets are now available but I wasn’t sure how to transfer the data into a new sheet?
again, very grateful for you work!
Permalink
Still getting only 100 results. Anyone else have this problem?
Permalink
Also only getting 100 results, even on lists of accounts that have previously generated 1000+. No clues as to why, unfortunately.
Permalink
Yup! same problem here. Did you find a way to fix your problem? We made two archives work in february, but this week, using v51, we’re only able to collect 100 tweets in retrospect – and after that, it’s only adding newer tweets in hourly updates. Tried paged searches for e.g. -7d, -6d, and older data are positively retrievable. But still, no initial or continuous inclusion of those. Strange.
Permalink
Sadly, I never heard back and had to go with a paid version.
Permalink
Permalink
Permalink
Permalink
Permalink
Permalink
Hi Martin,
thank you very much for sharing your work, works fine for me so far. I was wondering if it is possible to set the starting point for the data collection even below -1 day. That would be a great help for me dealing with discussions, which generate thousands of tweets a day. The perfect scenario would be that the run now command just starts gathering data from the moment of its first execution. Is that somehow possible?
Thanks
Cornelius
Permalink
Permalink
I have archived six different documents on the same hash tag, is it possible to merge them without losing the metrics such as number of unique users etc
Permalink
Dear Martin,
Is there any way to retrieve the number of times a tweet has been favorited. I would like to retrieve tweets, and 1/2 days later, using just its id, retrieve the number of times (and by who/date-hour) it has been favorited during these 2 days.
Thanks again for your great tool and help!
Frederic
Permalink
Hi Martin,
Thanks for setting this up!
I am currently having a few technical difficulties. First, I am getting a error message when clicking on the web views link. I did file>publish to web and I am still getting the error message.
I am also having a problem using the mozcontemplate – http://builtvisible.com/wp-content/uploads/2013/07/template.xlsx (SEO Tools is installed)
Once I open the template and enable editing I get 5 different error messages.
Error Messages received:
1.Unshort URL error
2: Twitter count error
3.Google Plus Utils error
4. Xpath On URL error
Once I clear the errors messages, I import data from web (using the link in the tags google doc-file>publish to web section). The data is populated in the twitter archive sheet. The data in the charts sheet do not refresh and the analysis sheet once information is dragged down, everything to the right of the extracted URL does not populate.
I am at a lost and looking forward to your advise.
Thanks,
Karla
Permalink
Thank you so much, this is incredible.
Permalink
Permalink
Many thanks for this Martin, It is really simple to follow and I am hoping to use it for a conference this week. However I am having a small issue with the fact it will collect the #enconf14 but will not pick up @enconf if I put them in the box as
#enconf14 AND from:@enconf
I realise you are not a Q&A service! But could you let me know if I am doing something obviously wrong? I have googled but can’t seem to find anything.
Many thanks
Kathryn
Permalink
Kathryn, try the search without the @ symbol.
Permalink
Hi Martin, I set everything up smoothly. However when entering my selected tags into the search bar and moving onto the archives page, nothing is popping up. It does say ‘running script’ at the top however. Does it take a while for data to pop up if a commonly used tag? Or has something gone wrong here? Thanks!
Permalink
This happens to me from time to time. Sometimes, authorisation does not always work with the Twitter API, and you have to try a few times to make sure you see both pop-up boxes asking you to authenticate in the main TAGS>Twitter Api authetication, AND the Tools>Script Editor menus. What I advise is to:
1) Close and reopen the sheet (to kill any scripts that might have crashed),
2) Once the sheet is reopended, go back to Tools> Script Editor, and Run>authenticate Twitter again (make sure the white box connecting to the Twitter API opens and asks you to allow the script app to use your Twitter account.
3) Then go back to the sheet and run the collection process from the TAGS>Run now menu, and it should work after a few moments.
Permalink
Permalink
Hi Martin, thanks for sharing your FANTASTIC tool! It’s a great help for my PhD as I am working on very similar graphs, but they are not interactive yet (http://www.md2c.nl/twitter-community-overview/). However, I could not get the getCommunityGraph() working: I got vertices, but no edges. Any idea?
Permalink
Hello, Martin. Thank you for your great work. I really like to use it.
But I cannot use the function of collecting tweets on specific time within 7days.
I set 1) Tags> Update archive every hour 2) Tool>Script editors> project’s triggers>specific date and time, but it was not operating.
Would you let me know the right way to use this function?
Many thanks.
Seungho
Permalink
Permalink
Permalink
Permalink
Permalink
Great tool! You might want to get also the https:// links counted by tweaking your formula a little: =ARRAYFORMULA(SUM(IF(ISERROR(FIND(“http://”,LOWER(‘Archive’!C:C))),0,1);IF(ISERROR(FIND(“https://”,LOWER(‘Archive’!C:C))),0,1)))
Permalink
good point, forgot about https. Given (i think) everything gets wrapped in t.co you could probably just find “://t.co/”
Permalink
Good idea! Are you considering connecting your spreadsheet to a fusion table? Might be nice as you can then display graphs on a website with “real-time” data, like i.e.: http://www.md2consultancy.com/news/new-project-display-your-twitter-network-on-your-website/
Permalink
Hi,
First, thanks for your great work. It has been very beneficial to my reasearch. I however noticed that the columns “in_reply_to_user_id_str”, “in_reply_to_screen_name” and “geo_coordinates” remain empty in my spreadsheet. Is there a reason or something I’m doing wrong? Thanks in advance.
Cheers, philipp
Permalink
Hi Philipp,
“in_reply_to_user_id_str” and “in_reply_to_screen_name” will only be populated if the tweet was a reply to someone.
Similarly with “geo_coordinates”, this will only be populated if the user adds their location to the tweet.
Daniel
Permalink
Hello,
I am new to this twitter scraping spreadsheet. When I test the collection or run the script, it throws the following error:
“Message details
TypeError: Cannot read property “length” from undefined.”
Any idea(s) on how to solve this?
Permalink
PS – I have completed the instructions for Twitter OAuth key and secret, but no joy.
Permalink
SOLVED – PEBKAC – incorrectly performed the following:
3. When your Key and Secret are saved you need to open Tools > Script Editor … and run the ‘authenticateTwitter’ function
Permalink
Having trouble with triggering:
getTweets>Time-driven>Hour timer>Every hour
Execution failure notifications
via email at [email protected]>immediately
No notifications have been emailed, and no updates have occurred without manually using “Run Now!”. Any ideas?
Permalink
Ok, so a failure notification email was eventually sent. I guess setting this to “immediately” is not working.
The email says:
Summary:
Error Message Count
Authorization is required to perform that action. 11
Followed by a list of 11 instances were getTweets tried to trigger but failed because of “authorization required to perform that action”
I do not know what authorization I need to give this, as I’ve got it working for “Run Now!”. Any ideas?
Permalink
hi – when you open tools > script editor and run authenticateTwitter does it popup an authorisation window?
Permalink
Well, it did open a popup window for authorisation in the past. Now it does not, though.
Instead I now get a script warning:
ScriptProperties API is deprecated.
File: TAGS Line: 27
The API has been marked as deprecated which means that the feature should be avoided and may be removed in the future. Consider using an alternative solution.
Line 27 being the following from authenticateTwitter:
ScriptProperties.setProperty(‘active’, SpreadsheetApp.getActiveSpreadsheet().getId());
Permalink
This Google Developers page lists the deprecated codes, but I am still looking for how to replace it.
https://developers.google.com/apps-script/reference/properties/script-properties
Permalink
Deprecated but still works for now. Next version of TAGS is close to completion and is a major code rewrite to use all the latest apps script services. Hopefully out next week but you can use my contact form http://mashe.hawksey.info/contact-form/ for early access (all I need is an email to share the new template with)
Permalink
Permalink
Permalink
Permalink
Hi, I’m trying to find a way to download an entire hashtag (#thoughtvectors) for research. Is the number of tweets I can archive using TAGS v5 limited to the last seven days? Thank you!
Permalink
yes – but someone else might be archiving it using tags (worth pinging @cogdog)
Permalink
Thanks a lot, I’m exploring few other tools, but will connect @cogdog if I can’t find a good tool
Permalink
Permalink
Having same issue as @LondonVotes. You say it still works tho? This is my first time using. I hit “Run Now!” and it still says “Running Script” after about 20 minutes. How long does it normally take?
Permalink
Permalink
Permalink
For those subscribing to TAGS v.5.0 new via this comments thread
Update: TAGS v6.0 is here! You can jump straight to the new support site. If you are using version 5.x here are instructions on how to upgrade (there is an old Sheets version of TAGS which is recommended for use with TAGSExplorer).
Permalink
Permalink
Permalink
Permalink
Permalink
Permalink
I love this script, but the loading of the datavisualization takes too long en mostly it times out. I tried with a fresh archive (500 tweets), but no change.
Permalink
Permalink
Permalink
Dear Martin,
Thank for setting up this. I am having trouble running the script. It doesn’t seem to work. I don’t get the same window as you, and the list on archive.:( Am I doing something wrong?
Permalink
This is the old version. Try http://tags.hawksey.info The site has a support forum if you are still stuck