Graphing with the twitter archive and R : or how I tweet too much

So last weekend, @encephalartos produced a graph of his tweets as extracted from his twitter archive, and thereby tempted me to spend the rest of my weekend and some extra time beyond figuring out how he did it. Turns out he used excel to break the timestamps and R to do the rest, but I didn’t realise he used excel till I had spent hours figuring out time in R, so I present to you the entirely R way of doing it.

I like to use Rstudio for doing my R work. It’s available for Linux, Windows and Mac so you’ve no excuse. Most of my R knowledge comes from workshops that Kevin O’Brien ran in Tog. If you’re based in Dublin and want to hang with some R folk, the Dublin R group meets (ir)regularly around town.

The Twitter archive is a great excuse to practise your R skills, as (depending on how much you tweet) it’s a nice large dataset with results that will be interesting but not critical to the running of the world. You can download your archive by going to the settings page on twitter.com and at the very end, there’s a button to click to request your archive. After a few minutes you get an email with a link to download the zipped archive that contains lots of delicious data. For these graphs, you only need the file tweets.csv which contains all your data as a handy flat file, where columns are separated by commas.

Getting the archive ready to graph

Importing your archive in Rstudio is really easy. Go to Tools>Import Dataset and follow the instructions (or you could look up a tutorial how to do it the proper R way, but why have an IDE if you don’t use it :) ). The default settings should cover the tweets.csv but double check that it looks right in the preview pane.

The timestamps column is what’s interesting to us today. It’s formatted in ISO 8601 format, but Rstudio will have imported it as a character rather than a date, so we have to do some quick conversions.

You will also need to install/load the relevant libraries for the date handling and graphing.

install.packages("ggplot2")
install.packages("scales")
install.packages("lubridate")
#You only need to install once, but you need to load with library() every session.

library(ggplot2)
library(scales)
library(lubridate)

Convert the timestamp to POSIXct format using as.POSIXct() and put it in a new column (I don’t like overwriting old columns).

tweets$posix_timestamp <- as.POSIXct(strptime(tweets$timestamp, '%Y-%m-%d %H:%M:%S'))

If you run data.class(posix_timestamp) it should return "POSIXct", confirming the data translation worked. Incidentally, when I looked at my new posix_timestamp column, I saw that it ended in GMT and IST depending on the time of year, it would seem IST is Irish Standard Time, which is equivalent to BST (British Summer Time). I think the conversion to IST might be due to my system settings being for Ireland.

Graphing tweets by week

Once you have the dates converted to POSIX format you're pretty much there, you just need to generate the graph!

ggplot(tweets, aes(x=posix_timestamp)) + geom_histogram(binwidth = 60*60*24*7, aes(fill = ..count..)) +scale_fill_gradient("Count", low = "skyblue", high = "blue") + xlab("Date") + ggtitle("Tríona's Twitter output by week")

The natural bins for POSIXct objects are 1 second, so to get week-long bars, you have to multiply them up. The start of the "week" is presumably the start day of the archive itself, I need to get around to figuring that out.

You can play around with the binwidth to get days or years as you fancy. Likewise you can change the colours and titles.

All my tweets since I joined twitter, graphed by week. Includes retweets and I have no idea when the start of the "weeks" are...

All my tweets since I joined twitter, graphed by week. Includes retweets and I have no idea when the start of the "weeks" are...

In Rstudio, you can export your graph by clicking on the little "export" button over where the graph appeared. I like to export and .png but you can make your own choices about your preferred image type.

Graphing hours vs. weeks

Now that the time is in a POSIX friendly format from earlier, it's easy to extract parts of the date using the Lubridate package we installed.

tweets$day_of_week <- wday(tweets$posix_timestamp, label = TRUE, abbr = FALSE)
tweets$hour_of_day <- hour(tweets$posix_timestamp)

We then take these new columns and convert them into a table to make them easier to graph as their frequencies will be listed. A table can't be directly graphed, so we convert it into a dataframe and we can work on from there.

daytime <- table(tweets$hour_of_day, tweets$day_of_week)
dfdaytime <- as.data.frame(daytime)

dfdaytime should be return data frame rather than table when you run data.class(dfdaytime).

R will have renamed the columns when it created the table, with hour_of_day becoming Var1 and day_of_week becoming Var2. Frequency will be in a third column called Freq

Now that we have the new dataframe made, we can plot the graph!

ggplot(dfdaytime, aes(x=Var2, y=Var1, fill=Freq)) + geom_tile() + scale_fill_gradient(low = "skyblue", high = "hotpink") + ggtitle("Heatmap of Tríona's tweets by day vs. hour") + xlab("Day of week") + ylab ("hour of day")

As before, fiddling with the colours and labels of axes and graph title are easy. Choosing a colour that makes the data clearest is the hard part...

Graph of my tweets by day vs. hour. You can see when I tweet most, and when I sleep most. As this archive has 5 years of tweets in it, the during-work-hours tweets may be from before I started the PhD proper...

Graph of my tweets by day vs. hour. You can see when I tweet most, and when I sleep most. As this archive has 5 years of tweets in it, the during-work-hours tweets may be from before I started the PhD proper...

Further rambling

ggplot2 is a pretty powerful package in R for making graphs, and thanks to this bit of twitterage, I'm that bit closer to mastering it. Part of its power comes in the piecemeal assembly of the graphs (you spotted the +'s between each chunk of graph code), so after declaring what you want in the graph and the type of graph, you can start adding on other bit to en-fancy-fy the graph further.

The graphs include retweets, so I need to figure out the easiest way to sieve them out (about 25% of my tweets are retweets). I also need to figure out how to make the bins align with the start of the week. I should probably also reproduce the heatmap for the last year, so my supervisors can see I don't spend my entire work day tweeting :D

Peanutbutter Brownies

A number of members of my collaborator’s group are leaving for pastures new, so I have made cake as a goodbye-you’re-really-gonna-miss-it-here gift.

raw brownies

The brownies prior to baking. The peanutbutter and chocolate chunks sink into the hot brownie batter during cooking, so this method allows even distribution of filling.

The recipe is my usual brownies recipe (also found in the cheesecakebrownies recipe) with the addition of blobs of peanutbutter instead of nuts. I did have nuts, but it was a bag of hazelnuts, and would have needed roasting and skinning, but peanut butter is delicious with chocolate, so this “laziness” worked out well too.

  • 225g butter
  • 375g caster sugar
  • 3 eggs (orignal recipe called for 4 medium, but we buy ex large normally)
  • 75g cocoa powder
  • 100g self raising flour
  • 100g bar of chocolate (or “chocolate”, as I often use scotbar)
  • Peanutbutter (about 3 tablespoons, have more than enough anyway, there should always be peanutbutter in a kitchen)

Grease and/or line a cake tin (I used my 17cm tin and a small dish, the small dish is for have a small set of home brownies when the big tin is brought to work). Pre-heat the oven to 180°C (170° for fan ovens).

Melt the butter and add the sugar. Beat in the eggs one by one. Sift the cocoa and flour together, and add to the mix in three parts.

Divide between cake tin(s). Break up the chocolate bar and plop pieces around on the brownie. Get a teaspoon and a knife and plop 0.5tsp sized dollops about the surface. The chocolate and the peanutbutter will sink during cooking anyway. Put into the oven for 40 mins.

Cooked brownies

See? Allllllll sunk into the brownies. The greaseproof paper means the sunk/melted chocolate chunks wont glue the cake to the tin (trust me, voice of experience, chocolate glue is hard to get off tins without heating).

Let the brownies cool in their tin, then turn them out and cut them up. Alternatively, don’t wait for them to cool and attack them with a spoon… just mind yourself, they’re pretty hot.

spoon on a brownie

Too impatient to wait for them to cool…

Why do we communicate science?

At the weekend, I had the opportunity to participate in a science communication master class as part of the Famelab competition. During it, the question of “why do we communicate science” was raised and the same sort of answers I had heard before were given: science is important to society, people need to be educated, science is entertaining and fun, people need to know what tax payers money goes on, PhD students need to tell people why they’re being paid by the tax payer without ever having contributed a penny in income tax. I had heard these before, but this weekend, I decided that the last two weren’t a good enough reason on their own.

First the PhD student and their non-tax paying ways. The PhD student might not be paying income tax or PRSI, but they do pay plenty of VAT in their day to day lives. While they might enjoy their research and are considered a student, they are still performing a job (even if it’s not defined as such). They provide teaching and demonstrating hours for undergraduate students, and by the very nature of carrying out research are doing a job for their supervisor and the greater research community. PhD students work long hours for low “pay” (it’s a stipend, not a wage, this definition matters to HR and Revenue), so it’s not as if they’re taking this tax-money and spending it on all the luxury while never seeing the inside of a lab. Is there some sort of underlying guilt for getting to study and research and be paid for it?

The point of “the tax payer needs to know what their money is being spent on” is a fair one, but I have some concerns around it being an absolute reason. Yes, I agree that information on government spending, including science, must be available to the citizens of that country. But, scientists seem disproportionately pressured to communicate with the citizens that the money being spent on science is justified. I am unaware of a similar drive for goverment accountants to justify themselves, or tax office workers, or healthcare workers. It may be that I mainly encounter scientists and keep up with science policy in particular, but it seems that it maybe some failing in the public perception of what science is that doesn’t exist with other occupations. While I have no idea how aspects of tax are handled from a government accountant’s point of view, I haven’t heard one come out and explain it for me, the lay-audience, either.

Spending public money on science can be viewed in two ways (and should probably be divided between both sides), either as a commercial venture or as a cultural undertaking. Yes, science can have commercial benefit, it advances technology on a daily basis and this can have a monetary return. Where government money is spent on such commercial science, the tax payer should see a return on their money (whether or not they do directly, much of the argument is that the return is in jobs or a reduction in cost of living). The cultural side of science is equally important. Government spending isn’t soley on commercial ventures, they fund social projects and provide healthcare, but they also fund cultural projects such as museums, libraries, public gardens and no one is about to ask the St. Stephen’s Green garden to explain why it’s there. The cultural impact of science ranges from simply contributing to our understanding of the world to imparting knowledge to the next generations of scientists (whether the knowledge is of immediate practical use or not).

I love communicating science to people, whether lay audience or people who are more knowledgeable than I am. I do it for fun, the education of others and because my work is interesting and worth shouting about. I feel the overemphasis on the taxpayer is misplaced somewhat, the media and goverment have a role in explaining science to the world as much as the researcher in the lab. Placing the burden of justifying science on the researchers alone is wrong. Science and not simply scientists have a huge role in our society.

Oh, and PhD students, don’t feel so guilty and keep on researching.

Rasberry Drizzle Cake

I always like to have a box of frozen raspberries in the freezer. They keep pretty well and sure, when they’re going into cake they’ll be mushed up a bit anyway.

For the rasberry drizzle cake, I gave the rasberries a quick stewing, to both defrost and collapse them, and to collect a tasty syrup for drizzling on top. To be honest, I hadn’t quite decided what sort of cake I would make before I began stewing them, but when I couldn’t find the recipe I wanted in my email archive, I went with adapting the lemon drizzle cake.

cake

Serving suggestion: cake, and some tasty coffee

  • A box of frozen rasberries (mine was about 3/4 full)
  • 2 tablespoons of water
  • 3 teaspoons of vanilla sugar (or regular sugar)
  • 125g butter (softened, leave it out for a few hours at room temperature)
  • 175g caster sugar
  • 2 large eggs (beaten in a mug)
  • 175g self-raising flour (sifted)

Put the rasberries, water and vanilla sugar in a pot. Put on a medium heat and stir it occassionally until all the rasberries are defrosted. Mush them a small bit to get some extra juice out, but not so much you end up with all mushed rasberries. Strain the rasberries through a sieve and set them aside, returning the syrupy juice to the pot. Heat the pot, swirling from time to time, until it has reduced about three times to a thicker syrup and set aside.

rasberries in all their forms

The reduced raspberries and their syrupy guts.

Preheat the oven to 180°C. Grease and line a cake tin (I used my 16cm square one). Cream together the butter and sugar. Add a little egg, mix well, then a tablespoon of flour and mix that in. Repeat until the egg and flour is gone. This gives a nice smooth cake batter. You could do all this in a food processor if you own one of course.

Mix the rasberries into the cake batter. I mixed it into all the batter, but if you divide the batter you can get a nice marbled cake. Pour into the cake tin, and bake for 25-30 mins (put a knife in, if it comes out clean, it’s done).

Turn out onto a wire rack, remove the baking paper and flip it over. Poke loads of holes in the top of it with a cocktail stick or skewer. Pour over your reserved syrup from earlier. It’ll probably look like a blood (jam) bath as the red syrup doesnt blend in as well as the lemon syrup. That’s ok though, you can always dust it with icing sugar just before you present it to the happy eaters.

If you happen to have a lemon to hand, I think the juice might be a good tangy addition to the rasberry syrup. Rasberries are tart enough, and the sugar added above isnt enough to neutralise this, but extra tart flavour is always welcome in a drizzle cake.

r drizzle cakes

Most of the rest of the tray, little do they know their fate…

Phospholipid bilayers made easy

Some scientists are artists, I am not one of these. Sadly, that doesn’t get me out of making diagrams to explain my work.

Some glycoproteins (proteins with sugar groups attached) sitting pretty in a phospholipid bilayer (that twin layer of pink circles with tails).

My work deals primarily in cell membranes. Glossing over the details, these can be condensed down to some proteins lodged in a phospholipid bilayer. The proteins are drawn when needed, but the bilayer occupies all the other space and its many repeating units need to be drawn to make sure the illustrated cell doesnt spill its contents into the extracellular space… (ok, that won’t actually happen, but the membrane still needs to be drawn).

Phospholipids are generally drawn the same way, circle for the phosphobit (it’s a short charged group) and a pair of long tails for the lipid part (very hydrophobic). So the phopho head points into the aqueous compartments (the cell interior or the external environment), and the lipid tails of each layer point toward each other (like oil and water, these lipid tails like to stay away from water too).

So how did I draw them I hear you say, by using the awesome free, opensource image editing software that is the GIMP (runs on any platform too, so give it a try).

First thing I did was draw a phospholipid unit with a transparent background and zealous crop so the edges of the image are tight with the drawing. Then I saved it to my images folder (as a .xcf for later use) and to my brushes folder as a .gbr (on linux it’ll be in the .gimp directory). Then when you go to the brush tool, hit the “brush” box in the tool box, then down the bottom right “open brush selection dialog” and hit refresh. Your new paint brush should be ready for use.

There are two options to consider then, the spacing between the brush images and the scale of the image. The scale is pretty easy to figure out, resize until you like the size of it. The space is measured in percent (be nice if they pointed that out more clearly), as you dont want your phosopholipid units overlapping, go for 110% or greater.

You can then draw with the phopho-brush. You can stamp it one piece at a time, or if you draw a curve, the image will be spread across it and spaced by your chosen percentage. You can also use it to stroke a curved path, which is what I ended up doing, very handy (just set the paintbrush settings before you select stroke path). I made two phospho-brush units, one pointing up and the other pointing down, then later I made brushes for each sugar unit I’d have to draw on to glycoproteins.

Don’t forget to erase some of phopholipids to make room for your proteins!

Rashery appley pancakes

The cooking apples havent been great this season. It seems most fruits haven’t been faring too well with the wet cool summer. I managed to find three small decent cooking apples a few weeks ago, and I turned them into stewed apple deliciousness. Add some rashers and pancakes and I had a particularly delicious breakfast.

Delicious breakfast, beware the venus fly trap monster coming to steal it away….


Continue reading

Beans on Toast or “Sorry I’ve been up the walls”

I have been quiet of late haven’t I. We just moved house (gone is the gas hob and fan oven, and it’s back to conventional oven and electric hob) and we’re still unpacking two weeks later. On top of which I caught that marvellous virus that’s doing the rounds. I also got a new camera not so long ago (you may have noticed the browniecheesecake picture was better than the ones I took with my phone camera). So to try out the new kitchen and my camera, I have documented my beans on toast method.

Our new back garden and windowsill

Our new back garden and kitchen window sill. The agave is still doing well, and the houseleek went to flower so the mother plant is about to die off to leave room for the chicks.

Continue reading

Cheesecake brownies (bonus three recipes in one)

I’ve been meaning to make cheesecake brownies for a few years. I’ve a great recipe for brownies from Chocolate Cookery (you’ll have to look this one up on abebooks) and an equally great recipe for white chocolate cheesecake. Others have combined these before with much success, so it was my turn to have a go off them.

Slices of cheesecake brownies waiting to be brought to college

Continue reading

Delicious cinnamon snails!

It took me a while to realise I liked cinnamon. I think I must have been in my late teens when it hit me that this stuff is AMAZING. A wonderful warm flavour that goes great with apples, or better yet, some sugar and butter and little else.

Action shot of the icing going on the cinnamon rolls

These delicious rolls of joy were made from a recipe by Smörgåsblog. You should have a good look around that site, there’s some very delicious treats to be found.

The cinnamon snail recipe can be found here.

Continue reading

Further adventures in biscuit land

I love a good shortbread so I do. It’s quick enough to make, and the buttery delight means it’s hard to make shortbread that doesn’t taste good. People always go on about shortbread being quick enough to make if guests suddenly arrive and you want to show off / not go to the shop for biscuits, but seriously, who does actually do that. I just make it on a whim, so the lads in college will be in for a treat tomorrow, lucky folks.

The recipe was derived from a barely remembered (misremembered?) ratio of 3 parts flour to 2 parts butter to 1 part sugar. It’s VERY buttery, so if you don’t like buttery shortbread you won’t like this (also, how can you not like buttery shortbread). It’s a simpler cousin of this recipe, it’s not as delicate and fragile, an altogether more robust shortbread than can survive the trip to work.

It is pretty quick to make, about ten to fifteen minutes making the dough and then fifteen to twenty while they bake in the oven. If you want to chocolate coat them then that will take extra time. This makes about twenty biscuits (4-5cm in diameter).

Delicious chocolate covered shortbread waiting for the chocolate to set, and some naked ones waiting to be nommed.

  • 240g plain flour
  • 160g butter
  • 60g caster sugar
  • 1 tsp almond essence

Preheat the oven to 170°C (mine’s a fan oven). In a bowl, mix together the butter and flour until it starts to resemble breadcrumbs. Then add in the sugar and essence. Mix about a bit, then start to knead it all together. The kneading takes a little work as it’s a pretty crumbly dough (the only thing holding it together is butter!). When it’s come together as a lump, you can roll it out. (I stuffed it into my biscuit gun and cut off neat circles, as it’s too stiff to pass through the shapes). (I made about 20 biscuits)

Put your (about 8mm) thick cookies on your parchment lined tray. Leave a little space around the biscuits, they’ll spread by about 20% due to the massive amount of butter in them (they melt basically). Put them in the oven for 15 to 20 mins until starting to brown very very lightly. Leave to cool on the tray for five minutes before transferring to the wire rack.

Optional:
You can coat the biscuits with chocoalte if that’s your thing. I coated 15 of the biscuits with about 200g of chocolate. While the oven is cooling, put the broken chocolate into a borosilicate (pyrex is a tradename dontchano) bowl and while the biscuits are cooling the chocolate can melt and you dont have to watch for it burning in the microwave. Dip each biscuit into the melted chocolate using a pair of forks, turn and then place on parchment to set (I’m using one of these nonstick sheets for the oven that you get in Aldi). And be patient. When they’re set, you can eat them!