Using the OS X clipboard in R

One annoyance with using a command line based statistical system, i.e. R, is that if you use a spreadsheet program to handle your data (as most of you do) you have to incessantly export to csv and import to R. For big data sheets and complex calculations this generally a Good Idea(tm). It improves reproducibility and readability when going back to the project at some later time. However, the complexity of that is enough that you often, at least I, end up doing statistics in Excel. This is generally a Bad Idea, and often entails much more work in the long run. In order to make importing and exporting data easier R can handle the clipboard directly, which is what this post is about.

In OS X the clipboard is easily accessed through the command line programs pbcopy and pbpaste (pb means "pasteboard" but everyone knows it as the clipboard so that's what I'm going to call it). These are easily scriptable programs that can be used with any unix command in the shell or scripts (enter "man pbpaste" in the terminal to learn more). "pbpaste" pastes the current clipboard to std-out (that is where you call it from), while "pbcopy" copies std-in (what you put there) to the clipboard. An important quirk is that they only handle plain text, RTF and EPS, that is formats that are transferred as plain text. This is because unix is a text based system. What it means is that you can't copy stuff from formatted text, e.g. word, but with spreadsheets you are OK.

As command line programs they can be handled through the R command: pipe(). Pipe does not produce the current clipboard contents, just a description of the pipe as such. You have to use a read command to get the actual contents.

> readLines(pipe("pbpaste"))

which will produce one string per line (completely useless as data, but often useful to see how a given copy is formatted). To format it for data you use the same as you would any import, using one of the higher level read-commands: read.table(), read.csv(), etc.

> x <- read.table(pipe("pbpaste"))

This will put your copied excel table directly into the table "x" in R. Often you will be forced to tweak the function a little to get a usable table.

> x <- read.table(pipe("pbpaste"), header=TRUE, sep="\t")

That should be able to read your copied Excel data ("header" specifies that the first row is the header, and "sep" specifies the separator which defaults to tab or "\t" in excel).

To write to the clipboard you use "pbcopy". It is a bit more involved, but not much. First you connect a variable to the pipe from pbcopy.

> osxclipboard <- pipe("pbcopy", "w")

Now you can write to the variable "osxclipboard" and it will turn up on the clipboard so that you can paste it into another application. You have to remember that it is a pipe to an external file, which means that you have to use "write()" to put data in the clipboard. If you just assign some data to "osxclipboard" it will be changed from a pipe to that data. Thus:

> write(x, file="osxclipboard")

where "x" is your data, will to the trick. Then you just have to clean up using:

> close(osxclipboard)

In the same way as with copying from the clipboard, this will not format your data optimally. so you have to add some formatting command for Excel to read it properly.

> write(x, file="osxclipboard", sep = "\t")

There we are, but that is rather alot of code. Not really that much easier than a regular export from excel and then import to R. So here are two simple functions to do it all at once, with the most used arguments (I'm sorry for the code, I haven't figured out how to get indenting to work properly in blogger).

Simple readlines from the clipboard:

clipboard.readlines <- function (

clipboardname = "pbpaste",

n = -1L) {

readLines(con = pipe(clipboardname),

n = n)}

Read table from the clipboard. I have only included the most commonly used arguments to read.table(). You could easily add all the rest if you need them:

clipboard.readtable <- function (

clipboardname = "pbpaste",

header = FALSE,

sep = "",

quote = "\"'",

dec = ".") {

read.table(file = pipe(clipboardname),

header = header,

sep = sep,

quote = quote,

dec = dec)}

Write to the clipboard:

clipboard.write <- function (x,

clipboardname = "pbcopy",

sep = " ") {

osxclipboard <- pipe(clipboardname, "w")

write(x = x,

file = clipboard,

sep = sep)

Close(osxclipboard)}

That was it for now. I hope someone finds it helpful. I'm probably not putting any of this into a formal extension, so you will just have to run the code to use it. Consider the code GPL-licensed, although the text is under regular copyright.

MD or PhD

Let us return to the discussion about combining research and clinical work. Specifically starting a new group in a physiology lab and working as a junior doctor at a university hospital.

I have now finished my first spat of full time research in the new lab after a year as a clinician. I have been able to do most of what I set out to do this semester, but I will probably have to work as much next spat of research to make it work then too. The reason for this is that none of the senior PIs are directly behind my project, therefore none of them thinks that I should encroach on their space. At the same time they are all very supportive and positive to me being there. Just in a more theoretical way. What will happen is that in a year or two (hopefully sooner), there will have to be a centrally controlled reallocation of the lab space at the department, until then I will probably have to continue on borrowed space.

Funding wise, it is going alright. With a couple of minor grants I have had enough production of preliminary data to be able to apply for some real grants. Career wise, the next step will be a docenture, which is kind of like the Swedish version of tenure. You need a number of original publications and you need to show that you are independent as a scientist. On top of that you have to have taught a minimum number of classes. In combination with clinical work it is a chore to get it all done as quickly as possible, but nothing much happens until it is done.

Clinically I have spent the first year in medicine, which is my home field, so that has been good. Now I am going to rotate out into psychiatry and surgery this autumn, and while that may be interesting and possibly good for something, it is also a bit of a waste. Especially as I am just picking up speed in the lab again after moving back to Sweden. I know, it was my choice to try to combine the clinic with research and rotation is a necessary part of a junior doctor's life, but I can still complain. There is surprisingly little time over when working clinically, but I hope to spend the autumn revising and submitting some new and exciting manuscripts, and writing even more grants.

But now for my last week of vacation.

Advice for a Young Investigator

Santiago Ramón y Cajal was a 19th century neurophysiologist. I'm told he is well known, though he seems to have ignored the kidneys completely. Strange.

Any way, he wrote a book - "Reglas y consejos sobre investigacion cientifica" translated as "Advice for a Young Investigator" - to help budding scientists through their first years as researchers. While some of his advice is dated, such as the section on how to choose a wife, most is well worth the $14. What follows is a brief summary and some comments on the 1999 MIT Press pocket edition (with one of the worst quality bindings in modern history), translated by Swanson and Swanson (who have done a good job).

There is a Foreword by Larry W. Swanson with a short biography of Dr. Cajal and some suggested readings from his quite large production. Then there are three prefaces, for each of the second, third and fourth editions.

There are nine chapters, each addressing a different potential difficulty.

1. Introduction

The introduction concerns itself with which theory of science a scientist should build his understanding of nature upon. The answer is determinism. Cajal is very clear that a scientist should not spend his time delving too deeply into the various theories of knowledge and science. He writes:

"...by abandoning the ethereal realm of philosophical principles and abstract methods we can descend to the solid ground of experimental science..."

In this he comes quite close to my own thoughts on the matter (which may be why I like the book in the first place). Today, pure determinism is still usefully utilized in Chaos Theory, or the study of non-linear dynamics, but in general the focus today should be on statistical determinism. That is, the fact that we cannot know the full state of any system (at least any interesting system) induces variation and that this punches a quite large hole in classical determinism, but can be compensated for by studying larger populations and using statistics. Actually Cajal does not mention statistics at all, probably because it was just starting out as a field at the time.

2. Beginner's Traps

In the second chapter Cajal goes through some things that might unduly discourage the beginning scientist from his chosen path. Basically that well known feeling that those who went before were smarter, saw farther and thought deeper than you ever could. He makes the case that it only seems so after the fact, and that with hard work and time a scientist starting out today will appear equally daunting to those starting out in the future.

3. Intellectual Qualities

Cajal's main point about the mind of the investigator is that it does not have to be exceedingly brilliant as long as it is tenacious and a bit vain. He does mention originality, but his focus really is on hard work and the striving for glory. These are qualities he dejectedly notes are none too present in his contemporary Spaniards, while he holds the Germans in high regard. He writes:

"In Spain, where laziness is a religion rather than a vice, there is little appreciation for how the ... work of German [scientists] is accomplished - espeially when it would appear that the time required ... might involve decades!"

Rather poignantly, the endnotes have been expanded with the editions. In relation to a section written in the original 1893 edition Cajal later notes:

"This frank optimism is now greatly undermined by the hideous international war that began in 1914... It is sad to admit, but all nations become ferociously imperialistic as soon as possible... So much for the weak and unpatriotic!"

4. What Newcomers to Biological Research Should Know

In this chapter Cajal handles the opposing needs for the investigator to know a lot about disparate fields, especially the basic sciences, and at the same time specialize as much as possible to be able to produce original work.

Perhaps his most important point is the short section on "Mastery of Technique", as he writes:

"...the most important scientific conquests have been won by only a dozen men who have become known for their invention of improvement of a research method..."

5. Diseases of the Will

This is clearly the most entertaining section. The six most dangerous personalities - "who never produce any original work and almost never write anything." - are described in detail. I will only quote the his classification, the rest you should read in his original, quite wonderful prose.

"These illustrious failures may be classified in the following way: the dilettantes of contemplators; the erudite or bibliophiles; the instrument addicts; the megalomaniacs; the misfits; and the theory builders."

While these are quite fun, and you can always find some colleague that fits each personality, the more important lesson is that everyone gets trapped in these mires of the mind from time to time. The important thing is to realise that you are stuck in an unproductive mindset and be able to move on.

6. Social Factors Beneficial to Scientific Work

These are the ever current problems of funding, combining a profession with science and combining work and family. His basic stance is that you can always do science, but you should be ready to sacrifice having a life. Good advice.

7. Stages of Scientific Research

Here Cajal goes over the practicalities of the experimental method. Observation - Experimentation - Working hypotheses - and Proof. It is material gone over many times in other books, although Cajal has the benefit of being brief.

8. On Writing Scientific Papers

To start with Cajal references a Mr. Billings and states four rules: Have something to say; Say it; and Stop once it is said. The fourth rule wasn't as catchy so I'll just drop it. Then he writes a bit about credit and curtesy.

9. The Investigator as Teacher

The final chapter expounds on the importance of fostering future researchers to continue the work once you have gone. He concedes that it might be restful and rewarding to work in solitude but that "Posterity has always been generous with the founders of schools." He then goes on to describe the pains and pleasures of trying to combine research and teaching. First, how to find a suitable potential investigator; then how to guide him; and finally how to see him become a successful scientist leave science for profit and glory.

In conclusion, it is one of the few books that actually describes what it means to be a scientist, specifically a physiologist. It is certainly one of the very, very few that does so while being well written. I heartily recommend it to all my colleagues, and to any one else who might be interested in what science really is.

When is the evidence good enough?

Saturday Morning Breakfast Cereal is the name of a web-comic that is just about as good as xkcd, and they pose some very intimidating questions.

Anecdotal evidence is popular in medicine. Obviously it's not called that. It's called a "case report" or an "observational study". Basically you see a number of patients and then you pick one that supports some notion you have and put it on a pedestal. Another manner of doing it is to collect a number of these cases and call them a case-series. The next level is to collect all your cases and dig around until you find some common feature in all of them or in a sub-population. Finally you get to the level of looking at all people with a certain condition, usually nationality and some disease, and then you can call it "epidemiology". The problem with all of these designs is that if you look hard enough you can find something interesting-looking in any collection of data. This is especially true if you don't care what you are looking for.

The next level of scientific quality is to care what you are looking for from the beginning. This means you pick a group of people and say that, for example: "I believe those with higher blood pressure now will die sooner than those with lower", and then you wait. After some time, in the best case a predetermined time, you check how many have died and draw your conclusions. Such a conclusion might be that high blood pressure is a risk factor for death. This is not necessarily such a bad way of doing science. The problem is that then you start muddling through your data to look for a reason for the difference in death rates, and then you are suddenly producing anecdotal evidence again.

The highest level of evidence is not to use a found population with some difference, but to use a homogenous population and induce a change in some of them while comparing them to the rest. This is experimental science and it is the only way to show causal relationships. You still have to preconceive what you are looking for, otherwise you are still just muddling through the data looking for differences. This means that an experimental study is only valid as such for the question it was designed to answer. If you take the population in an experimental study and look at something else then you are back to doing an observational study, producing anecdotal evidence.

So, what is the problem? Anecdotal evidence often turns out to be correct. This is how much science comes about; you have an anecdote (say a case study, or some epidemiological finding, or a post-hoc analysis of an experimental study) that leads you to a hypothesis. Then you test the hypothesis with an experimental study to see if it holds up. Easy. This is how it is supposed to be. The problem is that anecdotal evidence is often taken for true directly (by the media, policy makers and the public at large), without further testing, and the later experimental evidence does not get the same attention even though the science is better.

For scientists this is not as big a problem as for the public. By following the field you learn what questions are being asked and which study was designed to answer which question. With clinical trials this is solved today by publication of the protocol and hypothesis before the study starts. Pre-clinical papers on the other hand are mostly experimental in nature, but they are often written so that it is not clear if the conclusion is based on the original question or if it is something the authors picked up on the way. This is because there is not enough space to tell the story as it happened, and that it frankly wouldn't be that good a read. The principle is that you take what you have and write the best story possible. That's how to get published. What it also means is that, unless you know the investigators and what their main focus is, you don't know if the study was a preconceived experimental study or a post-hoc study.

Getting know the investigators in your field means going to a lot of conferences and listening to a lot of talks. That's when it is a good thing to have some online comics to fall back to. Because falling asleep is embarrassing.

Conspiracy against science

This is a warning to all scientists out there (and anyone else who has too much control over their working hours). Stay away from Minecraft! I first found out about it through the online comic xkcd that published an excellent strip about policy makers, the media and science, or as I like to call it: Stay away from the green jelly beans!

Since I always do what xkcd tells me to. I immediatly tested the old version of Minecraft, which is free, and then bought the beta. Yes, it's still in beta and yet they have over two million paying users.

The home page describes it as:

"Minecraft is a game about placing blocks to build anything you can imagine. At night monsters come out, make sure to build a shelter before that happens."

Which describes the pretty well. Depending on your imagination and persistence you can build anything, it's like LEGO that way. You can either go the drug-induced but orderly xkcd-way, or you can go all out crazy and build a scale model of the USS Enterprise (that's United federation of planets Star Ship Enterprise in case you thought anything else). The other way to go in the SF - Fantasy spectrum would be a recreation of Tolkien's Middle Earth, but no one could be that crazy, or could they?

The alternative is to play the survival mode, where monsters come at night. It starts all peaceful with sheep and cows milling around in a serene, sun-lit landscape. Then the sun goes down leaving the world pitch dark, and then the monsters come. First you hear the clacking of bones, or the groaning of zombies, and then you die. If you know this you build a house and light it with torches before night comes, then you are safe.

By the way, parents of small children don't always appreciate your explaining this aspect of the game to their children. Apparently it makes them afraid of the dark, or makes them excitedly tell their younger siblings that the monsters come at night, which amounts to the same thing.

So, that's about it.

Man modestly mines minerals.

Moonlight marks midnight.

Man-eating monster mangles.

Mutinously mutes machine.

Mired mind must muster motion.

Meets miserable methods-mountain.

Meekly manages more Minecraft

Two-season country

Sweden has apparently turned into a two-season country. A month ago it was still below 10°C in the day and frost at night. Now it is 28°C in the shade and too warm to live in the sun. There is only one thing to do: Skip work during the day and bring the dog down to a local stream.

She has completely lost all respect for water now. If it looks wet (and there's a frisbee), you can just jump right in. If the bank is too high to get up, then that is a problem for later.

Once home again she goes to sleep, I surf on facebook and blog until sundown, and then; then I continue working on my grant. It's due Monday, so there are oceans of time.

Return to the Dark Side

In 1984 my father brought home our first computer. From the year you should be able to guess what it was. In '89 my most used system changed to the new Good Guys(TM): MS-dos, later know as the Bad Guys(TM): M$-Windoze. Being able to play better games was the main reason for that change. Later, 1995 or so a friend brought some disks he had got of a BBS with something called GNU/Linux. For the next ten years I used Linux almost exclusively, including my early years at university and in my research. However, it eventually became apparent that you cannot do research on Linux alone. You need Microsoft Word, and you need EndNote. In addition, you would rather like to have Adobe Photoshop and Illustrator as well.

For years I either double-booted with both MS-Windows and Linux, or, had several computers in parallel. The idea was to have one for heavy work, like statistics and dual-boot it for Windows-programs like Word and Photoshop. Then another computer for most other things. This other computer would run Linux only and be quite small, a netbook, as they later became known as. The result was that I had to bring both computers most of the time. That was a good idea that turned out horribly wrong.

I didn't consider Apple for years, it was expensive, didn't run games and wasn't GNU (If you don't know why that matters, just ignore it). Then I bought a iPhone 4, upgrading directly from a phone with buttons that could send SMS and make phone-calls (Yes, I am a late adopter). And it was slick. As slick as I remembered thinking the Apple Macintosh Plus was in its time. It was easier to check mail with the iPhone than to use the computer. One thing led to another, and yesterday my 13 inch MacBook Pro arrived.

That's enough personal history for one post. The rest is the mandatory setup-story: How to migrate from Linux/Windows dual-boot with Linux-on-the-side to OSX Snow Leopard.

All my old files were easily copied by way of a couple of 1TB USB hard-drives. The hard part was consolidating the three different versions of the project-folders from different computers. For file-organization I opted to keep my age-old folder hierarchy and using the Desktop-folder as root so that I get easily accessible icons. Basically, I have a "job" folder for work-related stuff, and a "div"-folder for other stuff. I then linked the Pictures, Download, Documents, Public and Dropbox folders to the desktop as well. In bash and emacs it makes for a longer path (/Users/user/Desktop/job vs. /home/user/job), but otherwise I think it will be easier.

The Apple backup-facility: "Time Machine" is a dream to setup. It needs no explanation, just use it.

Email was easily imported as mbox-files and, I guess, most other formats too. It automatically handles gmail, appropriately compared to other accounts. It is well built and apparently handles my 15000+ mail-archive without any particular complaints.

Syncing with my iPhone got all my contacts and calendars automagically transferred. Brilliant.

Adobe programs are in essence designed for aApple so they pose no problems, so far. I got CS5 Design Premium to get Illustrator, Photoshop and Bridge (the rest are a bonus). Note that image analysis functionality only is included in Photoshop extended, thus Premium.

MS Word, Powerpoint and Excel are necessary. The Apple variants Pages, Keynote and Numbers are probably nice, but from experience the compatibility is not good enough. I got Microsoft Office 2011.

Then the problems started. EndNote X4, does not work with Word 2011 under OSX. You have to update to 4.0.2, and activate Cite-While-You-Write again in EndNote "Customizations". This may be enough but you might have to copy the "EndNote CWYW Word 2011.bundle"-subdirectory from the CWYW subdirectory to the Word Startup directory in the Office directory. And, then you have to activate the Startup directory for Word under "Preferences - File locations".

(If you didn't know what GNU was earlier, then you can stop reading now. The rest is qualified nerdery.)

One fantastic thing, which I hadn't realized, is that OSX uses proper movement-bindings by default. C-f, C-b, C-n, C-p, C-a, C-e etcetera move the cursor as they should. As usual the Microsoft applications do not conform by default, but it can be fixed. In Word open "Tools - Customize Keyboard" and look for EndOfLine, StartOfLine, LineUp, LineDown, CharLeft, CharRight and EditClear. You have to look under "All Commands", but you can type the first few letters to search. Then just add the proper bindings. Even C-k can be made to work by selecting "Tools - Macros" and creating a new macro with the lines:

Selection.EndKey Unit:=wdLine, Extend:=wdExtend

Selection.Cut

Put them just before the "End Sub" line. Then you go back to Customize Keyboard and set your key-binding.

Emacs is always a bit of a chore to install on any new system. You have to get your bindings just so, and get all your local enhancements to work as they always have (emacs-users are conservative that way, not like the insanely fundamentalist vi-users, but still). The first thing to do is always (setq inhibit-splash-screen t). The splash-screen is a Bad Thing(TM). Then the best way is to use the Apple command-key as Meta since the Macintosh uses both Alt-keys for character-bindings, i.e.

(setq mac-option-key-is-meta nil)

(setq mac-command-key-is-meta t)

(setq mac-command-modifier 'meta)

(setq mac-option-modifier nil)

Today OSX (and most good systems) use UTF-8 as the default character coding system. You can make sure that this works by setting

(set-terminal-coding-system 'utf-8)

(set-keyboard-coding-system 'utf-8)

(prefer-coding-system 'utf-8)

I found the easiest way to install packages was to just place them in my user's site-lisp directory and use load-path for each one. The user site-lisp directory is: "~/.emacs.d/site-lisp/". Then I pretty much copy-pasted my old .emacs and everything seems to be working splendidly.

Caps-lock is Evil, the button to the left of "A" should be a control-key. Luckily, you can change your modifier keys under "System preferences - Keyboard - Modifier Keys".

I end with a short note of thanks to those who wrote the web-pages and manuals where I found many of these hints. I'm sorry to say I didn't save the references while I did the setup, so I have no idea where I stole the ideas from.