Analysis of genome-scale data

I thought I should do a post on strategies for handling genome-scale data from large experiments. Like many happy novices we rushed in and performed a couple of huge microarray experiments. Now we are trying to compare these against eachother and against strain-differences in in the form of SNPs. To say that this is not trivial is an understatement. In the end we have come up with a couple of strategies that might be useful to others as well. They might seem simple, but they will help you wade through your data, without getting stuck there for too long.

Don't look at untested data for anything but quality control. In most analysis programs it is very easy to just plot a gene by its intensity, either over all samples or by crude difference between group-averages. This is fun, and can give you some idea of what you are looking for. However, there are anything from 20k-30k genes in your dataset and you will not have time to look at all of them and make a reasoned decision. That means that if you pick genes this way you will indeed pick some hits, but not in a systematic way, and that leaves you open to errors. Either of picking a gene where the variation is too large, or that may only be the n+100th gene in your set and not the top one. The point is: You just don't know. The suggestion is to test your genes first, and then use the list of significantly differentially expressed genes to pick your top hits.

Consider doing a single comparison across several groups. If you have run a large experiment, with several groups (where several is anything from four to whatever). It is a chore to keep track of which genes were significant in which comparison, and even more to then pick a small number of genes to validate that are somehow representative across groups. So, try to pool your data by phenotype. If you have one group where something happens and lots of different controls. Just pool the controls and do a single comparison. That will give you just one list and make your selection process much easier.

When comparing experiments, use rankings. So, now you have a list, or several if pooling didn't make sense. Now you have to find your top candidates for validation. It is then helpful to use rankings. The easiest (and often quite accurate) way is to rank by average difference between groups (that would be your fold-change or log-ratio). If you have just one list, then you are done. If you have two, or several lists, then you can order your lists by rank-sum or rank-product. The rank-sum is a kind method that gives you some hits even if a gene is poorly-ranked (high number) in one of your groups. The rank product will pronounce those that are highly-ranked in all lists. Then you have the very particular rank-difference and rank-ratio. These may be very interesting if you have two experiments with wildly different phenotypes, but that you can not compare directly. If a gene is ranked in the top 10 in one experiment and 7815th in the other, it may be a very interesting gene and will then be ranked highly by the rank-difference or ratio.

Photographic necessities

Coming back to serious photography entails some additional expenses. Foremost being the latest version of Photoshop Lightroom, at the moment version 4. I spent quite some time testing raw-conversion and image database software some years ago. GIMP was fun but useless, not because it is impossible to do the stuff you want to do but because it is so complicated. Capture One was sleek and very nice, but head and shoulders above the rest was Lightroom, version 2 at that time. Subsequently I have also tested the combination of Adobe Camera Raw and Photoshop. Lightroom is based on Camera Raw so the difference shouldn't be that big, but it is. Lightroom is much more focused a on photographic workflow, while ACR and PS are supposed to be able to do everything from science to design.

Previously there were some things you had to go to Photoshop to do well, or at all. Keystone-correction was one of them. Now it is available under "Lens Correction" if you change from "Profile" to "Manual." I find it makes almost any picture look better, the background becomes less intrusive if it is rectilinear, and you just get the feeling of technical quality. For pictures such as the first one, it is absolutely necessary. Either you shoot it with perfect alignment, or you correct it. With sloping verticals it doesn't work at all. In the second it is not as important, but I think it improves the feeling that the image was made with precision.

For the last one it is maybe the most important. Either it is a snap-shot with some toys and a dog, or it is a well-aligned architectural photograph that uses the toys to lead into the image where the dog is. I find it makes all the difference. These are images that you would otherwise need a tripod and a tilt/shift capable camera for, and really quick too, if you wanted the boxer where she is.

So, how did I find this out? This is the second photographic necessity, I would argue, for all serious photographers. At least if you don't have an imp employee who does post-capture for you. You go to the Luminous-Landscape, click on "store" and buy their Lightroom 4 tutorial. With intro if you haven't used it before, without if you just haven't used version 4. It is the best instruction to be had for money. Well person-to-person teaching might reach the same level, but arguably not in the evening after the family has gone to bed, which is when I have time. Buying Lightroom without the tutorial is like buying a car without a key.

A walk in the park

Swedish national holiday, the government decided that Sweden should be more like other places and celebrate a national holiday. The Swedes are reluctant to say the least. It's fine to celebrate midsummer or Valborg (last of April), but you don't celebrate Sweden as a nation. However, you get a day off, so I went out with my new and Shiny™X100 for a stroll.

Not quite clear what it is supposed to be, but drawn by one or other of the kids in the neighbourhood with street-crayon. These kind of stylised, rectilinear things are perfectly corrected using "Enable Profile Corrections" in Lightroom 4. The slight barrel-distortion and vignetting disappears at the click of a button.

Poppy in front of lawn-chair. The macro capabilities of the X100 seem excellent. A thing that bothers me is the slow focus-update in macro mode. Another thing is that it seems quite difficult to achieve critical focus just where you want it. It may take a bit more practice is all.

Tagged utility-box. Critical focus is easy if the subject is flat, contrasty and in good light.

More macro, flower work.

Black and white is also fine.

River Fyris, more like a creek of mud. Still, it is pretty in black and white.

I found the X100 easy to work with. The finder is excellent, and the ability to switch to EVF for critical framing is brilliant. I still get lost in the variety of display and finder-options, but with any luck that will improve with experience. Most controls were easy to find, and usable even without moving your eye from the finder. It is annoying that the menu-system sucks so badly. Switching between manual and automatic ISO should be simpler. All options that you regularly use when shooting should be in the same place. Those would be ISO and auto-ISO, flash-compensation, external flash, ND-filter, Self-timer, and Dynamic range. The separate macro function should be done away with. As it is you have to use the macro mode from portrait range, and then you loose your window-finder and get an EVF that isn't fast enough for portrait work. You should be able to frame using the optical viewfinder even in the macro-range, no matter how inaccurate, it is still faster for capturing expressions.

Physically it is of a good size to walk around with. Certainly lighter and smaller than a Leica. Now I am using it with the filter adapter and lens-shade, half-case and strap. The lens-shade does make it too large for a jacket pocket, and the strap is too wide and too long, but apart from that I find it well made. The half-case fits perfectly, and provides just enough of an edge to grip it well with one hand. All told, I found it quick and easy to work with and quite intuitive, and the files are awesome.

X100 First Light

My X100, which arrived today, saw first light at 19:31:21. First I thought I would wait until night and shoot a star field, like a proper telescope. Especially tonight with a venusian solar crossing and all. However, I have weak character. Here's a picture of my foot.

X100 in the post

Saturday evening I buckled. I ordered the X100, and not just any X100 (no, not that X100 either, I have that already), I ordered the black limited edition Fujifilm FinePix X100. They say that many of the quirks that people complained about in the original reviews have been improved upon in the upgraded firmware, so that's good. Earlier you had to test stuff, now with the Internetz you can just read a ton of reviews.

The online photographer:
Fuji Finepix X100 Review by Ken Tanaka.
Shooting with a Fuji X100 by Robert Plotkin.

Luminous Landscape:
Fujifilm X100 On Test by Michael Reichmann.
Fujifilm X100 Follow-up Report by Michael Reichmann.

Reid Reviews (pay-site, but well worth it):
Fuji X100
Fuji X100 at the "Strolling Of The Heifers"
Fuji X100 New Firmware
Fuji X100 (Four Window-finder Cameras)

Zack Arias:
Fuji X100 :: Review

Steve Huff Photo:
Everything tagged X100

I have also had a touch and a feel. Both on the X100 and the NEX7. The NEX7 sits more like a small SLR in hand than I would have thought. You automatically grip the lens with a solid under-grip because otherwise it gets front-heavy, while the X100 have an excellent one-hand balance. Like a rangefinder, but lighter, much lighter. In my book that was 1 - 0 to the X100.

Then we have the finders. In my limited, in-store, testing I would say that the electronic part of the finders are about equally good. Keep in mind I couldn't test them both at once because I had to go to different stores. However, the window-finder in the X100 with digital overlay is possibly the best new idea in photography in ages. It's not as large and bright as a Leica finder, but it is not bad. That's 2 - 0 to the X100.

Then the NEX7 has a higher resolution, and a better sensor-score at base-ISO. So 2 - 1. But the X100 has a much better high-ISO performance, 3 - 1.

Finally the Black, limited edition X100 is just so much sexier than the NEX7, or the the silver X100 for that matter, or any non Leica M. That makes it 4 - 1.

It ships today.

Photographic restart

I have now spent more than half a year without a digital camera other than my iPhone. For this summer I find myself wanting to photograph again, there are weddings, 80th birthdays and road-trips a head, so I have been internet-shopping. Quite a number of exciting cameras have arrived. The digital Leica M iterations are a delight to work with, easy to use in manual, and relatively small. They are however very solid, and you get tired of them after walking around with them for a day. They are also very expensive unless you buy used, in which case I am back to the M8 I had before, M9s are still very expensive used seeing as they are the current model.

After seriously looking around for cameras that allow eye-level photography and are smaller than the Leica-M I am left with three cameras: Sony NEX7, which is class-leading in image quality at low ISO, 24 megapixel, interchangeable lens goodness in a small, really small package. In the micro four-thirds (MFT) class there is the very appealing Olympus OM-D. It is modelled on the old, film OM cameras, and going by the reviews one usable cameras out there. Finally there is the Fujifilm's X100 windowfinder camera, its like a Leica M only with a hybrid digital/opticalfinder and autofocus. Have a look at these excellent size comparisons: Front view, Top view.

Lens-wise there is only one good NEX lens: the Zeiss 24mm f/1.8, but it is getting such good reviews that it is a really appealing possibility. However, the NEX lenses (the Zeiss especially) are quite large for the miniscule body, so much so that when you attach them the camera isn't really pocketable any more. MFT has a number of nice lenses, including thinner, so called pancake lenses. Most of them get quite weak reviews though, especially the pancake-versions. There are two purpose-built Voigtländer MFT-lenses with huge apertures (which is good), but then they are huge, heavy lenses (bad), and the sharpness leaves something to be desired (worse). A good thing with NEX or MFT is that being interchangeable lens cameras with very short registers, they can take practically any lens ever made with an adapter. It is therefore tempting to get one of them to use with a Leica-M-mount adapter, which again turns out to make them quite a bit larger. On the other hand the Fujifilm X100 is not an interchangeable lens camera, but it does have a wonderful, sharp little f/2 lens with a 35mm effective field of view. However, everyone says manual focus on the X100 is horrendous so I would be forced to use automatic focus, which on the other hand is rumored to be quite good. The only way to know how usable and portable they really are is to test and compare the cameras in person before I possibly buy one. So, it is off to the camera shops for me.