Researchers produce reams of images as they watch model organisms grow and develop or study the structures of proteins, and Nature News writes that organizing and processing these images so that they can be shared with others can be challenging.
One astrophysics effort that studies the sun simply doesn't share the original data — it collects about 1.5 terabytes of image data each data — through its Helioviewer image browser. According to Nature News, each image the Solar Dynamics Observatory takes is about 12 megabytes in size, and it takes pictures every 12 seconds. Helioviewer, though, includes every third picture and compresses it to 1 MB in size. "We have images of the data," Jack Ireland, a solar scientist at ADNET System, says, "not the data itself."
Other research groups like that of Philipp Keller at Howard Hughes Medical Institute's Janelia Research Campus have developed their own approach to transferring large image files, Nature News reports. Keller's team, which tracks GFP-tagged fruit flies, zebrafish, and mice over time, devised a new file format, the Keller Lab Block (KLB) file format, for their images that divides an image in to parts that are then compressed in parallel. This, they say, triples the speed at which the images are compressed.
At the same time, researchers studying molecular structures have created a separate database to house the raw cryoEM data of molecular structures deposited in the Electron Microscopy Data Bank. That whole collection, Nature News notes, is 34 terabytes. To be able to transfer that data, the European Bioinformatics Institute pays for users' access to two high-speed file-transfer services. The database's grant ends next year, but EBI's Ardan Patwardhan says that field has voices its support for it to continue. "They feel that this should be considered a vital archive for the community — which is nice to hear," he tells Nature News.