The Gruesome Job of Managing Petabytes of Scientific Data

Mar 26, 2024 | Media

“Oh, wow, the IT department charges us $11,000 a month for data storage. Why don’t we put some data on cheaper storage? Mark, why don’t you look into that?” the principal investigator asks the postdoc who just joined the team.

There are thousands of similar discussions in research labs around the world, sometimes because there is pricing sticker shock and at other times because the IT department is asking to curb storage consumption through quotas. In some cases, there is a quick solution (“How hard can that be?”), but in many other cases, this leads to a real quagmire. Let’s follow Mark’s journey through this quagmire and propose a solution with the help of tools such as Starfish.

Read More at Medium →

Recent Posts

Starfish Helps Tame the Wild West of Massive Unstructured Data

“What data do you have? And can I access it?” Those may seem like simple questions for any data-driven enterprise. But when you have billions of files spread across petabytes of storage on a parallel file system, they actually become very difficult questions to...

Upcoming Events

21 Surprising things you can do with Starfish