The Gruesome Job of Managing Petabytes of Scientific Data

Mar 26, 2024 | Media

“Oh, wow, the IT department charges us $11,000 a month for data storage. Why don’t we put some data on cheaper storage? Mark, why don’t you look into that?” the principal investigator asks the postdoc who just joined the team.

There are thousands of similar discussions in research labs around the world, sometimes because there is pricing sticker shock and at other times because the IT department is asking to curb storage consumption through quotas. In some cases, there is a quick solution (“How hard can that be?”), but in many other cases, this leads to a real quagmire. Let’s follow Mark’s journey through this quagmire and propose a solution with the help of tools such as Starfish.

Read More at Medium →

Recent Posts

Managing unstructured data to boost performance, lower costs

Unstructured data is the fastest growing data around. It's increasing at a compound annual growth rate of 61%, according to IDC, and will account for 80% of worldwide data by 2025. For many large IT organizations, it passed that mark a while ago. Unstructured data...

Upcoming Events

21 Surprising things you can do with Starfish