The Gruesome Job of Managing Petabytes of Scientific Data

March 26, 2024

“Oh, wow, the IT department charges us $11,000 a month for data storage. Why don’t we put some data on cheaper storage? Mark, why don’t you look into that?” the principal investigator asks the postdoc who just joined the team.

There are thousands of similar discussions in research labs around the world, sometimes because there is pricing sticker shock and at other times because the IT department is asking to curb storage consumption through quotas. In some cases, there is a quick solution (“How hard can that be?”), but in many other cases, this leads to a real quagmire. Let’s follow Mark’s journey through this quagmire and propose a solution with the help of tools such as Starfish.

Read More at Medium →

Medium

Recent Blog Posts

Chapter Two in a Series: Transforming the Research Data Management Culture at Harvard Faculty of Arts and Sciences Research Computing

October 3, 2025

Chapter One in a Series: How Harvard Faculty of Arts and Science Research Computing and Starfish built an innovative chargeback system that is flattening storage growth and recovering $2M a year in annual storage costs

October 1, 2025

Customer Spotlight: Starfish Storage makes data easier to find, store and protect for Clemson University’s Research Computing and Data Infrastructure team

September 27, 2025

Upcoming Events

NVIDIA GTC Washington, D.C.

Oct 26, 2025

SC25

Nov 15, 2025

21-things-banner-600x600