Solution

Data Observability &
Cost Control

See everything, fast, and find out what it's costing you.

Data-observability-cost-control-1

This usually lands on someone's desk when…

A budget cycle, and a finance conversation IT cannot win because it has no current numbers.

A capacity crisis, with no supportable strategy to free up space.

A refresh quote nobody can justify arrives.

A data analysis or AI project stalled because the files it needs are spread across vendors and locations.

The problem, in cost and risk terms

You cannot manage a file estate when you can only guess what you have.

IT and storage teams cannot answer basic questions about their own environment: what data exists, where it is, how fast it is growing, who owns it, how old it is, what it costs to store. Conventional tools cannot keep pace with billions of files across multiple file systems, NAS, HPC, and object storage, spanning flash, HDD, and tape, on-premises and in the cloud.

Without near-real-time, storage-neutral visibility, duplicate and orphaned data quietly consumes the most expensive tier, and workflow failures mount as the data needed for them cannot be found.

And there is no defensible basis for a chargeback, showback, or capacity-planning conversation, or even a supportable strategy to free up space when storage systems reach maximum capacity.

Who Feels It

CIO / VP or Director, Infrastructure

Get a defensible, current basis for capacity forecasts and budget requests.

FinOps and Finance

Own chargeback and showback and increasingly fund the whole program.

Storage Buyer

Get insight into what’s driving storage consumption and the tools to implement appropriate controls.

Storage Administrator / HPC Operations Manager

Manage file placement across every storage system you own, from one place.

What changes with Starfish

Ask your catalog, not your file system.

Starfish keeps a near-real-time catalog of every file across every storage system, at petabyte scale. Users can view the catalog contents through dynamic visualizers such as Sunburst, file tree, and historical trend views that show directory sizes, file counts, data age, and cost at a glance. Cost per GB and loaded cost per GB are assigned per volume and device, so file and byte counts become financial metrics.

The catalog provides visibility into data for all of the key stakeholders. Analytics reach administrators through the GUI, CLI, and dashboards; managers through executive and finance dashboards, top-user pivots, capacity forecasts, and churn reporting; and individual data owners through Zones. Report generation can be on-demand or automated and delivered via email, Slack, and similar channels. Catalog contents can also be queried by third-party systems over the REST API and/or MCPs. Once users can see it, they can act on it, in the same platform.

“We already have this from our storage vendor.”

You have it for that vendor. The question is what happens to the files on your other four platforms, and what happens when you replace this array. Vendor tools are the reason you cannot answer a question about your whole estate in one place, and they are the mechanism by which the next migration becomes a project or simply locks you in to that vendor.

Built from these use cases

One outcome, multiple capabilities

Storage Analytics

Turns weeks-long manual audits into on-demand reporting and gives IT a defensible basis for forecasts and budget requests.

Chargeback & Showback

Measure and report storage consumption and cost for users, projects, departments, or cost centers, either for visibility (showback) or to allocate costs back to budgets (chargeback).

Archive & Tiering

Automatically classify unstructured data by age, activity, and value, then move inactive or lower-value data off primary storage to more cost-effective tiers.

Capacity Reclamation & ROT Reduction

Redundant, obsolete, and trivial (ROT) data accumulates because nobody has a fast, safe way to find it, prove it is unused, and get authorization to remove it.

Data Movement & Migration

Platform refreshes, cloud adoption, consolidations, and M&A integrations routinely overrun because legacy copy tools cannot move billions of files fast enough.

Data Classification & Metadata Management

Most files carry only basic filesystem metadata (name, size, timestamps) with no reliable way to know what is inside them, who should own them, or how sensitive they are.

Proof

Over $1.5M a year and growing

Chargeback revenue: $500K in year one and $1.5M in year two, as the program expanded from 10 labs to more than 230.

65 PB and more than 30 billion files on the Cannon cluster

“With Starfish, you rely on metadata and query a database rather than accessing the actual files. That’s critical because our environment supports billions of files.”

- Raminder Singh, Associate Director
Data Science & Research Facilitation,
Harvard FAS RC

  • Identified upwards of 20 PB of deletion candidates
  • Users deleted nearly a petabyte within weeks of rollout.
  • Generating $500K in revenue from chargebacks in year one and $1.5M in year two

Seeing is Believing

Ask for a read-only scan of one or two volumes.

Four to six weeks, no agents in the data path, and you keep the report either way.

Cost Reduced

$2 million

A university healthcare system tiered more than 2 PB to object storage over two years, avoiding $2 million in additional capacity purchases.

Risk Avoided

70% steady

ASU's 4 PB scratch file system stabilized at 70% utilization once consumption became visible, after repeatedly hitting 99% and crashing.

Accountability

230+ labs

Chargeback participants at Harvard FAS, up from 10 labs at launch.

Under Management

2.5 exabytes

Capacity under Starfish management across the customer base today.

What teams do instead

Your real alternatives, side by side

Storage-vendor consoles, backup products, point utilities, and homegrown scripts each answer part of this. Starfish is out of band, vendor-neutral, and stores nothing in a proprietary format.

On small screens this table is shown as stacked cards. Column headers are repeated as labels on each cell.

Status quo (scripts & spreadsheets)Traditional vendor toolsStarfish
Data currencyManual, already stale by the time it is readPeriodic scans with long gaps between runsNear real-time catalog, updated from changelogs
Multi-vendor coverageOne-off scripts, one per platformUsually tied to one vendor's hardwareStorage-agnostic across NAS, HPC, object, cloud, and tape
Cost attributionNot trackedAggregate capacity onlyCost per GB and loaded cost per GB, per volume and device
Who can see itWhoever can read the spreadsheetAdmin-only consoleZones give data owners their own scoped, self-service view
Historical trendsNot retainedLimited windowHistory Mode: metadata as it appeared on any past date, kept forever
Production impactTree walks compete for I/OTree walks compete for I/OOut of band; can read snapshots and changelogs, production undisturbed

Dig Deeper

Keep reading before you commit to anything

Harvard-FAS-Case-Study
Case Study

How Harvard FAS built defensible chargeback across 230+ labs

$500K in year one, $1.5M in year two, and the ColdFront plug-in Harvard open-sourced along the way.

8-min read

ASU-Case-Study-Feature-Image
Blog

How ASU scratch storage went from instability to control

ASU used Starfish to help researchers stop using scratch storage for permanent archive and bring storage utilization down to 70%

7-min read

Glossary-card-catalog-v1
Glossary

Get everyone using the same words

Storage admins, data owners, and finance can all say “cold data” and mean three different things. The glossary pins each term to one plain definition.

Quick reference

FAQ

Frequently asked questions

We can’t afford the disruption of a data management project.

Then do not start one. Start with a read-only scan of one or two volumes, out of band, no agents in the data path, no change to how anyone works. Four to six weeks later you have an aging, ownership, and ROT report. Nothing has moved, and you decide whether there is a business case.

Our backup product already reports on this.

It reports on what it backed up, which is not the same as what you have, and it found out by walking the tree, which is why the backup window no longer closes. We provide much richer analytics and back up files at a scale where most backup products fail.

We have scripts that do this.

Most of our customers did too, and the scripts were fine until they were not. The failure mode is always the same: the person who wrote them leaves, or the file count crosses a line and they stop finishing. A biomedical institute we work with replaced a homegrown copy pipeline that had become unmaintainable; the Starfish sequence now completes in 30 minutes on average, with full reporting and history.

What happens to file access when data moves?

Recovery is available through the GUI, the CLI, and, for data owners, inside their own Zone, without a ticket. Where transparency to the user matters, a Starfish script generates soft links so the movement is invisible: Salk’s researchers noticed no difference.

How do we know a restore will actually work?

You verify rather than assume. Starfish validates file metadata before and after every job run and compares cryptographic checksums across source and destination; on cloud and object targets it enforces protocol-level header matching. Governance comes from role-based permission boundaries, metadata cataloging, policy-driven automation, and audit reporting.

Ready When You Are

Find out what you're paying to store.

One or two volumes, read-only, four to six weeks. The business case comes out of your own data, not ours.