Solution
Workflow Automation
Automate how users, devices and applications manage files and objects by implementing custom workflows or building bespoke file-centric applications on the platform.
This usually lands on someone's desk when…
Instruments fill local disks over a weekend.
Data owners and IT are disconnected.
An audit, an IP review, or a journal reproducibility requirement.
Sneakernet is the common mode of file sharing, and USB thumb drives line researchers’ desks.
The problem, in cost and risk terms
Every recurring task becomes its own project.
Researchers, engineers, and operations teams lose significant time manually tracking new data, moving it between systems and processing stages, and reconstructing what happened to it, because most environments lack an automated, auditable platform managing data from creation through processing to cold storage.
Every recurring movement, classification, or cleanup task becomes its own project: a new ticket, a new script, a new meeting.
Without automatic versioning, tagging, and metadata capture at each step, reproducing a result, demonstrating compliance, or simply finding the right files is guesswork.
Who Feels It
VP of R&D IT
Align the needs of data owners with the needs of IT.
Head of Scientific Computing / Director of HPC
Turn recurring one-off tasks into policy enacted by automation.
Core Facility Director
Stop instruments filling local disks over a weekend.
Research Data Manager
Inspect your entire file estate to verify data integrity, data classification and policy compliance.
What changes with Starfish
Go well beyond just moving data.
Starfish allows users to build workflows or bespoke file-centric applications that manage files and objects at massive scale. These applications and workflows use the catalog of file metadata to inform the business logic that the Automation Engine then can move, backup, archive, migrate, replicate, burst to cloud, delete, add tags, extract or append metadata, enact policies, classify, implement data protection, or run any code against any file or directory, directly via script, sidecar file ingestion, or 3rd party application integration, then maintains an audit trail and full history of the action. Every file version is assigned a unique Starfish ID (SFID) in the Starfish Unstructured Data Catalog, enabling strict experiment tracking, metadata lineage, and audit trails across complex workflows. Custom tags extend file metadata for classification and organization, governance and analytics, and automated policy actions.
Find files and assemble datasets by searching for inheritable directory metadata, filesystem attributes, tags, and extracted metadata in a single query. Once the workflow or application is defined, it can be scheduled, triggered or manually invoked. The Automation Engine then executes the actions within the workflow. Workflows can be chained together (the output of one becomes the input or trigger for the next) with metadata recorded at each step. The tight integration between the Starfish catalog and the Automation Engine creates a feedback loop, where the output of a catalog query is the input to a workflow automation step executed by the Automation Engine. The output of that process writes back to the catalog as key-value metadata updating the history, and describing what was done to the file or what was discovered about it.
“We have scripts that do this.”
Most of our customers did too, and the scripts were fine until they were not. The failure mode is always the same: the person who wrote them leaves, or the file count crosses a line and they stop finishing. A biomedical institute we work with replaced a homegrown copy pipeline that had become unmaintainable; the Starfish sequence now completes in 30 minutes on average.
Built from these use cases
One outcome, composed from proven capabilities
File management workflows, automations and applications at scale.
Storage Analytics
Turns weeks-long manual audits into on-demand reporting and gives IT a defensible basis for forecasts and budget requests.
Archive & Tiering
Automatically classify unstructured data by age, activity, and value, then move inactive or lower-value data off primary storage to more cost-effective tiers.
Capacity Reclamation & ROT Reduction
Redundant, obsolete, and trivial (ROT) data accumulates because nobody has a fast, safe way to find it, prove it is unused, and get authorization to remove it.
Data Movement & Migration
Platform refreshes, cloud adoption, consolidations, and M&A integrations routinely overrun because legacy copy tools cannot move billions of files fast enough.
Data Protection
Traditional backup relies on slow, metadata-heavy tree walks to determine what changed, which cannot complete inside a window once an environment reaches billions of files.
Scientific Workflows
Research environments generate data continuously from instruments, sensors, and pipelines, but it lands in ad hoc locations with no consistent naming, tagging, or lineage.
Data Classification & Metadata Management
Most files carry only basic filesystem metadata (name, size, timestamps) with no reliable way to know what is inside them, who should own them, or how sensitive they are.
User-Driven Data Management
Centralizing every data decision with IT does not scale.
Proof
30+ instruments
A leading US research hospital automated across two affiliated institutes, transferring up to 250 GiB daily to deep cloud archive and HPC clusters, and deleting files from instrument servers only after valid duplicates are verified as stored and backed up.
Mass spectrometers, lipidyzers, calorimeters, and chromatographs
“Once we implemented Starfish, complaints from researchers about managing data went to nearly zero.”
- Senior Programmer Analyst, a leading US research hospital
Seeing is Believing
Pick one instrument and one destination.
We will automate that path end to end, and you will see the audit trail it produces.
Cost Reduced
~200 TB
The Salk Institute reclaimed approximately 200 TB by automating cleanup of redundant BCL sequencing files, deferring capital expenditure.
Risk Avoided
250 GiB / day
Moved with MD5 validation at a leading US research hospital, with instrument cleanup only after backups verify.
Time Saved
30 minutes
A biomedical research institute replaced homegrown scripts with an automated Starfish sequence completing in 30 minutes on average.
Accountability
100% history
Every data action, metadata change, and administrative operation is recorded to the catalog.
Dig Deeper
Keep reading before you commit to anything

Case Study
The Salk Institute
A three-stage workflow: offsite copy to Amazon Deep Glacier on processing, movement to on-premises archive at six months, and automated deletion of superseded BCL files, with no change to how researchers access files.
8-min read

Blog
New White Paper: How ASU Built a Searchable DICOM Catalog for Global Health Research using Starfish
Learn how ASU and Starfish normalized DICOM metadata, enabled self-service CT data access, and reduced research requests from months to minutes.
5-min read

Glossary
Get everyone using the same words
Storage admins, data owners, and finance can all say “cold data” and mean three different things. The glossary pins each term to one plain definition.
Quick reference
FAQ
Frequently asked questions
Our researchers will revolt if you touch their data.
They should. That is why the sequence is archive first, delete later, with warnings before movement and the ability to recover afterwards. ASU freed 3 PB from scratch storage that way and reported very little pushback, because nothing was irreversible and the notifications went to PIs as well as users. Zones put the decision making in the hands of data owners.
What happens to file access when data moves?
Recovery is available through the GUI, the CLI, and, for data owners, inside their own Zone, without a ticket. Where transparency to the user matters, a Starfish script generates soft links so the movement is invisible: Salk’s researchers noticed no difference.
We can’t afford the disruption of a data management project.
Then do not start one. Start with a read-only scan of one or two volumes, out of band, no agents in the data path, no change to how anyone works. Four to six weeks later you have an aging, ownership, and ROT report. Nothing has moved, and you decide whether there is a business case.
We already have this from our storage vendor.
You have it for that vendor. The question is what happens to the files on your other four platforms, and what happens when you replace this array. Vendor tools are the reason you cannot answer a question about your whole estate in one place, and they are the mechanism by which the next migration becomes a project or simply locks you in to that vendor.
How do we know a restore will actually work?
You verify rather than assume. Starfish validates file metadata before and after every job run and compares cryptographic checksums across source and destination; on cloud and object targets it enforces protocol-level header matching. Governance comes from role-based permission boundaries, metadata cataloging, policy-driven automation, and audit reporting.
Ready When You Are
A year later, you can still prove where a result came from.
Starfish records what happened and stores the history.