Use Case - Scientific Workflows

From instrument to archive, automatically

Starfish orchestrates data from creation to long-term storage. It can chain incremental workflow steps, extract metadata via Meta-x, apply tags programmatically or by users through Zones, all with lineage tracked against every file version.

Built for research environments where data lands continuously and needs to be acted on quickly.

Scientific-workflows

Capture

Pick the data up where it lands

Chained incremental workflow steps move data from multiple instruments to designated analytical areas, with zero-loss standards, auto-resume, and fault tolerance.

Chained incremental steps

Data moves from multiple instruments to designated analytical areas as a sequence, not a set of manual handoffs.

Zero loss, auto-resume

Zero-loss standards, auto-resume, and fault tolerance, because instruments do not pause for a network problem.

Secure transit

Movement to object storage enforces TLS certificate validation rather than assuming the path is safe.

Meta-x extraction at capture

Internal metadata is read from scientific and media formats, including DICOM, HDF5, and BAM, as the data arrives.

Scientific-workflows-block2

Starfish Automtion Engine

See a workflow built against your own instruments

No synthetic demo. We take one real instrument path and show you what capture to archive looks like automated.

Scientific-workflows-block4

Describe

FAIR by construction, not by cleanup

Predefined configuration schemas and templates are applied via Zones, the REST API, or the CLI, and every file version carries a unique Starfish ID (SFID) so lineage is captured rather than reconstructed.

Templates

Required and optional fields, applied through Zones, the REST API, or the CLI - so description is part of the workflow.

Lineage on every version

Each file version is identified by a unique Starfish ID (SFID), which is what makes a result traceable to its source run.

Normalization maps

Data from different systems or instruments can be compared, instead of being merely co-located.

Integrity and gating

Compliance gating and ACID-compliant integrity throughout, so a non-conforming dataset does not quietly proceed.

Automate

Your logic, in your language

Custom logic runs in any language, including Bash, Python, and Go, with integrations for Airflow, n8n, Nextflow, Snakemake, and HPC schedulers - and curated data can be sent on automatically.

Any language

Bash, Python, Go, or anything else you already maintain. Starfish runs with the tools and scripts you use at scale rather than replacing them.

Workflow engine integrations

Airflow, n8n, Nextflow, Snakemake, and HPC schedulers, so Starfish fits the pipeline you already have.

Curated data, delivered

Send curated data automatically to HPC clusters, AI/ML environments, or cloud storage for backup.

Audit-ready by default

Lineage captured automatically supports funder and journal requirements on FAIR data and long-term retention.

Scientific-workflows-block5

In Practice

“Once we implemented Starfish, complaints from researchers about managing data went to nearly zero.”

Senior Programmer Analyst

A leading US research hospital

Dig Deeper

Resources for the curious technical reader

Starfish-Storage-automates-dozens-of-scientific-instrument-workflows-for-a-leading-research-hospital
Blog

Starfish Storage automates dozens of scientific instrument workflows for a leading research hospital

When researcher time was being eroded by having to manually process dozens of instrument data workflows, Starfish Storage stepped in and set researchers free.

4-min video

ASU-Case-Study-Feature-Image
Blog

How ASU scratch storage went from instability to control

ASU used Starfish to help researchers stop using scratch storage for permanent archive and bring storage utilization down to 70%

4-min read

Glossary-card-catalog-v1
Glossary

Get everyone using the same words

Storage admins, data owners, and finance can all say “cold data” and mean three different things. The glossary pins each term to one plain definition.

Quick reference

FAQ

Frequently asked questions

What does Starfish automate in a research workflow

Orchestration from instrument to archive: chained incremental steps that move data from multiple instruments to analytical areas, then on to HPC clusters, AI/ML environments, or cloud archive, with metadata captured along the way.

How does this support FAIR data principles?

Through structured metadata templates with required and optional fields, applied via Zones, the REST API, or the CLI, and lineage tracking where every file version carries a unique Starfish ID (SFID).

Can we keep the scripts we already have?

Yes. Custom logic runs in any language, including Bash, Python, and Go, and Starfish integrates with Airflow, n8n, Nextflow, Snakemake, and HPC schedulers.

Which scientific formats can Starfish read inside?

Meta-x extracts metadata from a wide and rapidly expanding catalog of scientific and media formats, including DICOM, HDF5, and BAM. Additional formats are supported on request.

What happens if a transfer fails midway?

Zero-loss standards, auto-resume, and fault tolerance apply, and job records mark what has already moved so a rerun does not repeat completed work.

Can data from different instruments be compared?

Yes. Normalization maps let data from different systems or instruments be compared rather than simply stored alongside each other.

Ready When You Are

Bring us the instrument nobody has time to babysit.

We will automate one real path on your own data - capture, description, and archive - and show you the result.