Manually handling instrument data streams was leading to errors and bottlenecks
A leading U.S. research hospital used more than two dozen different instruments to support diagnosing patient conditions and to track the progress of their treatments. However, the hospital’s researchers and administrators were frustrated by how much time the researchers had to spend manually copying and routing files to their next storage destination. The process was tedious and prone to errors, often sending professionals on wild goose chases for lost or misplaced data. Once they realized they were spending more time on data logistics than discovery, they knew it was time for a change.
Solving instrument data flow problems, however, is not simple. The ideal solution would need to automate many of their manual processes while ensuring data was accurately moved and stored. The solution would need to:
- Standardize file naming and storage for easy reference.
- Send copies of files to deep archive storage in the cloud.
- Transfer files for analysis on high-performance computing clusters.
- Free up space by automatically deleting files from the instruments after they were securely stored.
- Assure no bitrot of data as it moves through the pipeline.
Starfish Storage’s ability to automate scientific workflows solved the problem
Enter Starfish, the data management tool that makes file-based workflow automation easy and scalable. Here’s how it worked:
- Seamless Data Transfers: Starfish managed the entire list of instruments, including where data should be stored and for how long. New files were automatically transferred from instrument servers, validated with MD5 hashes to ensure data integrity, and directed to specific storage locations.
- Intelligent Workflow: The workflow was customized to handle the institutes’ unique needs. For example, older files were automatically deleted from the instrument servers only after duplicates were securely backed up in central storage and offsite cloud services.
- Scalability and Flexibility: The institutes’ tailored workflow now manages dozens of instruments across two labs, handling up to 250 GiB of data daily without a hitch.
Starfish-Managed Research Data Workflow
Starfish automation led to fewer bottlenecks, more time spent on research
With the automated system in place, researchers no longer worry about data management. The data is where it needs to be when it needs to be there, and they can focus their energies on patient care rather than moving files around.
The Mic Drop Moment: “Once we implemented Starfish, complaints from researchers about managing data went to nearly zero… having the Starfish GUI front end to manage the different pieces and parts of the environment is essential,” said a Senior Programmer Analyst at the hospital.
Daily processed data across dozens of instruments
You can learn more about this solution
Do you have researchers complaining about how much time they are spending in the lab handling data? Starfish makes it easy to automate all or parts of complex workflows and move data off of instruments to multiple locations simultaneously. Even better, it scales easily as your data volumes grow and you add instruments. Download the full case study and share with your colleagues how you can modernize your instrument data flow and get scientists back to doing science.
