Big Data. Monster Storage: The Northeast Storage Exchange

February 5, 2019

The world is in the midst of a data revolution. In response the five universities that form the MGHPCC consortium have each initiated a new data science institute, initiative or program, but how and where to store all that big data? The Northeast Storage Exchange (NESE) is a shared regional storage resource funded by the National Science Foundation which aims to do just that.
NESE is housed at the Massachusetts Green High Performance Computing Center (MGHPCC) and operated and maintained as a long-term partnership between the MGHPCC, its university partners Boston University (BU), Harvard University, the Massachusetts Institute of Technology (MIT), Northeastern University (NEU), the University of Massachusetts (UMASS) network, and Red Hat Inc.
In particular, NESE is expected to be a boon for those who have need of large volume (multi PB), high performance or archival storage with high bandwidth connectivity to the national research networks, specifically the flood of data expected from the inexorable advances in computing and storage technology, arising, for instance, through the "internet of things" or advances in DNA sequencing speed. “Over time, we will provide essentially all useful types of storage from POSIX file systems to block store for Cloud infrastructures to mounts to Globus endpoints and long-term archives,” says project Principal Investigator Saul Youssef, a Research Associate Professor in the Department of Physics at Boston University.
Youssef cites four keys strengths underpinning NESE’s success:
First its location at MGHPCC, placing it only a few feet from five of the great research universities in the Northeast. “It's hard to imagine a better spot,” says Youssef. “Such close shared proximity means that we can easily have high-speed networking with low latency to five institutions at once.”
Second, the agreement of Scott Yockel's FAS Research Computing team at Harvard ( to take on building and operating NESE for the long term. “Getting Scott and his deep and talented 25 person team’s commitment to NESE is a very significant indicator of long term success,” says Youssef.
Third, Youssef points to NESE’s relationship with Red Hat Inc. “Because we want to be offering multiple different storage types with a commonly managed back end, Ceph ( storage was a natural choice for us. Through a major agreement between BU and Red Hat Inc. (, we have had major contributions and ongoing involvement via Red Hat, including software licenses and the direct involvement of Sage Weil and his Ceph team within Red Hat.”
Finally, Youssef cites basic economics. “For research projects in the region, NESE storage is going to be far less expensive than either commercial cloud storage or running your own storage,” says Youssef. “For computing that's already happening at the MGHPCC, it is also going to be quite easy to get high bandwidths to NESE storage. This means that if we can do a good job technically, NESE will be in a position to grow rapidly with buy-ins from research, engineering and education projects, not to mention possible partnerships with local technology companies, biotechs and, startups.”
Learn more about NESE at their new website

Research projects

Dusty With a Chance of Star Formation
Checking the Medicine Cabinet to Interrupt COVID-19 at the Molecular Level
Not Too Hot, Not Too Cold But Still, Is It Just Right?​
Smashing Discoveries​
Microbiome Pattern Hunting
Modeling the Air we Breathe
Exploring Phytoplankton Diversity
The Computer Will See You Now
Computing the Toll of Trapped Diamondback Terrapins
Edging Towards a Greener Future
Physics-driven Drug Discovery
Modeling Plasma-Surface Interactions
Sensing Subduction Zones
Neural Networks & Earthquakes
Small Stars, Smaller Planets, Big Computing
Data Visualization using Climate Reanalyzer
Getting to Grips with Glassy Materials
Modeling Molecular Engines
Forest Mapping: When the Budworms come to Dinner
Exploring Thermoelectric Behavior at the Nanoscale
The Trickiness of Talking to Computers
A Genomic Take on Geobiology
From Grass to Gas
Teaching Computers to Identify Odors
From Games to Brains
The Trouble with Turbulence
A New Twist
A Little Bit of This... A Little Bit of That..
Looking Like an Alien!
Locking Up Computing
Modeling Supernovae
Sound Solution
Lessons in a Virtual Test Tube​
Crack Computing
Automated Real-time Medical Imaging Analysis
Towards a Smarter Greener Grid
Heading Off Head Blight
Organic Light-Harvesting Antennae
Art and AI
Excited by Photons
Tapping into an Ocean of Data
Computing Global Change
Star Power
Engineering the Human Microbiome
Computing Social Capital
Computers Diagnosing Disease
All Research Projects

Collaborative projects

ALL Collaborative PROJECTS

Outreach & Education Projects

See ALL Scholarships
100 Bigelow Street, Holyoke, MA 01040