Menu

The MGHPCC Supercloud

March 27, 2019

By Helen Hill for MGHPCC News
The MGHPCC Supercloud, operated by MIT Lincoln Laboratory Supercomputing Center (LLSC) is a facility intended to be of particular use to people who need to transition from their desktop workstation to a high performance computing cluster in order to solve larger or more complex problems, but do not have the time to gain in-depth specialized knowledge about parallelizing application software, running batch jobs, optimizing startup times for large collections of simultaneous jobs, and other details that end users must otherwise master in a Linux cluster environment.
Managed by MIT, but as a separate enterprise dealing with security-related (classified) research, Lincoln Laboratory began as a center for radar and early warning system research and has a long history of advanced technologies in sensors (signals intelligence, image recognition, etc) and associated computing. Today the Laboratory (now the largest US Department of Defense federally funded research and development center) is home to numerous unclassified open research activities, for example, development of the imaging devices and lenses for TESS or LIDAR-based tools now providing FEMA with automated post-disaster operational damage analysis.
The MGHPCC Supercloud project grew out of a desire to strengthen collaboration between Lincoln Laboratory and regional partners in the MGHPCC Consortium (BU, Harvard, MIT, Northeastern, and UMass). LLSC having already demonstrated its success in providing a similar cloud service to Lincoln Laboratory researchers, opened the doors to provide its service to members of the MGHPCC consortium.
The Supercloud supports most of the desktop computing environments that are commonly used in the research and engineering community today, including Matlab, R Studio, Jupyter Notebooks, and Machine Learning frameworks such as Tensorflow, Caffe, and Theano.
“If there’s something a user needs and we don’t already have it or something equivalent to it we will work with the user to help them get set up with what they need,” says Lauren Milechin, who is in charge of new-user liaison.
In particular, the MGHPCC Supercloud provides a software platform which allows users to launch large-scale interactive compute jobs from their desktop, as well as facilitating ready sharing of large volumes of project data. The Supercloud experience enables reference datasets to be pre-positioned in databases, while also providing access to software modules and training to reduce user ramp-up time, all within a responsive, interactive supercomputing environment.
Simple line commands allow users to tailor hardware selection (ie from a single processor core to a whole node), using LLMapReduce (an implementation of native MapReduce), carry out language agnostic parallel data analysis or run Parallel Matlab. Users are also able to couple Supercloud resources with web-based interactive development environments like the popular Jupyter Notebook, or manage dynamic databases from a simple GUI.
“While you can run traditional big MPI Fortran code on the MGHPCC Supercloud, our really big success has been our ability to get really large numbers of people with little if any prior parallel computing background effectively working on these systems in very short order,” says Jeremy Kepner, a Lincoln Laboratory Fellow and Founding Head of the LLSC.
High-performance data analysis (HPDA) requires high-level programming environments but also rapid interaction and fast turnaround. By optimizing every aspect of its HPDA system, the LLSC TX-Green system running on 32 000 cores (512 x 64-core Xeon nodes) with the same setup and software stack as the MGHPCC Supercloud has been demonstrated to be able to launch 260 000+ data analytics in just 40 seconds ie 6000+ launches per second, or a 1000x speed-up over standard approaches.
In another measure, some analytic applications are written uniquely for Microsoft Windows. Standard approaches based on Virtual Machines or Windows HPC can take hours to launch across thousands of cores. MGHPCC Supercloud launch system optimizations enable launch of 16 000 + Microsoft Windows environments (running WINE) on 16 000+ cores (256 x 64-core Xeon nodes) in just five minutes: ie 50+ launches per second or a 100x speed-up over standard approaches.

Request Your Account

visit supercloud.mghpcc.org/requesting-account where a member of the Supercloud team is standing by to help you migrate your application to their system.

Links

MGHPCC Supercloud
Lincoln Laboratory Supercomputing Center

Research projects

Foldit
Dusty With a Chance of Star Formation
Checking the Medicine Cabinet to Interrupt COVID-19 at the Molecular Level
Not Too Hot, Not Too Cold But Still, Is It Just Right?​
Smashing Discoveries​
Microbiome Pattern Hunting
Modeling the Air we Breathe
Exploring Phytoplankton Diversity
The Computer Will See You Now
Computing the Toll of Trapped Diamondback Terrapins
Edging Towards a Greener Future
Physics-driven Drug Discovery
Modeling Plasma-Surface Interactions
Sensing Subduction Zones
Neural Networks & Earthquakes
Small Stars, Smaller Planets, Big Computing
Data Visualization using Climate Reanalyzer
Getting to Grips with Glassy Materials
Modeling Molecular Engines
Forest Mapping: When the Budworms come to Dinner
Exploring Thermoelectric Behavior at the Nanoscale
The Trickiness of Talking to Computers
A Genomic Take on Geobiology
From Grass to Gas
Teaching Computers to Identify Odors
From Games to Brains
The Trouble with Turbulence
A New Twist
A Little Bit of This… A Little Bit of That..
Looking Like an Alien!
Locking Up Computing
Modeling Supernovae
Sound Solution
Lessons in a Virtual Test Tube​
Crack Computing
Automated Real-time Medical Imaging Analysis
Towards a Smarter Greener Grid
Heading Off Head Blight
Organic Light-Harvesting Antennae
Art and AI
Excited by Photons
Tapping into an Ocean of Data
Computing Global Change
Star Power
Engineering the Human Microbiome
Computing Social Capital
Computers Diagnosing Disease
All Research Projects

Collaborative projects

ALL Collaborative PROJECTS

Outreach & Education Projects

See ALL Scholarships
100 Bigelow Street, Holyoke, MA 01040