How a Duke-led HPC method brought cell-scale cancer simulation to a single cloud node 

Even on the largest supercomputers, simulations that explicitly resolve every blood cell can only cover a small volume. On Summit, Oak Ridge National Labs’ supercomputer, cell-resolved simulations topped out at roughly a three-millimeter cube. Exascale machines like Aurora and Frontier have pushed that to about a one-centimeter cube with several billion cells, up from around 500 million, said Amanda Randles of Duke University. But simulating several centimeters by brute force remains out of reach even on today’s leadership-class systems. 

Amanda Randles and her team at Duke University set out to solve this problem by creating a physics-based technique called Adaptive Physics Refinement (APR) that can capture a much larger stretch of vessel by splitting the domain into a high-resolution ‘moving window’ and a low-resolution bulk fluid.  

“If we want to understand how a cancer cell moves across a couple of centimeters, no computer today could do that. We needed a new approach,” said Randles, director of the Duke Center for Computational and Digital Health Innovation and Professor of Biomedical Engineering at Duke University.  

An APR simulation tracks a modeled cancer cell as it moves through a blood vessel. Near the cancer cell, the orientation of the red blood cells can affect its trajectory, but farther away this resolution is unnecessary. APR can use less computational power by modeling a small window around the cancer cell in high resolution and leaving the bulk fluid around it at a lower resolution.  

“Since what we care about is where the cancer cell is moving and how it’s interacting, and we’re focused on a single cancer cell of interest, the argument is that while red blood cell orientation matters right next to the cancer cell, two centimeters away we just need to know that the blood is viscous. The red blood cells still matter, but their exact orientation doesn’t,” Randles said.  

Inside a window that moves with the cancer cell, APR resolves red blood cells individually, work suited to GPUs, which handle many small parallel calculations at once. Outside the window, the framework treats blood as a continuous viscous fluid across a much larger domain, which is handled by CPUs. “Memory-intensive tasks can be assigned to CPUs, while highly parallel cell-scale calculations are accelerated on GPUs,” Randles wrote in an article. That split lets the expensive, detailed calculations run only where the cancer cell actually is. 

Modeling adhesion to the vessel wall 

Early APR versions simulated cancer cells floating freely in the channel, never touching the vessel wall, Randles said. For oncology, though, interactions between the cell and where it adheres to the wall are essential to understanding how it becomes a metastatic site.  

Once the moving window could reach and interact with the vessel wall, the team added individual ligand-receptor pairs, allowing the simulation to show how and where a cancer cell grips the wall.  

“It’s especially useful because resolving those ligand-receptor interactions requires an even higher resolution, down to the nanometer scale. With this multiscale approach, you only need that super-high resolution for adhesion when you’re inside the window, not outside it,” Randles said.  

The bulk fluid model captures how fluid drags against the vessel walls across the entire simulation, not just in the moving window. The endothelial cells lining the blood vessel respond to this shear force by changing which receptors they express. The model can then estimate what receptor expression looks like in specific regions based on local shear stress.  

When the cancer cell moves near a section of the wall, the model refines the coarse estimate using the local fluid behavior. 

“That gives you a much more dynamic system: how the endothelial cells are responding to the fluid, how that changes their behavior, and how that in turn drives whether the cancer cell can adhere there,” Randles said.  

Why one simulation isn’t enough 

One benefit of APR using less computational power is that it allows a researcher to run multiple simulations. Randles’ team has shown that a cancer cell can travel on a very different path due to the orientation of the red blood cells around it.  

“It would be misleading to pick one random packing of red blood cells, see how the cancer cell transited, and conclude that’s how it always goes. You need many replicates,” Randles said.  

“I’d argue it matters even more for cancer, because it’s the outlier that causes the problem, not the mean or the average. So, knowing what the outlier looks like is even more important, which means you need the full distribution,” she added.   

The team has been researching how many replicates are necessary and how to initialize the cells.  

Validating the jump between resolutions 

In a multiscale simulation, stitching together two regions computed at very different resolutions can cause the numerical coupling between the regions to become unstable if the jump in resolution is too large.  

To avoid this, the team ran convergence tests to make sure they were using the right resolutions for each region.  

“A lot of what we’ve done is run convergence tests in the bulk fluid region to make sure we’re using the right resolution there, one we’d trust for the geometry we’re looking at. Then we move to the smaller window. We’re not starting with the window and asking whether we can get, say, 10x resolution in the bulk fluid; we go in the other direction; running the standard convergence tests in the bulk fluid model first,” Randles said. 

Applying APR to drug discovery  

The team has used APR to simulate microfluidic devices. The team can build a digital twin of the device and match the properties that the real device can measure; the model can help fill in features the device can’t directly measure.  

“We can change the properties of the cell in the model and figure out what it had to look like to match their experiment, essentially augmenting their experiments,” Randles said.  

APR could be used to inform drug companies working in oncology. 

“I think we’ll see more near-term use in augmenting a complete blood count panel or metabolic panel, understanding what it is about circulating tumor cells, small cell lung cancer cells, for example, that causes them to marginate, adhere more, or be more metastatic. So, it may not be predicting a specific site in your body, but understanding those cells lets us tell drug companies: it’s really the deformability that’s the problem, or the nucleus, or the receptors being expressed, giving them a therapeutic target,” Randles said.