This article looks at the impact of high performance computing on the geological interpretation stage of the oil and gas exploration workflow. In this article, we look at how the Intel Parallel Computing Center (Intel PCC) at the SCI Institute, University of Utah performs geological interpretation by processing LiDAR databases using supercomputers and HPC tools for visualizing large scanned surface data, which can subsequently be used in models of oil and gas reservoirs.
Visualizing entire LiDAR databases with OSPRay
The Intel Parallel Computing Center (Intel PCC) at the SCI Institute, University of Utah performs geological interpretation of outcrops aimed at reducing risk for oil and gas exploration using a novel memory-efficient technique in the OSPRay6 framework. This allows them to visualize entire LiDAR database files directly on large-memory HPC workstations or supercomputers. This project fits within the open source “software-defined visualization” initiative, aimed at providing efficient CPU and many-core solutions for big data visualization problems. The approach differs from typical visualization workflows, which commonly convert large data sources into smaller meshes that can be rendered on the GPU. In the Software Defined Visualization model, the large memory of the Intel Xeon processor or Intel Xeon Phi coprocessor enables direct visualization of the full data, without simplification. This is made possible thanks to the engineering of the OSPRay tracing engine, and novel memory-efficient techniques for simultaneously storing and querying large LiDAR data.
Several different projects at the SCI Institute led to this research. Dr. Aaron Knoll is a Research Scientist at the SCI Institute, leading the Intel PCC at the University of Utah. Knoll is part of Dr. Valerio Pascucci’s Center of Extreme Data Management, Analysis and Visualization (CEDMAV), which in 2014 collaborated with Dr. Lisa Stright, now an Assistant Professor in the Deptartment of Geosciences, Warner College of Natural Resources, Colorado State University. Stright works with oil and gas companies to extract analog information from both aerial and ground LiDAR and 2-D outcrop imagery to apply to subsurface modeling and prediction. The traditional method of analyzing these data to convert point samples into a digital terrain/elevation model (DEM) mesh. However, this process is time-consuming and can result in loss of quality. Stright and Pascucci sought a way to visualize and analyze DEM data progressively in SCI Institute’s ViSUS framework.1 Knoll and a graduate student, Will Usher, were interested in methods for directly rendering DEM surfaces from LiDAR data, first as a stand-alone solution and, subsequently, as part of a larger out-of-core IO framework, such as ViSUS.
Data used in the project
Stright provided LiDAR and outcrop data to the SCI Institute for this outcrop modeling test. Stright states, “Data acquired from outcrop analogs can be used to better understand and interpret reservoir-to-basin scale geology and can be fed into modeling workflows to build more predictive subsurface reservoir models. We have been attempting to use large (billion-point) LiDAR scans to interpret sandbody sizes, shapes and stacking patterns. The advantage is that LiDAR data can help to provide correct three-dimensional referencing of the sandbodies. However, in practice, terrestrial LiDAR data are hindered by a need to process the data into a DEM that is not only inaccurate, but cannot handle the complex multi-z surfaces to represent existing outcropping rock formations. Furthermore, dealing with the raw point cloud presents visualization challenges that hinder interpretation at multiple scales.” Initially, the Intel PCC SCI team aimed to use its ViSUS software for progressive multiresolution analysis of LiDAR data, and OSPRay as a rendering engine. This pipeline would scale to a wide range of devices and eliminate the need for offline mesh computation.
As part of an ongoing project with Argonne National Laboratory, Knoll was already using OSPRay to render large particle data from materials science simulations, visualizing on systems using Intel Xeon processors and Intel Xeon Phi coprocessors. Applying these techniques to large LiDAR point data was a natural extension. The SCI team integrated the popular LASZip7 tools into OSPRay, enabling reading of large compressed LiDAR data. This allowed visualization of Stright’s ground LiDAR data, as well as even larger aerial LiDAR from full topographical databases. It was quickly apparent that, with the right hardware using OSPRay’s ray tracing algorithms, both small and large data could be rendered at similar rates.
Challenges encountered in the project
OSPRay’s state-of-the-art bounding volume hierachy (BVH) acceleration structure, based on Intel Embree8 allowed point data including LiDAR datasets, to be rendered efficiently on commodity workstation hardware. However, these structures entailed a high (up to 5x) memory overhead. In the short term, this suggested the use of large-memory hardware, such as the Intel C602J Chipset (formerly known as “Brickland”). It also motivated development of a new data structure in OSPRay, the balanced “P-k-d” (point k-d) tree,1 which enabled visualization of LiDAR point sets with zero overhead. In other applications requiring classification of point data with attributes, the P-k-d tree provides mechanisms for efficient query by value with minimal overhead. With this approach, the rendering and querying structure are the database consisting of raw re-ordered LiDAR data. Most importantly, the method still provides the logarithmic computational complexity of the ray tracing, enabling small and large data to be visualized at comparable frame rates.
Reconstructing smooth surfaces from ground-based LiDAR data without meshing proved more difficult. The data from Stright’s work exhibited significant striation due to the nature of its acquisition. Displaying these data as a continuous surface, with the same fidelity as a mesh processed via DEM, remains an active area of research in both the GIS and more general computer vision communities.
Visualizing LiDAR databases directly with OSPRay
Though meshless visualization of ground LiDAR proved challenging, the Intel PCC SCI team found that OSPRay is an excellent solution for visualizing massive aerial LiDAR from geospatial databases. Airborne LiDAR data are commonly stored on HPC and cloud resources, such as the OpenTopograpy server at SDSC.3 Typically, a user would select small subsets of LiDAR data, and process them in various GIS software such as ArcGIS, ENVI or Petrel. However, given a resource with sufficient memory, OSPRay and the P-k-d method can directly render these data on the cloud. Ray tracing allows both small and large data to be visualized interactively. This stands in contrast to rasterization techniques, which cost on the order of the number points rendered. Using Intel OSPRay in place of the standard OpenGL solutions, the Utah group was able to visualize billions of LiDAR samples directly from raw source data, without needing to generate any intermediate mesh, with system memory the only limiting factor.
Supercomputers used in SCI-University of Utah reservoir modeling research
The research in OSPRay at the Intel PCC SCI Institute was carried out using Intel Xeon Phi coprocessors code name Knights Landing, Intel Xeon processors E7-8890 v3 and various multi-socket IA server platforms supporting up to 6 TB RAM. With technologies such as these, Knoll foresees a future in which GIS data could reside on a single cluster, which could serve as a HPC, visualization and data analysis resource.
In contrast to existing terrain acquisition solutions, with large-memory Intel Xeon processors or Intel Xeon Phi coprocessor hardware, one can visualize entire LiDAR databases from a single system. This approach has the potential to dramatically change data workflows in GIS software such as ArcGIS, as well as common oil and gas software packages such as Petrel,” states Aaron Knoll, Research Scientist at the Intel PCC SCI Institute, University of Utah.
HPC software used to visualize LiDAR data
This research was carried out using the OSPRay6 ray tracer as a platform for large-scale visualization. OSPRay uses the Intel SPMD (Single Program, Multiple Data) program compiler (ISPC)9 to provide efficient cross-platform codepaths on Intel Xeon processor and Intel Xeon Phi coprocessor architectures. According to Knoll, “SCI’s research in particular has focused on massive unstructured particle data from UAV or aerial LiDAR sources. Our p-k-d data structure allows for data to be queried and visualized directly, with no additional acceleration structure or memory overhead.” In particular, being based on OSPRay and ISPC, all code developed in this research will work on both Intel Xeon processors and Intel Xeon Phi coprocessors, and will automatically use both threading and proper vector extensions of whatever processor it is running on. Intel Xeon Phi coprocessors can be used in either native or offload mode. While these algorithms are currently designed for a single node, both SCI and the Intel OSPRay team are researching solutions for distributed-parallel visualization,10 allowing use of multiple nodes on a supercomputer in parallel.
SCI Institute OSPRay visualization used in various projects
OSPRay has been used to visualize particle data from several environmental impact studies in the Utah computational framework, such as the Uintah project simulating accidental detonation of flammable materials including transported petrochemicals (PI: Todd Harman, University of Utah);4 and the PSAAP project involving carbon capture from coal and potentially gas power plants (PI: Phil Smith, ICSE, University of Utah).5 In a separate project at the Intel PCC SCI Institute, Knoll had collaborated with Dr. Ross Whittaker and Dr. Shankar Sastry in analytical modeling of subsurface (fault horizon) data, using the OSPRay framework for rendering.
Recommendations for future work in oil and gas exploration
“We believe that the main challenge for future work in oil and gas exploration lies in adapting oil and gas workflows (for example, in ArcGIS or Petrel) to better leverage scalable visualization with large-memory solutions, such as OSPRay. This would enable direct analysis of large data from in situ simulation or live acquisition sources,” states Knoll. In contrast to the static analysis pipelines employed today, future GIS and visualization software could use a combination of dynamic acquisition and query methods, and direct visualization using both ray tracing and scalable IO methodologies, in software such as OSPRay and ViSUS. Ongoing work at the Intel PCC – SCI Institute at the University of Utah seeks to merge these large in-core and large out-of-core approaches. The innovative research being done using supercomputers and HPC software has the potential for financial benefits for oil and gas exploration. Implementing these techniques can help lead to more insights, and provide information that helps find more oil and gas reservoirs and higher quality reservoirs.
References
- Ingo Wald, Aaron Knoll, Gregory P. Johnson, Will Usher, Valerio Pascucci and Michael E. Papka. CPU Ray Tracing Large Particle Data with Balanced P-k-d Trees; IEEE Visweek 2015 (accepted for publication).
- Valerio Pascucci, Giorgio Scorzelli, Brian Summa, Peer-Timo Bremer, Attila Gyulassy, Cameron Christensen, Sujin Philip, and Sidharth Kumar. The ViSUS visualization framework. In E. W. Bethel, H. C. (LBNL), and C. H. (UofU), editors, High Performance Visualization: Enabling Extreme- Scale Scientific Insight, Chapman and Hall/CRC Computational Science, chapter 19. Chapman and Hall/CRC, 2012.
- SDSC OpenTopography portal. http://opentopography.org?
- Uintah computational framework. http://uintah.utah.edu?
- Institute for Clean and Secure Energy. http://www.icse.utah.edu?
- Intel OSPRay. http://ospray.github.io?
- LasZIp tools. http://www.laszip.org?
- Intel Embree. http://embree.github.io?
- Intel SPMD Program Compiler, http://ispc.github.io?
- A.V. Pascal Grosset, Manasa Prasad, Cameron Christensen, Aaron Knoll, and Charles Hansen. TOD-Tree: Task-Overlapped Direct send Tree Image Compositing for Hybrid MPI Parallelism. Eurographics Symposium on Parallel Graphics and Visualization (EGPGV), 2015.
Linda Barney is the founder and owner of Barney and Associates, a technical/marketing writing, training and web design firm in Beaverton, OR.
