Synthetic Vision systems -- from sandbox to reality
| By |
|
Certification of Synthetic Vision (SV) systems must take into account the complexities of the underlying software used to create them, particularly 3D graphics software tools and libraries. Common software techniques used to prototype these systems on desktop computers are often fraught with certification risk, which can result in significant rework activities for the embedded, certifiable software. There are methods by which synthetic visions can be developed and certified that avoid these pitfalls, if the entire development life cycle of the system is taken into account up front.
Synthetic Vision systems are the subject of intensive ongoing research and development. Their promise lies in their ability to fuse three-dimensional (3D) data into intuitive displays that can provide lifesaving awareness to flight crews.
There exists a large gap between the desktop prototyping environment and the operational embedded systems environment, which has proven daunting to bridge. This gap is in three primary areas: hardware, data, and software. Advanced embedded graphics hardware subsystems are maturing, based on industry standards like OpenGL and desktop graphics capabilities from companies like NVIDIA and ATI migrating to embedded environments. These systems bring with them the necessary throughput and performance to close the hardware performance gap.
[ad]The data gap is closing in many ways, with commercial data providers such as Honeywell and Jeppesen offering certified data sets. But the software gap is most challenging, because it presents not just an engineering problem but a process, tools, and certification challenge as well. Simply put, most SV research efforts focus on the capability and the symbology, and pay too little attention to the end goal of operational embedded and DO-178B certifiable software. Desktop software, even attractive SV display software, is easy to write, but embedded certified software is not. Just pointing to a prototype as a “requirement” and expecting the embedded guys to “go do it” will not make it happen.
Simply focusing on certifying graphics drivers is not enough. A cultural change in how the research/prototyping industry does business is required. Mark challenges the SV R&D community to pay attention to how SV software is developed and to do business in a way that will lead to their innovative concepts becoming available as certifiable embedded systems.
Background Synthetic Vision as applied to the avionics display area, is a term used to describe the creation of a computer-generated 3D representation of the environment an aircraft is operating in. Typically, these environments are presented as 3D display symbology intended to portray a “synthetic” view that the crew can relate to the actual visual view of the environment. The symbology is typically composed of graphics to represent the terrain and additional symbology intended to represent flight paths, other hazards, automation cues, air data, runways, and so on. The graphics can range from line-based symbols to multicolored textured “virtual” views.
A stored terrain database is usually a key component of SV display systems. The stored terrain data, which comprises elevation information, can also include satellite imagery, culture data, feature data, vector data, and potential obstacles. The stored terrain data may also be fused with various sensor data such as IR or terrain-following radar to improve the accuracy. Database correction algorithms may also be used to augment the accuracy of the stored data.
Another key component of SV displays is symbology intended to aid the crew in following the intended flight path or monitoring aircraft performance relative to the intended path. Such symbology is commonly referred to as Highway In The Sky (HITS), but such symbology can contain many elements other than a virtual “highway.” Such HITS symbology is also typically presented in a 3D display format.
Synthetic vision displays can be viewed from different perspectives, and can be presented on the Primary Flight Display (PFD) or on Multifunction Displays (MFDs). The display the synthetic information is presented on can affect the resulting certification level of the underlying software: PFDs typically must be certified to a higher level than MFDs since they present flight-critical air data required for safe IFR flight to the crew. Figure 1 and Figure 2 show some synthetic vision display presentations.
There is a vibrant and active research community engaged in the development of SV systems. Many concepts have been refined and proven to aid flight safety and situational awareness. In essence, the research and development community has been engaged in a collective form of Simulation-Based-Development (SBD) for many years with SV systems. The challenge for this technology lies in several areas. Perhaps the greatest one lies in the area of transitioning the software to the certified operational environment. This has proven difficult to achieve. One reason lies in the nature of developing advanced graphical software such as SV displays atop powerful graphics accelerators.
OpenGL-based embedded graphics systems The aviation engineering community has recently started incorporating COTS graphics hardware technology into its glass cockpit design. Today’s COTS graphics technologies are raster based. They are often based on commercial graphics chipsets borrowed from the desktop and mobile computing world. They have floating-point accuracy, high display resolution, 32-bit color for an almost infinite palette, and internal hardware support for 2D and 3D matrix manipulation. They provide for advanced graphics features such as smooth shading (the smooth transition from one color to another), stenciling (dynamic masking of symbology), texture mapping, lighting, transparency, and depth testing (for 3D applications). The speed of these graphics systems provides customers with more flexibility in terms of the amount of data displayed and the image quality of the graphics.
Along with these systems has come widespread use of new standards designed to facilitate development of graphical applications that take advantage of the hardware. One such low-level Applications Programming Interface (API) is OpenGL. OpenGL provides a software interface that supports 2D and 3D definition of geometry and rendering functions. Some of the major functions OpenGL supports include:
- Matrix-based 2D and 3D geometry transformations
- Viewport and clipping regions
- Textured geometry
- Graphics pipeline state management
- Geometry caching through display lists and vertex arrays
OpenGL is a standardized API that has evolved over many years, initially through the efforts of graphics industry pioneer Silicon Graphics. It has achieved widespread adoption in the simulation, gaming, CAD, and professional graphics markets, and is widely available on many platforms. OpenGL is also meant to provide a standard interface to multiple graphics rendering devices, allowing an application to run with confidence on graphics chips from multiple vendors. It is also a key standard in the embedded avionics market because of its power and cross-platform nature.
Although OpenGL is a standard, demands of ever-increasing performance and features in the visual application market have strained its ability to be truly platform- and device independent. Graphics chips are continuously adding capability, such as pixel and vertex shader operations on-chip, that are supported through vendor-specific extensions to OpenGL. Additionally, several languages intended to supply special programming instructions to the chips (shader languages) complicate the picture further.
Scene management software systems While OpenGL is a standard API, most complex 3D applications are too complex to implement atop this API directly. To implement a 3D view with many objects displayed in it, scene manager libraries are commonly used. With an OpenGL-based scene manager, the software developer is offered a set of higher-level constructs with which to build the 3D application.[5] Scene managers typically offer the following types of capability to a software developer:
- Geometry loader software to read geometry files, textures, and terrain data output from modeling packages
- Constructs to manage matrix manipulations to provide camera, object, and scene level movement and manipulation in various coordinate frames, including culling unseen geometry
- Math routines to assist in 3D calculations
- Functions to provide intersection and collision detection
- Functions to control effects such as lighting and fog
- Functions to manage the presentation of terrain or gaming areas
Scene managers (Figure 3) exist to provide an object-oriented method to build 3D applications, hence, making their creating simpler. They also exist to abstract optimizations to the rendering process so high-performance applications can be easily built. There are quite a few commercial and open source scene manager APIs that provide mature programming interfaces and high performance. Most of these take the form of complete application frameworks, offering a simple way for developers to create complete graphical applications.
Most scene manager APIs are implemented using object-oriented techniques. Many are implemented as class libraries using object-oriented languages such as C++. A key factor in their design is that they typically assume a desktop or workstation operating environment, including system facilities such as windowing libraries, virtual memory, file I/O, and often multiprocessing. While making desktop applications such as games and visual simulations possible, these underlying assumptions are not always compatible with the embedded computing environment.
Embedded vs. desktop software development Many engineers begin their programming career working with UNIX, Microsoft Windows, Linux, or some other commercially available operating system. These operating systems lend themselves well to a wide range of applications programming paradigms. They run on high-speed processors, allocate considerable amounts of memory to each process, and typically have some form of virtual memory scheme granting almost limitless resource usage. These operating systems are updated yearly – if not more often – by their creators or by the Open Source community, providing optimizations for the latest technologies and programming paradigms and provide an environment of flexibility.
By comparison, the embedded applications programming environment is one of rigidity: a specific operating system supporting a specific set of functionality. The engineer may be working with a product that is decades old – and, most likely, the operating system is as old. The engineer does not have access to limitless resources: System resources are strictly assigned to processes monitored by the kernel and terminated if their resource consumption exceeds their allocation. The engineer must weigh the advantages and disadvantages of any programming paradigm when designing an application in this environment.
Some programming practices that are desirable when writing software for commercial operating systems are undesirable in embedded systems software. The following details some techniques that should be avoided and the reasons.
- Recursion – A recursive function is one that calls itself. Recursion is commonly used for traversing hierarchical data structures. Recursion can cause excessive stack overhead and usage, which can have unforeseen effects in a resource-constrained embedded environment.
- Dynamic memory allocation – Allocating memory from the heap using new or malloc is a practice that is taken for granted on commercial operating systems and is almost always avoided on embedded systems. There must be straightforward ways to determine the total resource requirement for a process. Many safety-critical operating systems do not provide a heap for this reason. Memory management schemes that provide similar capabilities can be implemented, but must be used with care.
- Large class-based systems – These are the norm for a new generation of programmers, but the object-oriented paradigm is not well-suited to many embedded environments. Using instances of class objects for commonly used data constructs is very helpful for encapsulation and protection. However, large class systems including inheritance and virtual functions are typically inefficient, difficult to maintain, and difficult to debug. More of the stack is required for each function call, more processor time is taken to call functions due to virtual function pointer table access, and construction time problems as described in the preceding paragraph are ever present.
- Overuse of threads – Embedded systems often require hard real-time scheduling of their processor time. A design heavily dependent on threads, locks (semaphore, mutex), and other synchronization may be extremely difficult to adapt to this type of operating environment.
Most of these programming practices that tend to make embedded software development problematic are commonly and even extensively used in modern scene management APIs. When safety-critical standards such as DO-178B come into play, the added requirements for testing, structural coverage, design documentation, and complete predictability make such practices even more problematic. Desktop programming techniques can certainly save time and allow for a larger pool of engineers to contribute, but they are not without their drawbacks when it comes to creating the eventual embedded certified system.
Recommendations The gulf between embedded and desktop software environments is real. It is particularly difficult to bridge in the area of advanced graphics systems such as SV display software, but it is not impossible to do so. There are several steps the R&D community and avionics industry can undertake to help bridge this gap.
Software architecture Development of any software intended for a certified embedded system must rest on a thorough understanding of the complete software architecture underlying the application. Constraints must be identified up front to avoid expensive rework later. Some areas for scrutiny include:
- Memory management – Will the intended environment support dynamic memory allocation of some form? If so, is there a custom API to replace new and delete? Does the implementation support freeing allocated memory areas, or must allocation be performed up front? The memory usage profile for the application must be developed up front and matched to the available facilities, or appropriate libraries must be developed.
- Data structures – Which data structures or libraries are assumed? Does the application use Standard Template Libraries (STLs) or other data structure libraries that may not be available? Do such libraries perform dynamic memory allocation or recursion?
- Language – Is the target C, or C++, or another language? Usage of such common environments as Java or C# may require unimplemented or uncertifiable interpreters on the target system.
- Threading – Does the system make use of multiple threads to perform such tasks as visual database paging? Can this be accommodated in the target environment?
- System facilities – Does the system use windowing or GUI facilities that are not available on the target? Does it assume file I/O when it may not be available?
An excellent way to address the software architecture is by using a constrained embedded system during early prototype design and subsequent system development. If this is unworkable, an alternative is to identify and partition all operating system dependencies using an operating system abstraction layer in the software that makes actual system calls. Coding standards can be used to ensure developers follow the proper rules and use only software constructs that are portable.
While this may seem a large challenge, with proper attention to process and some basic software reuse, it can be achieved. But a larger problem is the extent of development required to create an effective, high-performance scene manager for embedded SV applications. This is a highly specialized development task requiring graphics expertise, real-time performance savvy, and an effective design. Tool-based development and reusable certifiable components are two key areas where industry must assist.
Tools and reusable software components Well-enforced tool usage can mitigate many of the factors that make SBD difficult to apply to complex engineering environments like avionics. This has been shown to be true for the development of traditional EFIS displays, where tools that support specifying the display’s graphical appearance and behavior and automatically deploying the resulting embedded representation through code generation are used, as shown in Figure 4.
While this method has been shown to save time and address certification issues, it tends to be limited in its usefulness for SV displays. SV displays contain dynamically changing 3D geometry and can also use large geometric databases imported from 3D modeling tools. The industries that use scene management technology the most, such as video games and visual simulation, do not use code generation to represent 3D models. They instead concentrate on embedding efficient rendering algorithms for 3D geometry loaded in from a file store. They use standard file formats that represent 3D geometry, such as Multigen OpenFlight or 3D Studio Max, as the means by which they import 3D data.[2] They concentrate on compatibility with such standards in the scene management software. Figure 5 illustrates this process.
To best address SV display needs, the tools used must match the runtime infrastructure provided for 3D rendering. To better align avionics SV display development with common scene manager software practices, the focus of tool usage should be on the data they produce, not on the code they generate.[3]
Code generation has largely been used to address certification issues by replacing errorprone hand code with automatically generated code.[1] Scene managers solve the problem a different way, be replacing hand-coded 3D graphics with tool supported standard rendering algorithms that are packaged into a highly reusable API. To address certification and still offer this reuse, scene manager must be developed in a way that provides for certification of the scene manager APIs. Emerging FAA guidance on reusable software components (RSC) provides a way to do this.
The FAA is in the process of offering new guidance on how software vendors and the avionics development community can work together to provide reusable software. The new AC20.RSC guidance offers some key ways this can happen. AC20.RSC is a new FAA guidance document (currently in draft, but close to final) that specifies how a developer of reusable software (RSC developer) can transfer acceptance data across company boundaries to provide an RSC with acceptance data to the market.
For SV display development to be less prohibitive and easier, a process such as that offered by AC20.RSC is a key step. Using an approach such as this, a standards-based RSC providing the right scene management capabilities to the market would act as an enabler of SV technology. This library would make minimal use of desktop programming practices to enable its use in certified, embedded, resource-limited systems. It would provide scene management capabilities and would provide the means for real-world terrain display, as well as the special-purpose 3D symbology common in SV displays. Flexibility of use, enabling avionics developers to insert their own custom content and proprietary IP, would be another key.
To enable such development, the SV development community has means as its disposal. For instance, SBIR or BAA contracts or targeted development programs aimed at supporting the market with these capabilities could be undertaken. Leveraging developments designed for resource limited systems, such as cell phone game engines, might be possible.
[ad]API standardization API standardization is one area where the SV development community could help its technology have a better chance of eventual transfer to certifiable embedded systems. There are many ways to render 3D graphics, and many powerful techniques that can be used. OpenGL, for instance, is one API that is achieving acceptance in the avionics software community. Another commonly used API is Direct3D, common in the PC gaming arena. Even within a standard API, there are common vendor-specific extensions that enable developers to unlock powerful features on modern graphics cards.
While OpenGL is a good standard for avionics and should be used, it has grown large as an API. Keep in mind that it is usually prohibitive to certify an entire large library like OpenGL. There are many functions that may never be used in an avionics environment. The typical subset of OpenGL that an avionics or even an SV display will use is small compared to the entire offering. Standardized subsets provide a key to using OpenGL in the certified avionics environment.
The OpenGL ES standard by the Khronos Group is an important development in standardized special-purpose subsets of OpenGL.[4] Khronos is an industry consortium designed to foster the adoption of OpenGL into embedded and multimedia markets. OpenGL ES is a well-defined subset of OpenGL that is designed to provide a capable subset for advanced graphics on demanding embedded platforms, including mobile devices such as PDAs. Since the full OpenGL specification is large and graphics subsystems that support the full spec are resource intensive, a well-defined subset is required to provide a target rendering capability for embedded applications. OpenGL ES is that subset.
OpenGL ES has multiple well-defined profiles that are designed to support different usages of OpenGL. Two that are of key importance are the common and safety-critical profiles, shown in Figure 6 below. The safety-critical profile supports a subset of OpenGL oriented primarily toward traditional avionics. The common profile is oriented toward lightweight consumer devices, such as PDAs, set-top boxes, cellular phones, and so on.
Whichever OpenGL subset is chosen for a display development project, it is very important to target the development to that subset from the start. This means that if a scene manager is used, its API usage should be known up front.
Summing up the strategy We have discussed some of the issues that hold back SV display technology from the certified marketplace. SV displays commonly use scene-management systems to manage the display of 3D data. They rely on advanced graphics capabilities facilitated by the use of standard graphics APIs like OpenGL. Their application design often makes heavy use of common desktop programming practices difficult to support in a certified embedded system.
There is no substitute for understanding the intended embedded environment when designing software to be embedded. The common practice of developing for the desktop or workstation environment and then hoping to port the application to the embedded system is a risky strategy. A better strategy is to identify the tools, APIs, and limitations up front.
An industry and government effort to foster reusable SV technology in the form of scene manager libraries and suitable tools would be a desirable step in making SV technology a reality. An effort to foster the use of standards like OpenGL ES would also help.
