////
- Copyright (c) 2021-2023, Arm Limited and Contributors
-
- SPDX-License-Identifier: Apache-2.0
-
- Licensed under the Apache License, Version 2.0 the "License";
- you may not use this file except in compliance with the License.
- You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
////
= OpenCL interoperability
ifdef::site-gen-antora[]
TIP: The source for this sample can be found in the https://github.com/KhronosGroup/Vulkan-Samples/tree/main/samples/extensions/open_cl_interop_arm[Khronos Vulkan samples github repository].
endif::[]
== Overview
In certain scenarios OpenCL is used for compute, while another API is used for graphics, and interoperability between two APIs becomes important in this case.
For example, in AR applications graphics rendering is often combined with machine learning workloads, which can be executed using OpenCL.
In such cases we need zero-copy data sharing and efficient workload synchronization to achieve the best performance.
While there are no dedicated extensions for direct Vulkan - OpenCL interoperability at the moment, both APIs provide generic sharing mechanism that can be used to achieve it.
This sample demonstrates an approach for zero-copy data sharing using Android Hardware Buffers and corresponding extensions for Vulkan and OpenCL.
== Data sharing
Zero-copy assumes that both APIs use the same region of memory for an image or a buffer.
In case of OpenCL and Vulkan we can use extensions to share data using:
* Android Hardware Buffers
* dma_buf
* Host memory
This sample covers one of these options, which is more relevant to mobile developers working with Vulkan: Android Hardware Buffers.
== Android Hardware Buffers
Native hardware buffers on Android represent a region of memory which can be bound to Vulkan, OpenGL ES or OpenCL primitives.
This allows us to reuse it with two different APIs:
image::./images/shared_hardware_buffer.png[Sample]
=== Support by Vulkan API
In Vulkan hardware buffers can be imported or exported using https://www.khronos.org/registry/vulkan/specs/1.2-extensions/man/html/VK_ANDROID_external_memory_android_hardware_buffer.html[VK_ANDROID_external_memory_android_hardware_buffer] extension.
AHardwareBuffer can be bound to a `VkDeviceMemory` object, which serves as an allocation for objects of type `VkImage` or `VkBuffer`.
If we need to use an image backed by an AHardwareBuffer, we must specify while creating the image.
It is done by assigning a pointer to the following structure to `pNext` field of `VkImageCreateInfo`:
----
VkExternalMemoryImageCreateInfo external_memory_image_create_info;
external_memory_image_create_info.sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO;
external_memory_image_create_info.pNext = nullptr;
external_memory_image_create_info.handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID;
----
This image (or buffer) must be specified as target during memory allocation.
Below you can see, how the image object which is supposed to be shared with OpenCL is assigned to the `image` field of `VkMemoryDedicatedAllocateInfo`, while `buffer` field is assigned with `VK_NULL_HANDLE`:
----
VkMemoryDedicatedAllocateInfo dedicated_allocate_info;
dedicated_allocate_info.sType = VK_STRUCTURE_TYPE_MEMORY_DEDICATED_ALLOCATE_INFO;
dedicated_allocate_info.pNext = nullptr;
dedicated_allocate_info.buffer = VK_NULL_HANDLE;
dedicated_allocate_info.image = shared_image;
----
==== Export from Vulkan API
A pointer to `dedicated_allocate_info` is provided as `pNext` of `VkExportMemoryAllocateInfo`.
----
VkExportMemoryAllocateInfo export_memory_allocate_Info;
export_memory_allocate_Info.sType = VK_STRUCTURE_TYPE_EXPORT_MEMORY_ALLOCATE_INFO;
export_memory_allocate_Info.pNext = &dedicated_allocate_info;
export_memory_allocate_Info.handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_ANDROID_HARDWARE_BUFFER_BIT_ANDROID;
----
This is specified as `pNext` of `VkMemoryAllocateInfo`, while `allocationSize` is set to 0.
You can find more info on this in the https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#VkMemoryRequirements[specification].
Once the memory is allocated *and* bound to the image (or buffer), we can export a handle of type `AHardwareBuffer`:
----
VkMemoryGetAndroidHardwareBufferInfoANDROID info;
info.sType = VK_STRUCTURE_TYPE_MEMORY_GET_ANDROID_HARDWARE_BUFFER_INFO_ANDROID;
info.pNext = nullptr;
info.memory = shared_memory;
vkGetMemoryAndroidHardwareBufferANDROID(device, &info, &hardware_buffer);
----
==== Import to Vulkan API
In this case AHardwareBuffer is created first and then used for `VkDeviceMemory` allocation.
NOTE: there are certain restrictions on image or buffer format and usage.
You can find the list of Android Hardware Buffer formats and their Vulkan counterparts in this https://developer.android.com/ndk/reference/group/a-hardware-buffer#ahardwarebuffer_format[table].
A pointer to `dedicated_allocate_info` is provided as `pNext` of `VkImportAndroidHardwareBufferInfoANDROID`.
Also the hardware buffer we want to import must be provided during this stage:
----
VkImportAndroidHardwareBufferInfoANDROID import_info;
import_info.sType = VK_STRUCTURE_TYPE_IMPORT_ANDROID_HARDWARE_BUFFER_INFO_ANDROID;
import_info.pNext = &dedicated_allocate_info;
import_info.buffer = hardware_buffer;
----
We need to specify this data as `pNext` in `VkMemoryAllocateInfo`.
In case of import, `allocationSize` and `memoryTypeIndex` should be based on the results of `vkGetAndroidHardwareBufferPropertiesANDROID`:
----
VkAndroidHardwareBufferPropertiesANDROID buffer_properties;
buffer_properties.sType = VK_STRUCTURE_TYPE_ANDROID_HARDWARE_BUFFER_PROPERTIES_ANDROID;
buffer_properties.pNext = nullptr;
vkGetAndroidHardwareBufferPropertiesANDROID(device, hardware_buffer, &buffer_properties);
----
Then the memory is allocated and bound to the image as usual.
=== Support by OpenCL
On Arm devices Android hardware buffers (as well as dma_buf and host memory) can be imported to OpenCL using https://www.khronos.org/registry/OpenCL/extensions/arm/cl_arm_import_memory.txt[cl_arm_import_memory] extension.
`cl_mem` object is retrieved in the following way:
----
const cl_import_properties_arm import_properties[3] = {
CL_IMPORT_TYPE_ARM, CL_IMPORT_TYPE_ANDROID_HARDWARE_BUFFER_ARM,
0};
cl_int result = CL_SUCCESS;
cl_mem shared_cl_mem = clImportMemoryARM(context,
CL_MEM_READ_WRITE,
import_properties,
hardware_buffer,
CL_IMPORT_MEMORY_WHOLE_ALLOCATION_ARM,
&result);
----
In `cl_import_properties_arm` array the types and values of the properties are listed one by one with a 0 as a terminator in the end.
We need to specify the type of the imported object in this list (in our case it's `CL_IMPORT_TYPE_ANDROID_HARDWARE_BUFFER_ARM`).
== The sample
In this sample a simple OpenCL kernel is executed in the beginning of each frame to fill a `cl_mem` object backed by an `AHardwareBuffer` with a simple pattern (this pattern changes over time).
This `AHardwareBuffer` is exported from a `VkDeviceMemory` object which is bound to a texture.
After the OpenCL kernel is executed, the filled texture is displayed.
The texture is displayed as a quad using the approach and shaders from link:../../api/texture_loading[Texture Loading] sample.
The sample uses only one shared texture for simplicity, but in real applications it's worth having as many shared textures as there are framebuffers.
This way the OpenCL kernel would be executed for the current frame, while the previous frame is being rendered and displayed (double buffering).
Synchronization between APIs is performed on host:
* OpenCL queue is flushed before rendering
* A `VkFence` object is used to make sure the rendering is finished
image::./images/sample.png[Sample]
== Conclusion
Both OpenCL and Vulkan extensions allow us to import or export certain handle types, which represent a region of memory.
These memory regions can be used to achieve zero-copy data sharing.
In this sample one of such handle types is demonstrated: Android Hardware Buffers.
The extensions used in this case are:
* https://www.khronos.org/registry/vulkan/specs/1.2-extensions/man/html/VK_ANDROID_external_memory_android_hardware_buffer.html[VK_ANDROID_external_memory_android_hardware_buffer]
* https://www.khronos.org/registry/OpenCL/extensions/arm/cl_arm_import_memory.txt[cl_arm_import_memory]
