The BufferManager class handles host and device buffer allocation and deallocation. More...

#include <buffers.hpp>

Collaboration diagram for BufferManager:

Public Member Functions
	BufferManager (std::shared_ptr< nvinfer1::ICudaEngine > engine, std::vector< int64_t > const &volumes, int32_t batchSize=0)
	Create a BufferManager for handling buffer interactions with engine, when the I/O tensor volumes are provided.

	BufferManager (std::shared_ptr< nvinfer1::ICudaEngine > engine, int32_t const batchSize=0, nvinfer1::IExecutionContext const *context=nullptr)
	Create a BufferManager for handling buffer interactions with engine.

std::vector< void * > &	getDeviceBindings ()
	Returns a vector of device buffers that you can use directly as bindings for the execute and enqueue methods of IExecutionContext.

std::vector< void * > const &	getDeviceBindings () const
	Returns a vector of device buffers.

void *	getDeviceBuffer (std::string const &tensorName) const
	Returns the device buffer corresponding to tensorName. Returns nullptr if no such tensor can be found.

void *	getHostBuffer (std::string const &tensorName) const
	Returns the host buffer corresponding to tensorName. Returns nullptr if no such tensor can be found.

size_t	size (std::string const &tensorName) const
	Returns the size of the host and device buffers that correspond to tensorName. Returns kINVALID_SIZE_VALUE if no such tensor can be found.

template<typename T >
void	print (std::ostream &os, void *buf, size_t bufSize, size_t rowCount)
	Templated print function that dumps buffers of arbitrary type to std::ostream. rowCount parameter controls how many elements are on each line. A rowCount of 1 means that there is only 1 element on each line.

void	copyInputToDevice ()
	Copy the contents of input host buffers to input device buffers synchronously.

void	copyOutputToHost ()
	Copy the contents of output device buffers to output host buffers synchronously.

void	copyInputToDeviceAsync (cudaStream_t const &stream=0)
	Copy the contents of input host buffers to input device buffers asynchronously.

void	copyOutputToHostAsync (cudaStream_t const &stream=0)
	Copy the contents of output device buffers to output host buffers asynchronously.

	~BufferManager ()=default

Static Public Attributes
static const size_t	kINVALID_SIZE_VALUE = ~size_t(0)

Private Member Functions
void *	getBuffer (bool const isHost, std::string const &tensorName) const

bool	tenosrIsInput (const std::string &tensorName) const

void	memcpyBuffers (bool const copyInput, bool const deviceToHost, bool const async, cudaStream_t const &stream=0)

Private Attributes
std::shared_ptr< nvinfer1::ICudaEngine >	mEngine
	The pointer to the engine.

int	mBatchSize
	The batch size for legacy networks, 0 otherwise.

std::vector< std::unique_ptr< ManagedBuffer > >	mManagedBuffers
	The vector of pointers to managed buffers.

std::vector< void * >	mDeviceBindings
	The vector of device buffers needed for engine execution.

std::unordered_map< std::string, int32_t >	mNames
	The map of tensor name and index pairs.

Detailed Description

The BufferManager class handles host and device buffer allocation and deallocation.

This RAII class handles host and device buffer allocation and deallocation, memcpy between host and device buffers to aid with inference, and debugging dumps to validate inference. The BufferManager class is meant to be used to simplify buffer management and any interactions between buffers and the engine.

Definition at line 288 of file buffers.hpp.

Constructor & Destructor Documentation

◆ BufferManager() [1/2]

BufferManager::BufferManager	(	std::shared_ptr< nvinfer1::ICudaEngine >	engine,
		std::vector< int64_t > const &	volumes,
		int32_t	batchSize = `0`
	)

inline

Create a BufferManager for handling buffer interactions with engine, when the I/O tensor volumes are provided.

Definition at line 297 of file buffers.hpp.

◆ BufferManager() [2/2]

BufferManager::BufferManager	(	std::shared_ptr< nvinfer1::ICudaEngine >	engine,
		int32_t const	batchSize = `0`,
		nvinfer1::IExecutionContext const *	context = `nullptr`
	)

inline

Create a BufferManager for handling buffer interactions with engine.

Definition at line 322 of file buffers.hpp.

◆ ~BufferManager()

BufferManager::~BufferManager ( )

default

Member Function Documentation

◆ copyInputToDevice()

void BufferManager::copyInputToDevice ( )

inline

Copy the contents of input host buffers to input device buffers synchronously.

Definition at line 432 of file buffers.hpp.

Referenced by TensorrtEngine::infer().

◆ copyInputToDeviceAsync()

void BufferManager::copyInputToDeviceAsync ( cudaStream_t const & stream = 0 )

inline

Copy the contents of input host buffers to input device buffers asynchronously.

Definition at line 448 of file buffers.hpp.

◆ copyOutputToHost()

void BufferManager::copyOutputToHost ( )

inline

Copy the contents of output device buffers to output host buffers synchronously.

Definition at line 440 of file buffers.hpp.

Referenced by TensorrtEngine::infer().

◆ copyOutputToHostAsync()

void BufferManager::copyOutputToHostAsync ( cudaStream_t const & stream = 0 )

inline

Copy the contents of output device buffers to output host buffers asynchronously.

Definition at line 456 of file buffers.hpp.

◆ getBuffer()

void * BufferManager::getBuffer	(	bool const	isHost,
		std::string const &	tensorName
	)		const

inlineprivate

Definition at line 464 of file buffers.hpp.

Referenced by getDeviceBuffer(), and getHostBuffer().

◆ getDeviceBindings() [1/2]

std::vector< void * > & BufferManager::getDeviceBindings ( )

inline

Returns a vector of device buffers that you can use directly as bindings for the execute and enqueue methods of IExecutionContext.

Definition at line 357 of file buffers.hpp.

Referenced by TensorrtEngine::infer().

◆ getDeviceBindings() [2/2]

std::vector< void * > const & BufferManager::getDeviceBindings ( ) const

inline

Returns a vector of device buffers.

Definition at line 365 of file buffers.hpp.

◆ getDeviceBuffer()

void * BufferManager::getDeviceBuffer ( std::string const & tensorName ) const

inline

Returns the device buffer corresponding to tensorName. Returns nullptr if no such tensor can be found.

Definition at line 374 of file buffers.hpp.

Referenced by TensorrtEngine::initializeBuffer().

◆ getHostBuffer()

void * BufferManager::getHostBuffer ( std::string const & tensorName ) const

inline

Returns the host buffer corresponding to tensorName. Returns nullptr if no such tensor can be found.

Definition at line 383 of file buffers.hpp.

Referenced by TensorrtEngine::getOutput(), and TensorrtEngine::infer().

◆ memcpyBuffers()

void BufferManager::memcpyBuffers	(	bool const	copyInput,
		bool const	deviceToHost,
		bool const	async,
		cudaStream_t const &	stream = `0`
	)

inlineprivate

Definition at line 478 of file buffers.hpp.

Referenced by copyInputToDevice(), copyInputToDeviceAsync(), copyOutputToHost(), and copyOutputToHostAsync().

◆ print()

template<typename T >

void BufferManager::print	(	std::ostream &	os,
		void *	buf,
		size_t	bufSize,
		size_t	rowCount
	)

inline

Templated print function that dumps buffers of arbitrary type to std::ostream. rowCount parameter controls how many elements are on each line. A rowCount of 1 means that there is only 1 element on each line.

Definition at line 406 of file buffers.hpp.

◆ size()

size_t BufferManager::size ( std::string const & tensorName ) const

inline

Returns the size of the host and device buffers that correspond to tensorName. Returns kINVALID_SIZE_VALUE if no such tensor can be found.

Definition at line 392 of file buffers.hpp.

◆ tenosrIsInput()

bool BufferManager::tenosrIsInput ( const std::string & tensorName ) const

inlineprivate

Definition at line 473 of file buffers.hpp.

Referenced by memcpyBuffers().

Member Data Documentation

◆ kINVALID_SIZE_VALUE

const size_t BufferManager::kINVALID_SIZE_VALUE = ~size_t(0)

static

Definition at line 291 of file buffers.hpp.

Referenced by size().

◆ mBatchSize

int BufferManager::mBatchSize

private

The batch size for legacy networks, 0 otherwise.

Definition at line 499 of file buffers.hpp.

Referenced by BufferManager().

◆ mDeviceBindings

std::vector<void*> BufferManager::mDeviceBindings

private

The vector of device buffers needed for engine execution.

Definition at line 501 of file buffers.hpp.

Referenced by BufferManager(), BufferManager(), getDeviceBindings(), and getDeviceBindings().

◆ mEngine

std::shared_ptr<nvinfer1::ICudaEngine> BufferManager::mEngine

private

The pointer to the engine.

Definition at line 498 of file buffers.hpp.

Referenced by BufferManager(), BufferManager(), and tenosrIsInput().

◆ mManagedBuffers

std::vector<std::unique_ptr<ManagedBuffer> > BufferManager::mManagedBuffers

private

The vector of pointers to managed buffers.

Definition at line 500 of file buffers.hpp.

Referenced by BufferManager(), BufferManager(), getBuffer(), memcpyBuffers(), and size().

◆ mNames

std::unordered_map<std::string, int32_t> BufferManager::mNames

private

The map of tensor name and index pairs.

Definition at line 502 of file buffers.hpp.

Referenced by BufferManager(), BufferManager(), getBuffer(), memcpyBuffers(), and size().

The documentation for this class was generated from the following file:

apps/nnReconstructor/OLD/buffers.hpp

Public Member Functions

Static Public Attributes

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ BufferManager() [1/2]

◆ BufferManager() [2/2]

◆ ~BufferManager()

Member Function Documentation

◆ copyInputToDevice()

◆ copyInputToDeviceAsync()

◆ copyOutputToHost()

◆ copyOutputToHostAsync()

◆ getBuffer()

◆ getDeviceBindings() [1/2]

◆ getDeviceBindings() [2/2]

◆ getDeviceBuffer()

◆ getHostBuffer()

◆ memcpyBuffers()

◆ print()

◆ size()

◆ tenosrIsInput()

Member Data Documentation

◆ kINVALID_SIZE_VALUE

◆ mBatchSize

◆ mDeviceBindings

◆ mEngine

◆ mManagedBuffers

◆ mNames