This document is relevant for: Inf1

LibTorch C++ Tutorial#

Table of Contents


This tutorial demonstrates the use of LibTorch with Neuron, the SDK for Amazon Inf1 instances. By the end of this tutorial, you will understand how to write a native C++ application that performs inference on EC2 Inf1 instances. We will use an inf1.6xlarge and a pretrained BERT-Base model to determine if one sentence is a paraphrase of another.

Run the tutorial#

First run the HuggingFace Pretrained BERT tutorial [html] [notebook].

You should now have a compiled file, which is required going forward. Right-click and copy this link address to the tutorial archive.

$ wget <paste archive URL>
$ tar xvf libtorch_demo.tar.gz

Your directory tree should now look like this:

├── libtorch_demo
│   ├── example_app
│   │   ├── README.txt
│   │   ├──
│   │   ├── example_app.cpp
│   │   ├── utils.cpp
│   │   └── utils.hpp
│   ├── neuron.patch
│   ├──
│   ├──
│   └── tokenizers_binding
│       ├──
│       ├──
│       ├── remote_rust_tokenizer.h
│       ├──
│       ├──
│       ├── tokenizer_test
│       ├── tokenizer_test.cpp
│       └──
└── libtorch_demo.tar.gz

Copy the compiled model from Step 2 into the new libtorch_demo directory.

$ cp libtorch_demo/

This tutorial uses the HuggingFace Tokenizers library implemented in Rust. Install Cargo, the package manager for the Rust programming language.



$ sudo apt install -y cargo
$ sudo yum install -y cargo

Run the setup script to download additional depdendencies and build the app. (This may take a few minutes to complete.)

$ cd libtorch_demo
$ chmod +x && ./
[100%] Built target example_app
make[1]: Leaving directory '/home/ubuntu/libtorch_demo/example_app/build'
/usr/local/lib/python3.6/dist-packages/cmake/data/bin/cmake -E cmake_progress_start /home/ubuntu/libtorch_demo/example_app/build/CMakeFiles 0
Successfully completed setup


Run the provided sanity tests to ensure everything is working properly.

$ ./
Running tokenization sanity checks.

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Tokenizing: 100%|██████████████████████████████████████████████████████████████████████████████████| 10000/10000 [00:00<00:00, 15021.69it/s]
Python took 0.67 seconds.
Sanity check passed.
Begin 10000 timed tests.
End timed tests.
C++ took 0.226 seconds.

Tokenization sanity checks passed.
Running end-to-end sanity check.

The company HuggingFace is based in New York City
HuggingFace's headquarters are situated in Manhattan
not paraphrase: 10%
paraphrase: 90%

The company HuggingFace is based in New York City
Apples are especially bad for your health
not paraphrase: 94%
paraphrase: 6%

Sanity check passed.

Finally, run the example app directly to benchmark the BERT model.


You can safely ignore the warning about None of PyTorch, Tensorflow >= 2.0, .... This occurs because the test runs in a small virtual environment that doesn’t require the full frameworks.

$ LD_LIBRARY_PATH="libtorch/lib:tokenizers_binding/lib" ./example-app
Getting ready....
Completed 4000 operations in 22 seconds => 1090.91 pairs / second

Summary information:
Batch size = 6
Num neuron cores = 4
Num runs per neruon core = 1000

Congratulations! By now you should have successfully built and used a native C++ application with LibTorch.

This document is relevant for: Inf1