Step-by-Step Guide to Running Hello World on FPGA Instances

During the AWS Tech Summit in San Francisco on April 19, Amazon CTO Werner Vogels announced several new AWS features, including the highly anticipated FPGA instance, F1.

The F1 instance is equipped with the latest 16 nm Xilinx UltraScale Plus FPGA, currently available in two types: f1.2xlarge, which comes with 1 FPGA card, and f1.16xlarge, which comes with 8 FPGA cards.

Using F1 instances to deploy hardware acceleration is very useful in many high-performance computing (HPC) applications, solving complex scientific, engineering, and business problems that require high bandwidth, enhanced networking, and higher computational power. The F1 instance is particularly suitable for time-sensitive applications such as clinical genomics, real-time video processing, and financial risk analysis.

Since I’ve been learning about neural networks lately, what attracts me most about the F1 instance is deploying neural network models on the FPGA. The forward computation of the neural network occurs simultaneously in the neural network units formed by logic gates, which is exciting to think about.

However, the FPGA is indeed quite specialized, and the learning curve for beginners is steep. It takes some effort to start an F1 instance and run a simple Hello World.

So here, I will document the process of starting an F1 instance to run Hello World for your reference, hoping to help everyone begin their FPGA journey.

The process of starting an f1 instance is similar to the general EC2 startup process. For details on preparing your AWS account and the EC2 creation process, please refer to the relevant technical documentation. Below are some points to note.

For testing, I chose the “Northern Virginia” region, which is the us-east-1 region.

It is strongly recommended to use the AWS FPGA Developer AMI image when starting the f1 instance. The FPGA Developer AMI includes a pre-packaged tool development environment, which contains scripts and tools for simulating FPGA designs, compiling code, and building and registering AFIs.

In the first step of starting the instance, when selecting the system image, choose “AWS Marketplace” and then search for “FPGA” to find the FPGA Developer AMI. The ID of this image in the Northern Virginia region is: ami-3afc6f2c, and a screenshot of the image selection interface is as follows.

During the startup process, be sure to assign an IAM Role to your instance. You will need to configure static Access Key and Secret Key when using the AWS CLI command line tool later.

Also, regarding security group configuration, the default port 22 should remain open, and it is recommended to open port 3389 if you want to use a graphical interface remotely later.

Another point that should not be emphasized is that the f1 instance you start needs to have a public IP.

After the f1 instance starts, find the public IP address of this instance in the EC2 console, and then connect to the instance using the ssh command. Note that the username to use is centos. An example of the ssh command is as follows: ssh -i ~/.ssh/ centos@

After logging in, you can see the following information: ___ ___ ___ _ ___ _____ __ _ __ __ ___| __| _ \/ __| /_\ | \| __\ \ / / /_\ | \/ |_ _|| _|| _/ (_ |/ _ \ | |) | _| \ V / / _ \| |\/| || ||_| |_| \___/_/ \_\ |___/|___| \_/ /_/ \_\_| |_|___|AMI Version: 1.2.0Readme: /home/centos/src/README.mdGUI Setup Steps: /home/centos/src/GUI_README.mdAMI Release Notes: /home/centos/src/RELEASE_NOTES.mdXilinx Tools: /opt/Xilinx/Developer Support: https://github.com/aws/aws-fpga/blob/master/README.md#developer-support

Note the GUI setup documentation mentioned here: /home/centos/src/GUI_README.md, if you wish to use a graphical interface later, please refer to this document for operation.

Create a working directory on the remote f1 instance, and execute the following command to clone the AWS FPGA project:

git clone https://github.com/aws/aws-fpga.git

After execution, you will see a directory named aws-fpga

Install the FPGA SDK by entering the aws-fpga directory and then executing the following command to install the FPGA SDK:

source sdk_setup.sh

After the installation command is executed, the following output indicates successful installation:

Done with SDK install.Done with AWS SDK setup.

This process will install the FPGA management tools, which allow you to view the status of the FPGA card, list the image status on the FPGA card, and perform operations such as loading images.

First, we can check the status of the FPGA cards using the fpga-describe-local-image-slots command. Here is the status of the FPGA’s “image slot”; the specific command is as follows:

sudo fpga-describe-local-image-slots -H

On my f1.2xlarge instance, the output is as follows:

$ sudo fpga-describe-local-image-slots -HType FpgaImageSlot VendorId DeviceId DBDFAFIDEVICE 0 0x1d0f 0x1042 0000:00:1d.0

The listed information is the status of slot 0. Next, you can use the fpga-describe-local-image-slots command to check the image status on this slot:

sudo fpga-describe-local-image -S 0

Where the parameter -S is used to specify the image slot you wish to view, and the parameter value is the FPGA image slot number.

If you want to load a new FPGA image into an image slot, you can use the command fpga-load-local-image, specifying the image slot number with the -S parameter and the FPGA image ID to load with the -I parameter, such as:

sudo fpga-load-local-image -S 0 -I agfi-0123456789abcdefg

For more information on FPGA management tools, please refer to the following link:

https://github.com/aws/aws-fpga/blob/master/sdk/userspace/fpga_mgmt_tool…

After successfully installing the FPGA SDK, you need to install the FPGA HDK. The specific installation command is as follows:

source hdk_setup.sh

This process takes a little longer, possibly 5 to 10 minutes, and the output will prompt during the execution:

AWS FPGA-INFO: This could take 5-10 minutes, please be patient!

If everything goes well, a successful installation will yield the following information:

AWS FPGA-INFO: DDR4 model build passed.AWS FPGA-INFO: ATTENTION: Don’t forget to set the CL_DIR variable for the directory of your Custom Logic.AWS FPGA-INFO: AWS HDK setup PASSED.

As mentioned in the success message, if you want to build your own FPGA module in the future, you need to point the CL_DIR variable to your module directory, which we will set in the subsequent steps.

Since the FPGA image creation process requires the use of the AWS CLI command line tool, we need to configure the AWS CLI in advance.

The AWS CLI command line tool is already installed in the FPGA Developer AMI we are using. If you use other images and need to install the AWS CLI manually, please refer to the installation documentation for the AWS CLI command line tool.

It is worth noting that although the AWS CLI is already installed in the FPGA Developer AMI, this version may not be the latest, so it is recommended to upgrade it using the following command:

pip install –upgrade –user awscli

Then start the AWS CLI configuration process with the following command:

aws configure

During the AWS CLI configuration process, you generally need to provide the Access Key and Secret Key. However, since our f1 instance was assigned an IAM Role during startup, and I gave this IAM Role sufficient permissions, we do not need to configure static Access Key and Secret Key here. We only need to specify the region, which we will set to us-east-1.

$aws configureAWS Access Key ID [None]:AWS Secret Access Key [None]:Default region name [None]: us-east-1Default output format [None]:

After installing the HDK and configuring the tools, we can start running examples. Here, we will use the hello world example from the AWS FPGA project, which is located in the following directory in aws-fpga:

./hdk/cl/examples/cl_hello_world

As mentioned during the installation process, if we use the example in ./hdk/cl/examples/cl_hello_world, we need to set the CL_DIR variable to point to the ./hdk/cl/examples/cl_hello_world directory. The specific command is as follows:

cd ./hdk/cl/examples/cl_hello_worldexport CL_DIR=$(pwd)

The next steps require using the Xilinx Vivado tool, so we need to check if the vivado tool is installed correctly. The specific command is as follows:

$ vivado -mode batch

The normal output is as follows:

****** Vivado v2017.1 (64-bit)**** SW Build 1846317 on Fri Apr 14 18:54:47 MDT 2017**** IP Build 1846188 on Fri Apr 14 20:52:08 MDT 2017** Copyright 1986-2017 Xilinx, Inc. All Rights Reserved.

Sourcing tcl script ‘/opt/Xilinx/Vivado/2017.1/scripts/Vivado_init.tcl’INFO: [Common 17-206] Exiting Vivado at Fri Apr 21 02:42:35 2017…

Now we can start building the DCP. The build command is in $CL_DIR/build/scripts, and the filename is aws_build_dcp_from_cl.sh, so the specific command is as follows:

cd $CL_DIR/build/scripts$ ./aws_build_dcp_from_cl.sh

It is worth noting that this command takes a long time to run, requiring several hours to complete. To avoid build failure due to SSH session interruption, the example’s author chose to run the build process in the background. After we run the aws_build_dcp_from_cl.sh command, we will immediately receive the following output, but the build program will continue to run in the background:

$ ./aws_build_dcp_from_cl.shAWS FPGA: Starting the design checkpoint build processAWS FPGA: Checking for proper environment variables and build directoriesCreating the reports directoryCreating the checkpoints directoryCreating the checkpoints/to_aws directoryAWS FPGA: Environment variables and directories are present. Checking for Vivado installation.AWS FPGA: Build through Vivado is running as background process, this may take few hours.AWS FPGA: Output is being redirected to 17_04_21-025018.nohup.outAWS FPGA: If you have set your EMAIL environment variable and -notify is specified, you will receive a notification when complete.AWS FPGA: (See $HDK_DIR/cl/examples/README.md for details)

In the output above, we can note that the build log will be output to the file xxxx.nohup.out, so we can periodically check this log file to understand the build progress.

Of course, periodically checking the log file to see if the build is complete is not a very effective way. If you want the build program to send you an email when it ends, you can use the -notify parameter. Before using the -notify parameter, set the SNS using the following command:

$ export [email protected]$ ./$HDK_COMMON_DIR/scripts/notify_via_sns.py

For more information on the -notify parameter, please refer to the corresponding README.md file; this example will not set it and will use the less efficient method of periodically checking the log.

After the build is completed, we can see the following information in the xxxx.nohup.out file:

AWS FPGA: (07:00:53) Finished creating final tar file in to_aws directory.

Then you can check this directory: $CL_DIR/build/checkpoints/to_aws, and the directory will contain the packaged tar file. The result of executing the ls command is as follows:

$ ls checkpoints/to_aws17_04_21-025018.Developer_CL.tar 17_04_21-025018.manifest.txt 17_04_21-025018.SH_CL_routed.dcp

After building the dcp, we need to upload the tar file to S3 before we can build the FPGA image using the AWS CLI command.

To upload files to S3, we need to create the corresponding S3 bucket. This process can be completed through the AWS console or using the AWS CLI command line tool. For specific operations regarding S3, please refer to the relevant documentation.

This example uses the AWS CLI to create the S3 bucket and upload the file, with the command as follows:

$ aws s3 mb s3:// –region us-east-1$ aws s3 cp $CL_DIR/build/checkpoints/to_aws/*.Developer_CL.tar s3:////

Next, we also need to create a directory for the log files. In fact, there is no concept of directories in S3; the entire file path and filename are the key of this file. Therefore, the method used in this example to create a directory is to directly upload an empty file to the directory we need. The specific command is as follows:

$ touch LOGS_FILES_GO_HERE.txt$ aws s3 cp LOGS_FILES_GO_HERE.txt s3:////

Since the tar file we uploaded will be handled by the AWS corresponding account to complete the build work, and the build logs also need to be written back to our S3 bucket by the AWS corresponding account, we need to set the bucket access policy for our S3 bucket to allow the AWS account to access these files and directories. The specific access policy sample is as follows. We need to copy the policy below to the “Access Policy” settings of our S3 bucket. Note the placeholders and , which should be replaced with actual bucket names and corresponding paths.

{“Version”: “2012-10-17″,”Statement”: [{“Sid”: “Bucket level permissions”,”Effect”: “Allow”,”Principal”: {“AWS”: “arn:aws:iam::365015490807:root”},”Action”: [“s3:ListBucket”],”Resource”: “arn:aws:s3:::“},{“Sid”: “Object read permissions”,”Effect”: “Allow”,”Principal”: {“AWS”: “arn:aws:iam::365015490807:root”},”Action”: [“s3:GetObject”],”Resource”: “arn:aws:s3::://“},{“Sid”: “Folder write permissions”,”Effect”: “Allow”,”Principal”: {“AWS”: “arn:aws:iam::365015490807:root”},”Action”: [“s3:PutObject”],”Resource”: “arn:aws:s3::://*”}]}

After setting the S3 bucket access policy, we need to verify that the policy is correct. If the policy is written incorrectly, the AWS corresponding account will not be able to access the required tar file, and we will not know how to analyze the problem.

The script to verify the S3 bucket policy is in the following file:

`aws-fpga/hdk/common/scripts/check_s3_bucket_policy.py`

If the execution shows INFO: Passed, it indicates that the policy is set correctly.

However, in some Python environments, running check_s3_bucket_policy.py may report the following error:

AttributeError: PolicyStatement instance has no attribute ‘principals_re’

If you encounter this error, you need to manually modify the check_s3_bucket_policy.py file.

Open the check_s3_bucket_policy.py file in your preferred editor and find the following code:

class PolicyStatement:def __init__(self, statement, principal=None):self.statement = statementself.process_policy_statement(statement, principal)self.principals_re = []self.actions_re = []self.notactions_re = []self.resources_re = []

Then move the self.process_policy_statement(statement, principal) line to after the other variable settings, as follows:

class PolicyStatement:def __init__(self, statement, principal=None):self.statement = statementself.principals_re = []self.actions_re = []self.notactions_re = []self.resources_re = []self.process_policy_statement(statement, principal)

Then it will not report an error. The reference for executing the check_s3_bucket_policy.py command and the corresponding output is as follows:

$ check_s3_bucket_policy.py –dcp-bucket fpga.use1.damondeng.com –dcp-key fpgajarfile/17_04_21-025018.Developer_CL.tar

–logs-bucket fpga.use1.damondeng.com –logs-key logfileINFO: Passed

Once everything is ready, you can start running the aws ec2 create-fpga-image command to build the FPGA image. The command reference is as follows:

$ aws ec2 create-fpga-image –name DamonFPGAOne

–description “Testing FPGA Image” –input-storage-location Bucket=fpga.use1.damondeng.com,Key=fpgajarfile/17_04_21-025018.Developer_CL.tar

–logs-storage-location Bucket=fpga.use1.damondeng.com,Key=logfile

If you find that the AWS CLI command reports the following error, your AWS CLI version is insufficient, and you need to run pip install –upgrade –user awscli to upgrade:

Invalid choice: ‘create-fpga-image’, maybe you meant:

* create-image*

If the command runs normally, you will receive output similar to the following:

{“FpgaImageId”: “afi-046ead8eb3a0e3112″,”FpgaImageGlobalId”: “agfi-06fdb0f3cea076195”}

Where “FpgaImageId” is the unique image ID in this region, and “FpgaImageGlobalId” is the globally unique image ID. The one we will use when loading the FPGA image is the globally unique “FpgaImageGlobalId”, which starts with agfi.

After starting the FPGA image build process, you need to wait for a while. You can check the S3 bucket you specified for saving logs to understand the progress.

If you see a new directory generated in the log directory, and the State file inside contains {State=available}, it indicates that the build was successful. You can then load your FPGA image.

Before loading the new FPGA image, remember to clear the existing image:

sudo fpga-clear-local-image -S 0

Then load the FPGA image using the following command:

sudo fpga-load-local-image -S 0 -I agfi-06fdb0f3cea076195

As previously described, the -S parameter is used to specify the image slot, and the -I parameter is used to specify the FPGA image ID, which is globally unique and starts with agfi.

To check if the FPGA image loaded successfully, you can use the fpga-describe-local-image command. The output example is as follows:

$ sudo fpga-describe-local-image -S 0 -R -HType FpgaImageSlot FpgaImageId StatusName StatusCode ErrorName ErrorCode ShVersionAFI 0 agfi-06fdb0f3cea076195 loaded 0 ok 0 0x04151701Type FpgaImageSlot VendorId DeviceId DBDFAFIDEVICE 0 0x1d0f 0xf000 0000:00:1d.0

Here, you can see that the status of the image ID agfi-06fdb0f3cea076195 is loaded, which means it loaded successfully.

Finally, we need to run the software on the host machine for testing. Enter the cd $CL_DIR/software/runtime/ directory, where there is a pre-written C code for testing. Run the following command to compile the software test:

$ cd $CL_DIR/software/runtime/$ make all

After successful compilation, execute the ./test_hello_world command, and the following is the execution result:

$ sudo ./test_hello_worldAFI PCI Vendor ID: 0x1d0f, Device ID 0xf000===== Starting with peek_poke_example =====register: 0xdeadbeefResulting value matched expected value 0xdeadbeef. It worked!Developers are encouraged to modify the Virtual DIP Switch by calling the linux shell

command to demonstrate how AWS FPGA Virtual DIP switches can be used to change a CustomLogic functionality:$ fpga-set-virtual-dip-switch -S (slot-id) -D (16 digit setting)In this example, setting a virtual DIP switch to zero clears the corresponding LED, even if the peek-poke example would set it to 1.For instance:# fpga-set-virtual-dip-switch -S 0 -D 1111111111111111# fpga-get-virtual-led -S 0FPGA slot id 0 have the following Virtual LED:1010-1101-1101-1110# fpga-set-virtual-dip-switch -S 0 -D 0000000000000000# fpga-get-virtual-led -S 0FPGA slot id 0 have the following Virtual LED:0000-0000-0000-0000

This example has two parts. One part is the peek_poke part, which is the read and write of registers. The example illustrates whether the functionality of the FPGA register works by swapping the input bits.

As described in ./test_hello_world.c:

uint32_t value = 0xefbeadde;uint32_t expected = 0xdeadbeef;/* read it back and print it out; you should expect the byte order to be* reversed (That’s what this CL does) */

The second part is the use of virtual LEDs. Testers can set the virtual DIP switch using the FPGA management tool, which affects the display of virtual LEDs in a mask-like manner. For example, if the output of the virtual LED is 1010-1101-1101-1110 and I set the value of fpga-set-virtual-dip-switch to 1111111100000000, the output of the virtual LED will be 1010-1101-0000-0000:

[centos@ip-172-31-8-87 runtime]$ sudo fpga-set-virtual-dip-switch -S 0 -D 1111111100000000[centos@ip-172-31-8-87 runtime]$ sudo fpga-get-virtual-led -S 0FPGA slot id 0 have the following Virtual LED:1010-1101-0000-0000

At this point, our Hello World has been successfully executed. Although it is much more complicated than a regular software Hello World, it is important to know that this is directly manipulating hardware.

After running the Hello World example, many people may want to know about the graphical access to this development environment.

As mentioned in the article, when you log into the FPGA instance via SSH, the welcome message of the FPGA Developer AMI mentions the setup of the GUI interface. The method mentioned here can be summarized as starting xrdp on centos and then connecting via Microsoft’s remote desktop program.

The command to run on the centos side is as follows:

sudo yum install -y kernel-devel # Needed to re-build ENA driversudo yum groupinstall -y “Server with GUI”sudo systemctl set-default graphical.targetsudo yum -y install epel-releasesudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.e…sudo yum install -y xrdp tigervnc-serversudo systemctl start xrdpsudo systemctl enable xrdpsudo systemctl disable firewalldsudo systemctl stop firewalld

After setting up, remember to set a password for the centos user; otherwise, remote desktop login will not work:

sudo passwd centos

When you log in to centos via remote desktop, you will see the graphical interface. For more information, you can refer to Jeff Bar’s blog https://aws.amazon.com/blogs/aws/developer-preview-ec2-instances-f1-with-programmable-hardware/

The following is a screenshot of the remote graphical interface copied from that blog:

Finally, I hope everyone can find their direction in the world of FPGA and that starting to create your own chip system is truly an exciting thing.

AWS Solutions Architect; with 15 years of experience in the IT field, having served as an engineer and architect at companies such as IBM, RIM, and Apple; currently employed at AWS as a Solutions Architect. Enjoys programming, various programming languages, especially Lisp. Interested in new technologies and various technical challenges, currently focusing on learning machine learning algorithms in distributed computing environments.

Source: Amazon AWS Official Blog