Forlinx RK182X AI Computing Card Offers Up to 20 TOPS

Rockchip has announced two LLM/VLM accelerators in the RK182X family, the RK1820 with 2.5GB RAM to support 3B parameter models, and the RK1828 with 5GB of RAM to handle 7B parameter models. Forlinx Embedded has released an M.2 AI accelerator card based on Rockchip’s RK1820 and RK1828 processors, offering 20 TOPS of INT8 computing performance and up to 5GB of integrated DRAM. The memory is a 3D stacked architecture with DRAM on-chip with the processing hardware. The accelerator is in a standard M.2 2280 M-Key form factor and communicates with the host over a single PCIe 2.1 lane.

The Rockchip host processors supported are RK3568, RK3572, RK3576 and RK3588. The coprocessor architecture allows AI inference to be performed on the accelerator whereas the host processor processes application logic, peripheral interfaces and other system tasks. Supported computational tasks include convolutional neural networks, object detection, audio processing, time-series prediction, large language models and multi-modal models. The accelerator is for offline GenAI Inference with models in the 3B to 7B parameter range, including examples from the Qwen and Llama families executing 3B-7B LLMs/VLMs (e.g., Qwen, LLaMA2) at 100+ tokens/s for low-latency while avoiding cloud dependency.

Forlinx RK1820/RK1828 M.2 AI Computing Card

Forlinx RK1820/RK1828 M.2 AI Accelerator — Specifications:

• Hardware and computing capabilities:
  • AI processor: Rockchip RK1820 or RK1828
  • NPU performance: Up to 20 TOPS at INT8 precision
  • On-module memory: Integrated 2.5GB DRAM (RK1820) or 5GB DRAM (RK1828)
  • Supported numerical formats: INT4, INT8, INT16, FP8, FP16, and BF16
  • AI workload support: Vision processing, audio inference, time-series forecasting, large language models (LLMs), and multimodal models
• Form factor and connectivity:
  • Module format: M.2 2280
  • Connector: M.2 M-Key
  • Host interface: PCIe 2.1 x1
  • Compatible host platforms: Rockchip RK3568, RK3572, RK3576, and RK3588
  • Architecture: Dedicated AI coprocessor designed to offload inference workloads from the host processor
• Software and framework support:
  • Operating systems: Linux and Android
  • AI frameworks and toolchains: RKNN, TensorFlow, PyTorch, and ONNX
• Key highlights:
  • Integrated DRAM: Offers 2.5GB memory on RK1820 or 5GB on RK1828.
  • Offline LLM Inference: Supports local 3B–7B LLMs and VLMs, including Qwen and LLaMA 2

On an OK3588-C development board connected to an RK1828 via PCIe, the reported results are about 102 tokens/sec for Qwen2.5-3B and 70 tokens/sec for Qwen2.5-7B. The measurement valuations are performed with 128 input tokens and 128 generated tokens in performance mode. Real-world throughput will vary depending on model and operating conditions. Forlinx has also shown a multi-accelerator configuration using its OK3588-C development board, linking four RK1828 cards over PCIe. Leveraging Rockchip’s latest official RK182X_AI_SDK_V1.1.0 technology stack, the four-card configuration supports 27B and 31B parameter models with estimated overall accelerator power utilization of approximately 40W.

OK3588-C + 4x RK1828 PCIe Cascading

The model conversion, quantization and deployment are based on the accelerator’s Rockchip RKNN3 SDK. The software development environment supports CNN, LLM and VLM workloads and the documentation details model deployment using the OK3588-C development board. The software ecosystem supports models from TensorFlow, PyTorch and ONNX. Supported host operating systems are listed as Linux and Android. The wiki has rudimentary documentation but doesn’t tell how to get the RKNN3 Toolkit which is anticipated to be released soon enough on the AIRockchip Github account.  

Pricing and availability have not been disclosed yet. The users can learn more about it in the Forlinx product page and in the corresponding blog post. 

Images used courtesy of Forlinx.