Following NVIDIA’s Chat with RTX launch, AMD is now offering users their very own localized & GPT-based LLM-powered AI chatbot which can run on Ryzen AI CPUs & Radeon 7000 GPUs.
NVIDIA Chat With RTX Has A Competitor As AMD Unveils Its Own Localized LLM-Powered Chatbot That Can Run on Ryzen AI CPUs & Radeon 7000 GPUs
Last month, NVIDIA launched its “Chat with RTX” AI Chatbot which is available across its RTX 40 & RTX 30 GPUs & is accelerated with the TensorRT-LLM feature set which offers faster GenAI results based on the data you make available to it from your PC or in other terms, a localized dataset. Now AMD is offering its own LLM-based GPT chatbot which can run on a diverse range of hardware such as the Ryzen AI PCs which include Ryzen 7000 & Ryzen 8000 APUs featuring the XDNA NPUs along with the latest Radeon 7000 GPUs which feature AI accelerator cores.
AMD has published a blog where it provides a setup guide on how to utilize its hardware to run your very own localized chatbot powered by GPT-based LLMs (Large Language Models). For AMD Ryzen AI CPUs, you can get the standard LM Studio copy for Windows while the Radeon RX 7000 GPUs get a ROCm Technical Preview. The full guide is shared below:
1. Download the correct version of LM Studio:
2. Run the file.
3. In the search tab copy and paste the following search term depending on what you want to run:
a. If you would like to run Mistral 7b, search for: “TheBloke/OpenHermes-2.5-Mistral-7B-GGUF” and select it from the results on the left. It will typically be the first result. We are going with Mistral in this example.
b. If you would like to run LLAMA v2 7b, search for: “TheBloke/Llama-2-7B-Chat-GGUF” and select it from the results on the left. It will typically be the first result.
c. You can also experiment with other models here.
4. On the right-hand panel, scroll down till you see the Q4 K M model file. Click download.
a. We recommend Q4 K M for most models on Ryzen AI. Wait for it to finish downloading.
5. Go to the chat tab. Select the model from the central, drop-down menu in the top center and wait for it to finish loading up.
6. If you have an AMDRyzen AI PC you can start chatting!
a. If you have an AMDRadeon graphics card, please:
i. Check “GPU Offload” on the right-hand side panel.
ii. Move the slider all the way to “Max”.
iii. Make sure AMD ROCm is being shown as the detected GPU type.
iv. Start chatting!
Having a localized chatbot powered by AI can make life and work relatively easier if set up properly. You can be efficient in doing your work and get proper results based on your queries and the data path the LLM is targeted at. NVIDIA and AMD are accelerating the pace of AI-powered features for consumer-tier hardware and this is just the start, expect more innovations down the road as the AI PC segment reaches new heights.
