Skip to main content

Visual Chatbot

Prerequisites​

  • Install Docker Desktop 4.42.0 or later
  • Enable Docker Model Runner in Docker Dashboard
  • Enable the TCP host socket for DMR (use the default port of 12434)
  • Download a model
doocker model pull ai/llama3.2:1B-Q8_0

Step 1. Start the visual chatbot container​

docker run -dp 3002:3000 -v /var/run/docker.sock:/var/run/docker.sock mikesir87/visual-chatbot

Did you notice that we're running the chatbot on port 3002?

Step 2. Access the chatbot​

Open your web browser and go to http://localhost:3002.

Step 3. Choose the right model and LLM Backend​

For this demo, select the following options:

  • Docker Model Runner
  • Model: ai/llama3.2:1B-Q8_0

Click "Save" to apply the settings.

The system prompt currently indicates the LLM should act as a whimsical guide. This suggests it will be fun and probably use lots of emojis!

Step 4. Ask what it can do​

Enter a prompt asking "Hello! What can you do for me today?"

first-prompt

Note that the app does not stream the responses, so it may take a little while to get the response. So, try to stall for time a bit (without revealing it's all running locally).

Hooray! We have a message!

Step 5. Click the response to see the message​

Clicking on a message will provide the details of that message, which came directly from the LLM and will go back in the next API request.

second-prompt

Step 6. Change the system prompt​

system-prompt

Go to Settings -> System prompt and change the prompt to be the "Grumpy old man".

grumpy

Before submitting, enable the "replay messages". Then, press save.

Step 7. Notice the very different persona​

grumpy-prompt

Highlight the much shorter and direct response from the LLM now. At the end of the day, it's the same model, but with a very different set of instructions.

This starts to introduce prompt engineering and how it's important to set the rules and the persona for the LLM. Again, normally, this is done by the GenAI application and not something an end user can change.

Interact with the LLM for the real-time information​

Reset all the messages by clicking on the "Reset messages" button in the top right corner of the chat window.

Let's ask the LLM what time it is in New York.

Enter a prompt asking "What time is it in New York now?"

time-query

Step 8. Adding Time Tool​

Click "Add Time tool" to add a tool that will provide the current time in New York.

add-time-tool

Step 9. Ask the LLM again​

Now, ask the LLM again "What time is it in New York now?".

time-response