Reverse Image Search Based on Milvus & Towhee

This demo uses towhee image embedding operator to extract image features by ResNet50, and uses Milvus to build a system that can perform reverse image search.

The system architecture is as below:

Data Source

This demo uses the PASCAL VOC image set, which contains 17125 images with 20 categories: human; animals (birds, cats, cows, dogs, horses, sheep); transportation (planes, bikes, boats, buses, cars, motorcycles, trains); household (bottles, chairs, tables, pot plants, sofas, TVs).

Dataset size: ~ 2 GB.

Download location: https://drive.google.com/file/d/1n_370-5Stk4t0uDV1QqvYkcvyV8rbw0O/view?usp=sharing

Note: You can also use other images for testing. This system supports the following formats: .jpg and .png.

Local Deployment

Requirements

• Milvus 2.2.10
• Towhee
• MySQL
• Python3
• Docker
• Docker Compose

Option 1: Deploy with Docker Compose

The reverse image search system requires Milvus, MySQL, WebServer and WebClient services. We can start these containers with one click through docker-compose.yaml.

• Modify docker-compose.yaml to map your data directory to the docker container of WebServer

$ wget https://raw.githubusercontent.com/milvus-io/bootcamp/master/solutions/image/reverse_image_search/docker-compose.yaml
$ vim docker-compose.yaml

Then to change line 75: ./data:/data --> your_data_path:/data

• Create containers & start servers with docker-compose.yaml

$ docker-compose up -d

Then you will see the that all containers are created.

Creating network "quick_deploy_app_net" with driver "bridge"
Creating milvus-etcd          ... done
Creating milvus-minio         ... done
Creating img-search-mysql     ... done
Creating img-search-webclient ... done
Creating milvus-standalone    ... done
Creating img-search-webserver ... done

And show all containers with docker ps, and you can use docker logs img-search-webserver to get the logs of server container.

CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS                      PORTS                                                                                      NAMES
f24ddb15a529   milvusbootcamp/img-search-server:2.2.10    "/bin/sh -c 'python3…"   35 minutes ago   Up 34 minutes               0.0.0.0:5000->5000/tcp, :::5000->5000/tcp                                                  img-search-webserver
d26b5cf21822   milvusdb/milvus:v2.2.10                    "/tini -- milvus run…"   35 minutes ago   Up 35 minutes               0.0.0.0:9091->9091/tcp, :::9091->9091/tcp, 0.0.0.0:19530->19530/tcp, :::19530->19530/tcp   milvus-standalone
1e9254f56006   minio/minio:RELEASE.2023-03-20T20-16-18Z   "/usr/bin/docker-ent…"   35 minutes ago   Up 35 minutes (healthy)     9000/tcp                                                                                   milvus-minio
445cef3fe474   milvusbootcamp/img-search-client:2.2.10    "/docker-entrypoint.…"   35 minutes ago   Up 35 minutes (unhealthy)   0.0.0.0:8001->80/tcp, :::8001->80/tcp                                                      img-search-webclient
d9bbab39f325   mysql:5.7                                  "docker-entrypoint.s…"   35 minutes ago   Up 35 minutes               33060/tcp, 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp                                       img-search-mysql
807c2ace0b28   quay.io/coreos/etcd:v3.5.5                 "etcd -advertise-cli…"   35 minutes ago   Up 35 minutes               2379-2380/tcp                                                                              milvus-etcd

Option 2: Deploy with source code

We recommend using Docker Compose to deploy the reverse image search system. However, you also can run from source code, you need to manually start Milvus and Mysql. Next show you how to run the API server and Client.

1. Start Milvus & Mysql

First, you need to start Milvus & Mysql servers.

Refer Milvus Standalone for how to install Milvus.

$ wget https://github.com/milvus-io/milvus/releases/download/v2.2.10/milvus-standalone-docker-compose.yml -O docker-compose.yml
$ sudo docker-compose up -d

There are several ways to start Mysql. One option is using docker to create a container:

$ docker run -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 -d --name image_search_mysql mysql:5.7

2. Start API Server

Then to start the system server, and it provides HTTP backend services.

• Install the Python packages

Please note the Milvus version should match pymilvus version in requirements.txt. And this tutorial uses milvus 2.2.10 and pymilvus 2.2.11.

$ git clone https://github.com/milvus-io/bootcamp.git
$ cd bootcamp/solutions/image/reverse_image_search/server
$ pip install -r requirements.txt

• Set configuration

$ vim src/config.py

Modify the parameters according to your own environment. Here listing some parameters that need to be set, for more information please refer to config.py.

Parameter	Description	Default setting
MILVUS_HOST	The IP address of Milvus, you can get it by ifconfig.	127.0.0.1
MILVUS_PORT	Port of Milvus.	19530
VECTOR_DIMENSION	Dimension of the vectors.	1000
MYSQL_HOST	The IP address of Mysql.	127.0.0.1
MYSQL_PORT	Port of Mysql.	3306
DEFAULT_TABLE	The milvus and mysql default collection name.	milvus_img_search

• Run the code

Then start the server with Fastapi.

$ python src/main.py

• API Docs

After starting the service, Please visit 127.0.0.1:5000/docs in your browser to view all the APIs.

/data: get image by path

/progress: get load progress

/img/load: load images into milvus collection

/img/count: count rows in milvus collection

/img/drop: drop milvus collection & corresponding Mysql table

/img/search: search for most similar image emb in milvus collection and get image info by milvus id in Mysql

3. Start Client

Next, start the frontend GUI.

• Set parameters

Modify the parameters according to your own environment.

Parameter	Description	example
API_HOST	The IP address of the backend server.	127.0.0.1
API_PORT	The port of the backend server.	5000

$ export API_HOST='127.0.0.1'
$ export API_PORT='5000'

• Run Docker

First, build a container by pulling docker image.

$ docker run -d \
-p 8001:80 \
-e "API_URL=http://${API_HOST}:${API_PORT}" \
 milvusbootcamp/img-search-client:2.2.10

How to use front-end

Navigate to 127.0.0.1:8001 in your browser to access the front-end interface.

1. Insert data

Enter /data in path/to/your/images, then click + to load the pictures. The following screenshot shows the loading process:

Notes:

After clicking the Load (+) button, the first time load will take longer time since it needs time to download and prepare models. Please do not click again.

You can check backend status for progress (check in terminal if using source code OR check docker logs of the server container if using docker)

The loading process may take several minutes. The following screenshot shows the interface with images loaded.

2.Search for similar images

Select an image to search.

Code structure

If you are interested in our code or would like to contribute code, feel free to learn more about our code structure.

Dockerfile
requirements.txt
src
├── __init__.py
├── config.py # Configuration file
├── encode.py # Convert an image to embedding using towhee pipeline (ResNet50)
├── logs.py
├── main.py # Source code to start webserver
├── milvus_helpers.py # Connect to Milvus server and insert/drop/query vectors in Milvus.
├── mysql_helpers.py # Connect to MySQL server, and add/delete/query IDs and object information.
├── operations
│   ├── __init__.py
│   ├── count.py
│   ├── drop.py
│   ├── load.py
│   ├── search.py
│   └── upload.py
└── test_main.py # Pytest file for main.py