onnxruntime-directml slower than onnxruntime cpu

[carter54]

System information

• OS Platform: Windows 10
• ONNX Runtime installed from: pip install onnxruntime-directml
• version: 1.9.0
• Python version: 3.7
• CPU: Intel i7-8700 CPU @ 3.20GHz
• GPU: Intel UHD Graphics 630

To Reproduce
Hi~ I tried to use Onnxruntime with Directml backend to accelerate the model inference. However, it turns out that Directml is slower than CPU...I did see the high usage of GPU during the Directml inference. Can anyone helps me to see whats going wrong here, please?

import onnxruntime as ort
import numpy as np
import cv2
import time

def load_video_frames(video_path):
    frame_list = []
    raw = cv2.VideoCapture(video_path)
    width = int(raw.get(cv2.CAP_PROP_FRAME_WIDTH))
    height = int(raw.get(cv2.CAP_PROP_FRAME_HEIGHT))
    # save image frame into a list
    while(raw.isOpened()):
        ret, frame = raw.read()
        if ret == True:
            frame = cv2.resize(frame, [512,512])
            frame_array = np.transpose(frame.astype('float32')/255.,
                                       (2, 0, 1))[np.newaxis,:]
            frame_list.append(frame_array)
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break
        else:
            break
    raw.release()
    cv2.destroyAllWindows()
    return frame_list, width, height

def onnx_inference(model_path, frame_list):
    providers = ['CPUExecutionProvider']   # use cpu
    # providers = ['DmlExecutionProvider']   # use directml
    sess = ort.InferenceSession(model_path, providers=providers)

    rec = [np.zeros([1, 1, 1, 1], dtype=np.float32)] * 4  
    downsample_ratio = np.array([0.25], dtype=np.float32) 

    pha_list = []  # save result in a list
    for src in frame_list:  # src is of [B, C, H, W] with dtype of the model.
        fgr, pha, *_ = sess.run([], {     # to prevend io between cpu and gpu, do not update rec here
            'src': src,
            'r1i': rec[0],
            'r2i': rec[1],
            'r3i': rec[2],
            'r4i': rec[3],
        })
        pha_list.append(pha)
    return pha_list

if __name__ == "__main__":
    video_path = "a_720p_video_path"
    model_path = "the_path_of_model"
    frame_list, width, height = load_video_frames(video_path)
    time1 = time.time()
    pha_list = onnx_inference(model_path, frame_list)
    time2 = time.time()
    print((time2-time1)/len(frame_list))

the model I test can be found here

[bleu48]

Nothing ia wrong. Old Intel GPU is just poor.

[carter54]

@bleu48 Thx for your reply. Although this Intel GPU is old. its calculation speed should be faster than CPU...especially for the CV model

[giaanthunder]

@carter54: have you solved this problem yet? I have it too >_<

[elephantpanda]

Me too.

[stealthinu]

Me too...

[elephantpanda]

Me too...

I solved mine. My tips are:
(1) Make sure you have enough RAM so it doesn't use your hard disk. 8GB or more.
(2) Make sure it is loading the latest DirectML.dll
(3) The GPU session will take a long time to load. But each inference should be faster. So overall, if you do a lot of inferences using the same session it should be faster overall.

My inferences are going 2x using DirectML as fast as the CPU. But the sessions are taking about 1-inference time to load.