What is the ideal way of measuring the FPS of a model?

[veds12]

I am trying to measure the FPS of a CenterNet model trained on a custom dataset.

1.) Ideally, what will be the best way to go about doing this to obtain the best possible FPS?
2.) What will be the best way to go about this to obtain a FPS which is as close as possible to the FPS I'll get when the model will actually be deployed (not necessarily the best one)

Would it be better to write a script for measuring the FPS like this one and simply measure the runtime of the relevant portion of code?

import sys
CENTERNET_PATH = /path/to/CenterNet/src/lib/
sys.path.insert(0, CENTERNET_PATH)

from detectors.detector_factory import detector_factory
from opts import opts

MODEL_PATH = /path/to/model
TASK = 'ctdet' # or 'multi_pose' for human pose estimation
opt = opts().init('{} --load_model {}'.format(TASK, MODEL_PATH).split(' '))
detector = detector_factory[opt.task](opt)

imgs = load_images()                                            # imgs is a list of images 

begin = time.time()
for img in imgs:
      ret = detector.run(imgs)['results']
end = time.time()

print(f"FPS = {len(imgs)/(begin-end)}

or would it be to use src/test.py like done in second part of the first comment on #247 ?

I also went through the relevant issues and noticed that the FPS would depend on a lot of factors like whether pin_memory=True (what does this option do btw?), --fix_res(when is this option needed? The model seems to work on images of res other than 512*512 w/o this option as well) and --flip_test are used or not (as discussed in #247) , whether --not_prefetched_test option is used or not (as discussed in #381) (again, what is not_prefetched_test used for?)

The pointers I have gathered till now are:

• Set the batch size 1 while testing
• Measure the time for the model to run over multiple images (not just one image) and then take an average

But I am particularly confused about what all options to use (as mentioned in prev paragraph) and what all settings to enable/disable.

PS: I am using a remote machine so I won't be able to display the images after the detection is done (don't want to go through the pain of setting up X11 forwarding). I am aware that this might lead to a difference in FPS than what will actually be the case when the model is deployed since it will be displaying the frames then but since my main aim is to compare CenterNet's FPS with some other model's FPS, for which I won't be able to display the detected images either, I am assuming this won't be a problem an I'll be able to get a general idea about how the two models perform relatively. Please correct me if I am wrong here. Any suggestions to circumvent this problem will also be really helpful

[xingyizhou]

Hi,
Thank you for the detailed question and sorry for my delayed response. By default, detector.run() will return the runtime of each step. You can find example use case here. In your case, you can do

for img in imgs:
      ret = detector.run(img)
      print(ret['tot'])

The runtime depends on the image sizes. --fix_res will use a fixed 512x512 input size. Without it the model will use the original image size (pad it to a multiplication of 32). --flip_test makes the model run slower with better accuracy. --not_prefetched_test uses a background thread to load and preprocess (resize/ normalize) images, and will exclude the load/ preprocessing time from runtime. However it is only implemented for dataset testing, and will not affect if you measure the time using the code above. --pin_memory does not matter as far as I know. The options depend on what you need. For basic use, you can just include --fix_res.

Best,
Xingyi

[veds12]

Thanks for the answer @xingyizhou ! It cleared most of the questions I had. I'll keep this issue open for some time in case I come up with some other questions while working on this and close it later.

[developer0hye]

Hi @xingyizhou,

What command do you use to measure the FPS on the below table?

Is the FPS on the table measured with flip augmentation?