12/04/2021 Onnx Machine Learning in Production / Blog / codingforentrepreneurs.

com

Home Blog / Onnx Machine Learning in Production

Projects
Your email

Courses Get Started for Free

Blog
Trending

Topics
Install Python 3.8,
Virtual
Environments using
Messages
Pipenv, Django 3+
on Windows
Questions

Install Python 3.8,

Code
Virtual

Onnx Machine Learning in

Blocks
Environments using
Pipenv, Django 3+

Production
Ideas
on macOS

Light on September 11, 2020 · Justin Mitchel · AI · Deep Learning · Deployment · Keras · Machine OpenCV & Python:
Learning · onnx · Production How to Change
Resolution or
Rescale Frame
I recently had a project that I needed to use PyInstaller along with a Keras-trained
model. Unfortunately, PyInstaller and Keras only work some of the time.. as in, not that
reliable of a build. How to Create a
Custom Django User

What to do? Model

Onnx was the solution.

Install Tensorflow
My project didn't need to run training, it just need to run inference (prediction) and GPU on Windows
that's something onnx really excels at. using CUDA and
cuDNN

So, what is onnx?

On onnx.ai, it says: ONNX is an open format built to represent machine learning Discover
models. I read this as onnx is essentially a file format for ML models and onnx can run
those models. Create a Timelapse
with Python &
Almost any modern ML framework can become an onnx model. PyTorch -- yep.
OpenCV
Tensorflow -- yep. Keras -- yep. Caffe2 -- yep. Scikit-learn -- yep.

How to Implement
Why is this important? interoperability Django's Built In
Let's say I have a flask web app in production serving a ML model. Doing so, is easy Password
enough. Let's say the model I originally created this in was in Keras. Then, a new Management
member joins my team that is just a pro at PyTorch. How do I deploy these two models
on the same project? Large File Uploads
with Amazon S3 +
Well, you could package up both tensorflow and PyTorch for running inference but that
Django
starts to make our simple web app a bulky one and, more importantly, one that's
significantly more difficult to manage.
Setup React
onnx simplifies this problem by providing a standardized way to run models in
production. All you have to do is export your ML model to an onnx model. Once you Create a Standalone
have that, you can easily run it in production as you'll see below. React App

A real-world example
Featured
Recently, I was working on a Python project that needed to be compiled into a single
executable. For this, I used PyInstaller since it's a very reliable way to turn python into an Django vs Node.js
executable.

Setup React

https://www.codingforentrepreneurs.com/blog/onnx-machine-learning-in-production 1/4
12/04/2021 Onnx Machine Learning in Production / Blog / codingforentrepreneurs.com

Unfortunately, pyinstaller doesn't play nice will all packages and package types and How to Create a
containers (like docker) cannot (as far as I know) be compiled into a single binary. Custom Django User
Model
I was having all kinds of trouble getting PyInstaller and tensorflow to compile
correctly so I decided to give onnx a try. Not only did it work, but it worked incredibly
reliably. But what are Django
signals?
The post below will show you exactly how to convert a keras model
How to integrate
into a onnx one and then put it into production.
Python with Airtable

Step 1. Virtual Environment

Whenever you build a Python project, use a virtual environment of some kind. For this
guide, I'm solving a real-world problem I had with PyInstaller so I'm going to use the
following:

Python 3.7
venv (and not my preferred pipenv)

$ cd path/to/your/dev/folder Copy

$ mkdir cfe_onnx Copy

$ cd cfe_onnx
$ python3.7 -m venv .

Activate
Mac/Linux

source bin/activate Copy

Windows

.\Scripts\activate Copy

Step 2. Installations
pip install tensorflow keras2onnx onnxruntime numpy pillow Copy

tensorflow: our machine learning framework (but using tf.keras which is built-in
to tensorflow now)
keras2onnx: our conversion package
onnxruntime: how we run inference on onnx models in production
numpy: numerical python; common for dealing with arrays and matrices in Python
& ML Projects
pillow: the Python Image Library installer (PIL) which makes it easy to open
images within python.

Step 3. Export a Keras Model to an Onnx Model

As outlined in the keras2onnx docs, I'm going to just be using a pre-trained Keras model
for illustration purposes, change as needed:

from keras.applications.resnet50 import ResNet50 Copy

model = ResNet50(include_top=True, weights='imagenet')
model.save("model.h5")

Now run the conversion

onnx_model = keras2onnx.convert_keras(model, model.name) Copy

keras2onnx.save_model(onnx_model, 'model.onnx')

For PyTorch you can easily export a model too by using PyTorch's
built-in method outlined here

https://www.codingforentrepreneurs.com/blog/onnx-machine-learning-in-production 2/4
12/04/2021 Onnx Machine Learning in Production / Blog / codingforentrepreneurs.com

After this is done, you will have to saved models:

model.h5 (keras)
model.onnx (onnx)

I recommend keeping a keras-saved model for future resumable training. onnx can be
converted but I don't use the extra step if I don't need to.

Step 4. Prepare for Production

The above model is for Image Classification (aka imagenet), so we have to be sure we
prepare our data prior to running inference.

Preprocessing
# preprocessing.py Copy
import numpy as np
from PIL import Image

def process_image(image_path, height=150, width=150):

'''
This method opens an image and converts it into a normalized
array that represents the image.
'''
image = Image.open(image_path)
image = image.convert("RGB")
new_image = image.resize((width,height))
np_image = np.asarray(new_image)
min = np_image.min()
max = np_image.max()
# normalize to the range 0-1
np_image = np_image.astype('float32')
np_image -= min
np_image /= (max - min)
return [np_image]

Response Encoding
Below is a json encoder that converts numpy data types.

# encoding.py Copy
import numpy as np
import json

class NumpyEncoder(json.JSONEncoder):
""" Special json encoder for numpy types """
def default(self, obj):
if isinstance(obj, np.integer):
return int(obj)
elif isinstance(obj, np.floating):
return float(obj)
elif isinstance(obj, np.ndarray):
return obj.tolist()
return json.JSONEncoder.default(self, obj)

Here's a couple ways to use this encoder:

**With Json Dumps

data = {"preds": np.array([0.87, 0.13])} Copy

json.dumps(data, cls=NumpyEncoder)

In Flask

app = Flask(__name__) Copy

app.json_encoder = NumpyEncoder

@app.route('/numpy')
def get_numpy_response():
data = {"preds": np.array([0.87, 0.13])}
return jsonify(data)

https://www.codingforentrepreneurs.com/blog/onnx-machine-learning-in-production 3/4
12/04/2021 Onnx Machine Learning in Production / Blog / codingforentrepreneurs.com

Step 5. Predictions with Onnx

# predict.py Copy
import json
import pathlib

import onnxruntime

from .encoding import NumpyEncoder

from .preprocessing import process_image

ONNX_SESSION = None

def get_session():
global ONNX_SESSION
if ONNX_SESSION == None:
model_path = str(pathlib.Path("model.onnx"))
sess = onnxruntime.InferenceSession(model_path)
ONNX_SESSION = sess
return ONNX_SESSION

def predict(img_path, use_array=False, *args, **kwargs):

onnx_sess = get_session()
sess_inputs = onnx_sess.get_inputs()[0]
input_name = sess_inputs.name
shape = sess_inputs.shape
im = process_image(img_path, height=shape[1], width=shape[2])
inference_preds = onnx_sess.run(None, {input_name: im}) # this is where
the inference_happens
results = inference_preds[0][0]
data = {str(k):v for k,v in enumerate(results)}
return json.dumps(data, cls=NumpyEncoder)

Next Steps
Now, you just need to take all of the above information and turn it into a webapp or
add it to a local python project. I'll leave that to you.

Good luck!

Write a comment

About · Blog · Suggestions · Contact · Connect · Coming Soon

© Copyright 2021 · codingforentrepreneurs.com

https://www.codingforentrepreneurs.com/blog/onnx-machine-learning-in-production 4/4