https://doi.org/10.2352/EI.2022.34.7.

ISS-232
This work is licensed under the Creative Commons Attribution 4.0 International License.
To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

ESP32-CAM as a programmable camera research platform

Henry Dietz, Dillon Abney, Paul Eberhart, Nick Santini, William Davis, Elisabeth Wilson, and Michael McKenzie;
University of Kentucky; Lexington, Kentucky

Abstract
Experimenting with custom-programming of cameras can be
difficult. Most consumer cameras are protected to prevent users
from reprogramming them. Industrial cameras can be flexibly
controlled by an external computer, but are generally not stand-
alone programmable devices. However, various inexpensive cam-
era modules, designed largely to be used for building IoT (Inter-
net of Things) devices, combine extensive programmability with
a camera in a compact, low-power, module. One of the smallest
and least expensive, the ESP32-CAM module, combines a 2MP
Omnivision OV2640 camera with a dual-core 32-bit processor,
Figure 1. ESP32-CAM actual size: front and rear views
802.11 WiFi and BlueTooth as well as wired I/O interfaces, a mi-
croSD slot, low power modes, etc., all supported by the Arduino
programming environment and a rich collection of open source
libraries. Why not use it for programmable camera research? fer excellent programmable camera support via either Android or
This paper describes how the ESP32-CAM had to be adapted IOS, very similar to the API developed for the research platform
to enable use in a variety of experimental cameras. For exam- Frankencamera[3][4], but the cost is high for the image quality de-
ple, some of these cameras do not use the lens screwed and glued livered, and the cameras are essentially unalterable sealed subsys-
onto the OV2640, and replacing this lens revealed a number of is- tems. Compact cameras are much cheaper while offering some-
sues ranging from spectral response to adjustment of lens correc- what better image quality, but only the reverse-engineered open
tions. There are numerous strange interactions between different source Canon Hack Development Kit (CHDK)[5] offers the de-
functions that end-up sharing the same I/O pins, so work-arounds sired level of programmability. At this writing, CHDK supports
were needed. It also was necessary to devise ways to handle vari- 160 different low-end Canons, including most PowerShot mod-
ous higher-level issues such as implementation of a live view and els and a few EOS M series mirrorless bodies; for more than a
synchronization across cameras. However, the key problems have decade, CHDK has been one of the most viable approaches for
been resolved with open source software and hardware designs experiments involving reprogramming of cameras.
described here. Cameras designed to be tethered to a computer do not offer
programmability directly, but can be intelligently controlled by
Introduction a host computer program. Although various interfaces are avail-
A wide range of digital cameras are readily available, but able, most such cameras use USB, often leveraging the driverless
very few make it easy to conduct experimental camera research UVC (USB video class) protocol[6]. Consumer-oriented teth-
requiring custom programming. ered cameras are generally marketed as webcams, whereas those
The most obvious options would be high-end consumer cam- marketed as industrial or machine vision cameras tend to have
eras, especially mirrorless interchangeable-lens cameras using more robust physical construction, accept C-mount interchange-
MFT, APS-C, or full-frame sensors. Although most consumer able lenses, and may offer additional features. Due to the limited
cameras employ a variety of mechanisms to discourage users from market for industrial cameras, cost tends to be higher than for
reprogramming them, some open-source efforts have managed to consumer cameras with similar specifications.
reverse-engineer ways to support 3rd-party programming. Magic There have been a few commercial cameras designed for
Lantern[1] was successful in adding programmability and various programmability, but they tend to be relatively expensive. The
features to several Canon EOS DSLRs and the original EOS M DevCAM[7] is a very promising open source design incorporat-
mirrorless, but porting to new models is difficult, and none of the ing an FPGA to support research use of up to six MIPI sensor
latest models are supported. The OpenMemories[2] project en- modules, but it also is relatively expensive per camera.
ables 3rd-party software to run in the protected Android camera The alternative explored in the current work leverages the
app environment that Sony created for their PlayMemories apps. large market that has developed for inexpensive camera mod-
Sony supported that app environment in a wide range of models, ules intended to be used for building IoT (Internet of Things) de-
from digital video cameras to the high-end full-frame 42MP A7r vices. These camera modules often are paired with fairly substan-
II, but discontinued support of Android apps in all new cameras tial microcontrollers, and supported as platforms for both hob-
immediately after OpenMemories became viable. byist/experimenter use and IoT product development. On the
Cell phones and compact consumer cameras also would be hobbyist/experimenter-oriented high end are boards providing a
obvious alternatives using smaller sensors. Most cell phones of- full Linux environment, such as NVIDIA’s Jetson[8] and various

IS&T International Symposium on Electronic Imaging 2022

Imaging Sensors and Systems 2022 232-1
Table 1: Potential Camera Platform support of Key Attributes and Features

Attribute or Feature Mirrorless Webcam CHDK PowerShot AI-Thinker ESP32-CAM

Image quality ≥20MP,≥12bpp ≥0.3MP,≥8bpp 20MP, 12bpp 2MP (1600×1200), 10bpp
Exposure control Extensive Basic Extensive Basic plus some features
Interchange lens Yes Some models No Simple modification
Sensor size ≥17.3×13mm ≥2.4×1.8mm ∼6×4.5mm 3.59×2.684mm
Near infrared (NIR) Hard mod Some models Very hard mod Simple mod
Wired connectivity USB USB USB UART, SPI, & I2C
Wireless WiFi No Some models WiFi & Bluetooth (with BLE)
Tethered control Proprietary Yes, UVC Yes, CHDK PTP Yes, programmable
Autonomous operation Very limited No Yes Yes
Programmable display No No LCD Options via connectivity
Programming support No No CHDK C & Lua Arduino & Espressif C/C++
Processor Various ARM ? Dual 80MHz ARM Dual 240MHz Xtensa
Usable main memory Varies None Several MB 520KB SRAM & 4MB PSRAM
Flash memory SD card No SD card 4MB Flash & TF card
Power management Minimal No Minimal Modes from 310mA to 6µA
Sensor inputs Camera UI No Camera UI 9 I/O pins; ADC, I2C, & SPI
Control outputs Flash sync. No No 9 I/O pins; PWM, I2C, & SPI
Real time sync support Remote Some models RTC, USB detect RTC, programmable sync
Ease of embedding Very hard Moderate Hard Easy: 27x40.5x4.5mm board
Cost ≥ $500 $8 − $150 ≥ $100 ∼ $7

Raspberry Pi models[9]. On the low end are the various ESP32-

CAM[10][11] versions, such as the AI-Thinker version used in the
projects discussed here. Despite being one of the least expensive
modules at around $7, the ESP32-CAM combines a 2MP Om-
nivision OV2640[12] camera with substantial computing facilities
centering on a dual-core 32-bit processor supported by the Ar-
duino programming environment and a multitude of libraries. For
example, a demonstration application implements a WiFi camera
web server with real-time face recognition.
Since 1999, our research group has constructed many pro-
grammable cameras and camera arrays[13]. Table 1 gives a com-
pact comparison of relevant features of some camera systems that
we have used to construct programmable camera research plat-
forms. From the table, it is not surprising that most of our earlier
efforts used Canon PowerShot cameras under CHDK. However, Figure 2. KameraflY Senior Project Team and system demonstration
the combination of lower resolution for the OV2640 camera, rel-
atively powerful computing hardware resources, and extensive li-
brary support makes the ESP32-CAM capable of implementing
tem was to be as inexpensive as possible, with a target cost of un-
significantly more sophisticated custom programming.
der $25 per stand-alone camera. Yet, it was to be able to support
The remainder of this paper demonstrates the suitability of tightly synchronized capture using dozens of cameras. In fact,
ESP32-CAM for such research use by exploring three very dif- this description is essentially the charge that was given to the un-
ferent programmable camera systems our group has constructed dergraduate Senior Design team of Nick Santini, William Davis,
since 2019: KameraflY, KISS-E, and Lafodis. Elisabeth Wilson, and Michael McKenzie – shown in Figure 2
with the working system they built[14].
Precise synchronization is critical for multicamera applica-
tions, so the team built a LED synchronization tester and experi-
Wireless Multicamera mented. Synchronization proved easier than expected, with WiFi
The concept behind KameraflY was to create a very flexible synchronization of the cameras to 10ms – shorter than the elec-
synchronized wireless multicamera that would allow a swarm of tronic rolling shutter traversal time of an OV2640. The swarm
stand-alone camera modules to be freely positioned for panora- cameras were programmed to work with the Blynk IoT smart-
mas, 3D capture, Matrix-style time-frozen fly-bys, etc. The sys- phone app[15] for capture to a TF card, image upload via FTP,

IS&T International Symposium on Electronic Imaging 2022

232-2 Imaging Sensors and Systems 2022
Figure 3. KameraflY control app and ftp image gallery

Figure 5. KISS-E front and rear views

a wide angle view, but a 50mm lens essentially becomes 485mm,

giving very long effective focal lengths and high magnifications
with cheap and compact lenses. This is somewhat problematic
for hand-held use, because any camera shake is greatly magni-
fied, but there is a standard tripod 1/4-20 screw thread on the base
of KISS-E.
The electronics, wiring, and 3D-printed parts of KISS-E are
shown in Figure 6. Total parts cost was about $14. Unlike Kam-
Figure 4. Swarm of four stand-alone KameraflY modules eraflY modules, there is no custom board in KISS-E. Instead, the
3D-printed KISS-E body incorporates structures to mechanically
hold the ESP32-CAM, FT232RL, SSD1306 boardlets and shut-
and live streaming of video. A PHP web app allows browsing the ter button, which additionally can be secured using hot glue. In
images collected from the multicamera. The two apps are shown fact, 3D printers even can print custom circuit boards – they sim-
in Figure 3. ply cannot print wiring. Thus, combining 3D-printed mechani-
Although bare ESP32-CAM can be used with the software, cal mountings with electrical connections implemented by wire
making a stand-alone KameraflY module requires a bit more. wrap directly to boardlet pins provides remarkable design flex-
Making self-contained units able to run off either USB power ibility. Note that the rear OLED mount is a printed-assembled
or an internal rechargeable battery was perhaps the largest chal- hinged door that can be pivoted up for waist-level live view and
lenge, especially since battery safety was a priority. The 3D- also serves as an access to the camera internals, including the TF
printed housing and custom board the team made neatly pack- card. The small rectangular object on the 3D model plate is a tool
aged the ESP32-CAM with a charging module and a BP-70A for removing/inserting a TF card.
3.7v Lithium-Ion battery – a battery used in many Canon Power- The software written for KISS-E is very flexible, allowing a
Shot models. The KameraflY modules shown in Figure 4 actually mix of WiFi, USB, and manual control. Most significantly, KISS-
came-in well under budget, at a total cost of $13.43 each. E fully supports real-time live view. Live view is a feature digital
cameras generally provide, but there is actually some complexity
KISS-E Interchangeable-Lens Camera in managing the OV2640 to provide a fast frame rate live view
While KameraflY modules are standalone cameras, they are stream while allowing on-demand image captures at a higher res-
designed to be programmed to be either fully autonomous or re- olution. KISS-E can deliver the live view stream via WiFi and
mote controlled via WiFi. For example, there is no shutter button also via the built-in rear OLED display.
and the camera has no display to show images nor even camera The SSD1306 OLED display is bright and costs just $3, but
status. In contrast, KISS-E, shown in Figure 5, fully supports it is tiny, only 128 × 64 pixels resolution, and each pixel is strictly
tethered control via either USB or WiFi, but also implements the on or off with no colors nor gray levels. Despite those restric-
basic features needed to directly operate the camera. However, tions, we were able to write special software to produce the live
the most important aspect of KISS-E is that it is designed to ac- display seen in Figure 7. Of course, bigger and better displays
cept any of a wide range of interchangeable lenses. KISS-E stands are available, but in addition to cost, better displays would have
for Kentucky’s Interchangeable-lens Small Sensor – E mount; the implied significantly more compute overhead, lowering camera
OV2640 lens is removed and any lens adaptable to Sony E mount responsiveness. The OLED display shows:
may be used instead.
Why Sony E mount? Most small-sensor cameras with in- • KISS-E version (“210716” in Figure 7)
terchangeable lenses use C mount. However, Sony’s mirrorless • Percentage of available TF card space used... or, in Figure
cameras have made the larger-diameter E mount the most popu- 7, the fact that no TF card was present
lar target for adapting lenses, so inexpensive adapters for almost • The last part of the current WiFi address (.21) and a single
any lens (including C mount) are widely available. Compared to character indicating WiFi status (“-” means on, no request
full-frame 36 × 24mm Sony cameras, the tiny OV2640 sensor im- being processed)
poses a crop factor of approximately 9.7; that makes it hard to get • The current camera state (“live” meaning showing live view)

IS&T International Symposium on Electronic Imaging 2022

Imaging Sensors and Systems 2022 232-3
Figure 6. KISS-E component wiring and 3D printed parts

• A live histogram of pixel values

• An 80 × 60 pixel live view image produced by a special ren- Figure 7. KISS-E capture of Lafodis and corresponding OLED live view
dering algorithm that combines error distribution with edge
enhancement to make the scene as recognizable as possible
• A new type of focus quality indicator (below the live view):
the gap between two horizontal lines gets smaller as manual for providing power, ESP32-CAM programming, or teth-
focus becomes sharper ered control, it currently cannot support the driverless UVC
protocol to behave as a webcam
The focus quality indicator is particularly important because the • The ESP32-CAM as mounted in KISS-E relies on the an-
poor quality of the live view makes it insufficient to check focus, tenna built-into the boardlet, which easily can suffer inter-
but focus is critical with typical lenses due to the very thin depth ference if wire-wrap wires are routed over the antenna
of field. Empirically, this line-gap focus aid based on contrast • There are actually at least two different versions of the
detection in the center of the frame is very effective. OV2640 supplied with the ESP32-CAM. Although func-
A complication is that, not only does removing the (fixed tionality is very similar, the sensor is rotated 90◦ in one rel-
focus) native lens from the OV2640 make it possible to use a dif- ative to the other
ferent lens that requires manual focusing, but it also has the side • Removing the standard lens from the OV2640’s plastic
effect of removing the NIR-blocking coating that is on the native
lens. Without that filtration, the camera responds strongly to a full
spectrum of light starting in the NIR region. Figure 8 shows the
classic NIR-dominated colors of an outdoor scene shot without an
NIR-blocking filter. Although an appropriate filter can be fitted to
the front of a lens, large NIR filters are not cheap; the best alter-
native was thus to 3D-print a holder that can use 8 × 8mm 650nm
NIR-cut filters which cost under $1, as shown in Figure 8. These
filters work well, but it is significant that the holder covers the
edges of the filter glass, because they otherwise caused strange
flare/tinting of the image.
There are a number of other minor issues worth noting in the
construction of KISS-E:
• Although KISS-E provides a USB interface that can be used Figure 8. KISS-E image captured without NIR filter, 8 × 8mm filter mount

IS&T International Symposium on Electronic Imaging 2022

232-4 Imaging Sensors and Systems 2022
 Figure 10. Inside Lafodis: electronics and Herringbone gear drive

Figure 9. Lafodis large-format wireless IoT camera

screw mount is complicated by the fact that it is glued in

place
• The software is more convoluted than one might expect due
to multiple functions sharing I/O pins

In sum, KISS-E is a very poor replacement for a Sony E-

mount camera, but it is highly functional and fully programmable.
Figure 11. Circuit design for 3rd and 4th Lafodis prototypes
Lafodis Scanning Camera
Lafodis (LArge FOrmat DIgital Scanning) is a large-format
camera that can scan a 160mm diameter lens-projected image to four control signals to select which coils should be energized, and
resolutions of approximately 2.6GP. Originally intended purely as the ULN2003 driver simply provides Darlington transistor pairs
a test platform for research investigating novel scan orderings[16], to switch power.
it became clear that such a device could be useful to others. In the first two Lafodis prototypes, the 8 stepper control sig-
Lafodis has thus far evolved through four prototypes with the de- nals were directly taken from 8 I/O pins on the ESP32-CAM.
sign goal of making it easy for others to build this sub-$50 camera The catch is that the various restrictions on how pins can be used
to capture ≥500MP images from a 4x5” sensor area. The third meant that only 7 appropriate digital outputs were available on
prototype is shown in Figure 9. boardlet pins – a wire had to be soldered to the pad normally driv-
Unlike KameraflY and KISS-E, Lafodis does not just use ing the red LED on the back of the ESP32-CAM boardlet for the
the ESP32-CAM as a programmable camera, but also as a micro- eighth. Although that was fully functional, we felt that the sol-
controller to control the physical scanning. The ESP32-CAM is dering needed could discourage others from building Lafodis, so
mounted on a moving platform via a traceless 3D-printed circuit the third and fourth Lafodis prototypes use an entirely different
board mount, and it orchestrates the rotational and radial move- method: I2C communication with a second microcontroller dedi-
ment of that platform using a pair of 28BYJ-48 steppers with cated to stepper control.
ULN2003 driver boardlets. The stepper and driver boardlet com- The second microcontroller is an Arduino Pro Micro (actu-
binations cost just $3 each because they are a standard part widely ally, a clone which cost us $1.25; pricing is now closer to $7), and
used in HVAC, vending machines, etc. There are 32 full steps it conveniently also provides a USB connector that can be used to
per rotation of the motor, but internal gears drop the speed by a externally power Lafodis. The wiring diagram is shown in Figure
factor of 32/9 × 22/11 × 26/9 × 31/10 = 63.68395, giving ap- 11, and again all components are mechanically held by 3D-printed
proximately 2037.8864 full steps per rotation of the drive shaft. mounts and electrically connected using standard cables or wire
This gives the steppers substantial torque even when stepping at wrap – without needing a custom board. This I2C solution is very
a peak rate of over 1000 steps/second. Each step is done using scalable; not only does the second microcontroller provide many

IS&T International Symposium on Electronic Imaging 2022

Imaging Sensors and Systems 2022 232-5
more I/O pins, but the I2C network can be used to connect up All the work discussed in this paper, and additional materi-
to 128 devices. I2C devices we have used in other experimental als, will be linked into:
systems range from an AMG8833 thermal camera to an MPU-
9250 9-axis inertial measurement unit – as well as displays like http://aggregate.org/DIT/ESP32CAM
the SSD1306 OLED used in KISS-E. The only negative issue we
found in using I2C for Lafodis is that the electrically noisy envi- Note that the authors have no affiliation with any entities in-
ronment within the camera tended to cause (rare) transmission er- volved in producing nor selling any of the various versions of the
rors between the ESP32-CAM and Pro Micro. These errors were ESP32-CAM, nor does our research group speak for the Univer-
essentially eliminated by building a simple higher-level protocol sity of Kentucky in advocating use of this platform.
on top of the I2C transport to detect and correct errors.
Thanks to the I2C offloading, the third and fourth Lafodis
prototypes leave enough pins open on the ESP32-CAM to allow
for USB reprogramming of the processor by adding the circuitry References
shown as a ghosted image in Figure 11. However, the ESP32- [1] Magic Lantern, http://www.magiclantern.fm/ (accessed 1/28/2022)
CAM also allows for OTA (over the air) reprogramming via WiFi, [2] OpenMemories, https://github.com/ma1co/OpenMemories-Tweak
and all our Lafodis software takes advantage of that. The only (accessed 1/28/2022)
complication is that some chip resources must be reserved to sup- [3] Marc Levoy. “Experimental platforms for computational photogra-
port OTA, so the maximum application code size is reduced to phy,” IEEE Computer Graphics and Applications 30, no. 5, pp. 81-87,
1.9MB. 2010
[4] Andrew Adams, et al. “The Frankencamera: an experimental plat-
Conclusion form for computational photography,” Commun. ACM 55, 11
The construction of programmable cameras for research pur- (November 2012), DOI: 10.1145/2366316.2366339, pp. 90-98, 2012
poses requires an appropriate camera platform. The one-line sum- [5] Canon Hack Development Kit (CHDK),
mary of the findings in this paper is that cameras designed for http://chdk.wikia.com/wiki/CHDK (accessed 1/28/2022)
IoT applications generally, and the ESP32-CAM in particular, are [6] Hans van Antwerpen et al. “Universal Serial Bus Device Class Def-
a good match to typical research system requirements. A 2018 inition for Video Devices,” USB Implementers Forum, Inc., revision
paper[13] summarized the key requirements for multicamera re- 1.5, August 9, 2012
search systems as: [7] Dominique Meyer, Meher Birlangi, and Falko Kuester. “DevCAM:
• Programmable camera modules: it is difficult to find a An open-source multi-camera development system for embedded vi-
better-supported programming environment than the combi- sion,” Electronic Imaging 2021, Image Sensors and Systems 2021,
nation of Espressif ESP32-specific libraries and the Arduino January 20, 2021
infrastructure; there are over 50 ESP32-specific example ap- [8] NVIDIA, “Advanced AI Embedded Systems,”
plications in the Arduino programming environment (and https://www.nvidia.com/en-us/autonomous-machines/embedded-
dozens more generic Arduino application examples that also systems/, (accessed 1/28/2022)
can run on ESP32-CAM) [9] “Introducing the Raspberry Pi Cameras,”
• Synchronization of local clocks: in addition to a multitude https://www.raspberrypi.com/documentation/accessories/camera.html,
of hardwired mechanisms, ESP32-CAM can use NTP to (accessed 1/28/2022)
synchronize clocks with internet time references [10] Sara Santos, “ESP32 Camera Dev Boards Review and Comparison
• Local storage and processing: compared to the resolution of (Best ESP32-CAM),” https://makeradvisor.com/esp32-camera-cam-
the camera, the RAM and non-volatile storage available are boards-review-comparison/, (accessed 1/28/2022)
quite large, and the combination of the OV2640’s ability to [11] Random Nerd Tutorials. “ESP32-CAM Camera Boards: Pin and
do JPEG and other processing reduces load on the dual-core GPIOs Assignment Guide,” https://randomnerdtutorials.com/esp32-
32-bit 240MHz Tensilica Xtensa LX6 ESP32 processor cam-camera-pin-gpios/, (accessed 1/28/2022)
• Physical mounting and alignment: in addition to the designs [12] OmniVision. “OV2640 Color CMOS UXGA (2.0 MegaPixel)
detailed here, there are 259 open source 3D-printed designs CameraChipT M with OmniPixel2T M Technology,” Version 1.6,
on Thingiverse involving use of ESP32-CAM February 28, 2006
• Live view: although not directly provided by ESP32-CAM, [13] Henry Dietz, Clark Demaree, Paul Eberhart, Chelsea Kuball, and
we developed reasonable support for live view in the KISS- Jong Yeu Wu. “Lessons from design, construction, and use of various
E project – and now it can be used as a library for other multicameras,” Electronic Imaging 2018, Photography, Mobile, and
projects Immersive Imaging, pp. 182-1-182-10(10), 2018
• Fault tolerance: ESP32-CAM is cheap enough to have mul- [14] Nick Santini et al, “KameraflY ESP32-CAM Camera Swarm Sys-
tiple spares tem,” https://github.com/npsantini/KameraflY-ESP32-CAM-Camera-
Swarm-System, (accessed 1/28/2022)
Beyond those concerns, KameraflY, KISS-E, and Lafodis all [15] https://blynk.io/, (accessed 1/28/2022)
clearly demonstrate extensive support for connectivity – espe- [16] Henry Dietz and Paul Eberhart. “An Ultra-Low-Cost Large-
cially wireless – which makes many research tasks easier. Finally, Format Wireless IoT Camera,” Electronic Imaging, Imaging Sen-
Lafodis shows that the ESP32 is also a good general-purpose mi- sors and Systems, pp. 70-1-70-7(7), 2021 DOI: 10.2352/ISSN.2470-
crocontroller, able to implement a wide range of real-time control 1173.2021.7.ISS-070
and sensing tasks in addition to managing an OV2640 camera.

IS&T International Symposium on Electronic Imaging 2022

232-6 Imaging Sensors and Systems 2022