A Buffer Overflow Detection and Defense

Liu et al.
Cybersecurity (2023) 6:45 Cybersecurity

https://doi.org/10.1186/s42400-023-00164-x
RESEARCH Open Access
A buffer overflow detection and defense

method based on RISC‑V instruction set
extension
Chang Liu1,2 , Yan‑Jun Wu1,3*, Jing‑Zheng Wu1,3 and Chen Zhao1,3
Abstract
Buffer overflow poses a serious threat to the memory security of modern operating systems. It overwrites the con‑
tents of other memory areas by breaking through the buffer capacity limit, destroys the system execution environ‑
ment, and provides implementation space for various system attacks such as program control flow hijacking. That
makes it a wide range of harms. A variety of security technologies have been proposed to deal with system security
problems including buffer overflow. For example, No eXecute (NX for short) is a memory management technology
commonly used in Harvard architecture. It can refuse the execution of code which residing in a specific memory,
and can effectively suppress the abnormal impact of buffer overflow on control flow. Therefore, in recent years,
it has also been used in the field of system security, deriving a series of solutions based on NX technology, such
as ExecShield, DEP, StackGuard, etc. However, these security solutions often rely too much on the processor archi‑
tecture so that the protection coverage is insufficient and the accuracy is limited. Especially in the emerging system
architecture field represented by RISC-V, there is still a lack of effective solutions for buffer overflow vulnerabilities.
With the continuous rapid development of the system architecture, it is urgent to develop defense methods that are
applicable to different system application environments and oriented to all executable memory spaces to meet
the needs of system security development. Therefore, we propose BOP, A new system memory security design
method based on RISC-V extended instructions, to build a RISC-V buffer overflow detection and defense system
and deal with the buffer overflow threat in RISC-V. According to this method, NX technology can be combined
with program control flow analysis, and NX bit mechanism can be used to manage the executability of memory
space, so as to achieve a more granular detection and defense of buffer overflow attacks that may occur in RISC-V
system environment. In addition, The memory management and control function of BOP is not only very suitable
for solving the security problems in the existing single architecture system, but also widely applicable to the combina‑
tion of multiple heterogeneous systems.
Keywords RISC-V, Operating system security, Buffer overflow, Control flow hijacking, NX bit, Xibop
*Correspondence:
Yan‑Jun Wu
yanjun@iscas.ac.cn
Full list of author information is available at the end of the article
© The Author(s) 2023. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which
permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or
other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line
to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory
regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this
licence, visit http://creativecommons.org/licenses/by/4.0/.
Liu et al. Cybersecurity (2023) 6:45 Page 2 of 16
Graphical abstract
Introduction In attack scenarios such as program control flow hijack-

Buffer overflow attack (One 1996) is a typical attack ing, the main goal of buffer overflow attack is to guide the
method to threaten the memory security of operating control flow to the incorrect code for execution. These
system. By making use of the loopholes such as bound- codes are located outside the Control Flow Graph (CFG
ary check lacking in the source program written by the for short). Some are external malicious codes prepared
memory insecure language, it can break through the by attackers, which will break the system from normal
buffer capacity limit, cover the data content in other work flow after execution and act directly according to
memory regions, and then destroy the integrity and cor- the attacker’s intentions, such as code injection attacks
rectness of a program. Due to low implementation diffi- (Pan et al. 2021). Some codes that exist in the system but
culty and convenient operation, buffer overflow attacks should not be executed, they will cause the system to
have developed rapidly, and their influence and harmful- process program data incorrectly after execution, such
ness have become increasingly prominent. According to
the statistics of China National Vulnerability Database
of Information Security (CNNVD for short) (CNNVD Table 1 CNNVD vulnerability statistics (2022)
2022), in all the 12 months of 2022, the total number of No Month Collection Buffer Errors Percentage Rank in all
security vulnerabilities has exceeded 24,800, including (%) Error Types
more than 2600 buffer errors, accounting for more than
1 2022.12 2173 248 11.41 2
10% (Table 1). The buffer overflow attack has the charac-
2 2022.11 1922 167 8.69 2
teristics of rewriting the memory content, which makes
3 2022.10 2001 265 13.24 1
it widely used in attacks such as program control flow
4 2022.9 2133 325 15.24 1
hijacking, and further increases the security threat to the
5 2022.8 2240 291 12.99 1
system. For example, the return oriented programming
6 2022.7 1924 215 11.17 1
(ROP) attack uses the buffer overflow to cover the correct
7 2022.6 2346 197 8.40 3
function return address information, hijacks the con-
8 2022.5 2204 195 9.54 2
trol flow and jumps to the code fragment designated by
9 2022.4 2098 190 9.06 3
the attacker when the program executes function return
10 2022.3 2065 208 10.07 2
codes, thus completing the malicious operation(Wang a
11 2022.2 1726 105 6.08
et al. 2019). Therefore, academia and industry have been
12 2022.1 2054 210 10.22 3
actively looking for effective methods to deal with buffer
Total 24,886 2616 10.51
overflow attacks, so as to curb and mitigate the adverse
a
impact of buffer overflow on system security as much as The monthly report of this month has not been published. The data of this
month is the approximate value accumulated from all weekly reports of this
possible. month, and the cumulative range is from January 31, 2022 to February 27, 2022
as code reuse attack (Zhao et al. 2021). In this regard, overflow and control flow hijacking. Developing the
some defense methods based on non executable mem- RISC-V architecture oriented system memory security
ory have been proposed. For example, OpenBSD imple- defense technology is the inevitable course to promote
mented the W ^ X mechanism (Wikipedia 2022) in 2003, the further development of RISC-V architecture.
so that every page in the address space of a process or To this end, we have done the followings:
kernel space cannot be writable and executable at the
same time. Red Hat put forward the defense mechanism 1. We analyzed the characteristics of existing security
Exec Shield (Molnar 2003) for Linux system in 2004 and schemes, and summarizes the relevant key points
it has been used ever since. The Exec Shield divides the and basic design ideas of system memory security
memory space of a process into executable and non exe- defense in RISC-V architecture based on the require-
cutable segments. It uses the Segment Limit Approach ments of RISC-V architecture itself.
to approximately separate the read and execution per- 2. We designed a set of RISC-V memory security
missions to ensure that the return address only points related instructions, which is called the Xibop exten-
to trusted code. Microsoft also designed a "Data Execu- sion. Based on this extension, a buffer overflow
tion Prevention" (DEP for short) security mechanism for detection and defense system for RISC-V architec-
Windows system (Gao et al. 2013). DEP tracks and moni- ture is proposed, which is called BOP method.
tors the portion of system memory which used to store 3. Around BOP method, we also discussed the possibil-
instructions, and when an application program attempts ity of supporting multiple security mechanisms and
to execute in a memory area which has been designated establishing RISC-V memory security defense sys-
as non-executable, closes it and issues a notification. tem. By using the BOP method, we can fully leverage
However, the existing solutions rely heavily on the the advantages of RISC-V architecture and achieve
hardware environment and system characteristics, which better software hardware collaboration when deal
limits their protection scope. For example, Exec Shield is with security problems.
designed for × 86 architecture, and the segment restric-
tion method it depends is also a fuzzy feature of Intel The content of this paper is arranged as follows:
processor; DEP is only applicable to Windows platforms, Sect. "Introduction" introduces the buffer overflow secu-
and Intel has also developed Execute Disable Bit (EDB rity challenges faced by RISC-V architecture; "Research
for short) memory protection technology for their pro- background" analyzes some existing buffer overflow
cessors to coordinate and support the implementation of prevention methods; Sect. "Memory security in RISC-V
DEP functions. Without × 86 architecture and Intel pro- architecture" briefly summarizes the basic idea of RISC-
cessors, these security solutions will be powerless. There- V architecture for memory security defense; Sect. "Xibop
fore, for many non- × 86 operating systems, we still need instruction set extension for RISC-V memory secu-
to continue to explore new ways to eliminate threats. In rity" focuses on the Xibop instruction set extension;
order to achieve this goal, researchers have started some Sect. "BOP Method and its Implementation" discusses
new attempts, combining other technologies (such as the specific implementation of the BOP method in the
StackGuard and SSP (Cowan, et al. 1998)) or adopting system; Sect. "Discussion" discusses the further work
new technology application methods (such as CHERI required for BOP method to support multiple security
(Davis et al. 2019)), but there are also problems such as mechanisms and establish RISC-V memory security sys-
limited application scenarios or ease of use, and there is tem; Sect. "Conclusion" summarizes the full text.
still room for improvement.
RISC-V is a new open source RISC architecture that Research background
conforms to the design requirements and architecture In order to suppress the buffer overflow attacks that sys-
development trend of modern information systems. tem may encounters, the academic and industrial circles
Because of its characteristics of openness, freedom, light have put forward a variety of coping technologies and
weight, convenience, low cost of learning and promo- measures, including the use of memory non-executable
tion, strong portability and customization, it is highly (NX for short) attribute, Address Space Layout Rand-
suitable for emerging intelligent devices and a variety omization (ASLR for short), Canary, and Control Flow
of intelligent scenarios. In recent years, it has gradually Integrity (CFI for short). Among them, using memory
become a major research hotspot in the field of system non-executable attribute is a typical technology that
architecture, and is expected to become the mainstream appeared earlier and has been developed and practiced
choice for customizing the next generation operating sys- in a variety of specific systems. The memory security
tem. However, the operating systems based on RISC-V schemes, including Exec Shield, DEP, etc., all use NX
architecture also face security challenges such as buffer as their technical basis, but all depend on the specific
Fig. 1 Memory layout of exec shield process

Fig. 2 Principle of Canary technology (Cowan et al. 1998)
hardware environment or system features.In addition,

there are also some efforts (StackGuard, SSP, CHERI, Data Execution Protection (DEP) is a security feature
etc.) to adopt other technologies or technology appli- that helps to protect your computer from viruses and
cation methods. Although they need to be improved, other security threats. Harmful programs may attack
they have also played a certain role in the follow-up Windows by attempting to run (also known as "execute")
exploration. code reserved in the computer’s memory for Windows
and other authorized programs. These types of attacks
Exec shield can damage your programs and files.
Exec Shield (Molnar 2003) is a security technology for DEP can help to protect your computer by monitor-
Linux x86 kernel, which was disclosed by Red Hat in ing programs to ensure that they use computer memory
2004 to prevent stack, buffer or function pointer over- safely. If DEP notices that a program on your computer
flow. In the x86 architecture, there is no difference in the is using incorrect memory, it will close the program and
permissions to read or execute code from a part of mem- notify you.
ory, as a result, Exec Shield needs to distinguish between In actual operation, DEP marks the memory location
the two. So Exec Shield tracks the executable mapping which only contains data as NX (non-executable); If an
specified by the application, and maintains a "maximum application attempts to execute code from a memory
executable address" value based on this, forming a "seg- location marked NX, it will be blocked by DEP to achieve
ment limit", thus dividing the memory address into an protection. In terms of implementation, the Windows
executable and a non-executable parts, approximately system automatically adds a group of special pointers to
separating the read and execute permissions (Fig. 1). By the data objects stored in the memory on the software
this setting, ensure that all program codes are on the level, and the EDB memory protection technology devel-
low address side of the limit, and all data are on the high oped by Intel processor is used to cooperate on the hard-
address side; When the program violates the execution ware level.
authority and crosses the segment limit, a segmentation DEP has a good defense effect when against code
error will be triggered to terminate. injection type attacks; However, because only data is
The "one size fits all" approach of Exec Shield makes protected, it is easy to be bypassed by code reuse type
its defense effect completely depend on whether the attacks. Also, the buffer overflow protection provided by
"segment limit" is properly selected. The memory area DEP has some side effects, that is, the affected applica-
near its boundary may lose protection due to inaccurate tions are often suspended. Frequent triggering may lead
estimation of "segment limit". In addition, this practice to other types of attacks, such as denial of service attacks.
requires that the memory layout must be arranged in
strict accordance with the rule of "executable code at low StackGuard and SSP
address, data and other content at high address", which StackGuard (Cowan et al. 1998) is a specific memory pro-
has a serious dependency on the system itself, even hard- tection mechanism provided by GCC compiler, which
ware, and limits the application scope of Exec Shield. uses Canary technology to protect stack security. Canary
is a small special character set between the buffer and
DEP control information such as EBP. When a buffer overflow
DEP (Gao et al. 2013) is a data execution protection ser- occurs, this character will be overwritten and destroyed
vice developed by Microsoft for the Windows platform to first, as shown in Fig. 2. Therefore, it is only necessary to
prevent the execution of malicious inserted code. In the detect whether the value of Canary has changed before
"Help and Support" function of Windows system, we can the jump execution to determine whether an overflow
find Microsoft’s official description of DEP: has occurred.
Fig. 3 Extract from the official document of system call mprotect
SSP (Stack Smashing Protection) is a further develop- page boundary, so it cannot accurately specify a memory
ment of StackGuard. It strengthens the storage security region at any location for protection. In addition, as a sys-
of Canary words and provides a variety of methods to tem call, mprotect can only function on Linux systems,
generate Canary words. But in principle, it is similar to and cannot be directly used in other non-Linux system
StackGuard. environments.
As its name describes, StackGuard and SSP only pro-
tect against stack overflow and do not pay special atten- CHERI
tion to heap space and other overflow problems. In CHERI (Davis et al. 2019) is a system security research
addition, if the area where the Canary word is located is jointly completed by Cambridge University, Stanford
managed to bypass (such as writing out of bounds), or Research Institute and other institutions. It aims to
the Canary value is disclosed, the defense effect of the achieve fine-grained memory protection and highly scal-
application cannot be achieved. able software partitioning, and significantly improve
system security. CHERI proposes a hardware supported
mprotect data format "architecture capability" for representing
mprotect is a system call provided by the Linux operat- integers and pointers in memory insecure languages,
ing system to set the protection properties of a memory and protecting underlying data addresses. "Capability"
region. When a user wants to modify a protection prop- consists of an integer address and a same size metadata,
erty (readability, writability, executability, etc.) of a mem- and is associated with a 1-bit validity tag in a register or
ory region, he can specify the starting address addr of memory (Fig. 4). Base on the content of the metadata and
the corresponding memory page for the region, the total the validity tag, hardware can then control the operations
length len from addr to region ending, and the property to the address so that the security can be guaranteed.
value prot. And then, call mprotect(addr, len, prot) to However, in order to support its new concept of
complete the modification. The description of the sys- abstract capabilities, it involves the adjustment and mod-
tem call can be found in the official Linux documenta- ification of a variety of software and hardware, and there
tion through the Linux system command man mprotect is still much room for improvement in ease of use and
(Fig. 3). other aspects.
The document states that the starting address of the
region protected by mprotect must be aligned to a certain
Fig. 4 128-bit capability for 64-bit address in CHERI (Watson et al. 2019). The permissions(perms), object type(otype), and authorization
bounds(bounds) are jointly formed the metadata part of this capability
Fig. 5 The structure of RISC-V PMP configuration register (Waterman et al. 2023). The R, W, and X fields indicate the readability, writability,
and executability settings of the target address, respectively. The A field indicates the address matching mode, and the L field indicates
whether the PMP entry is locked and becomes not writable
PMP to match it. This shows, on the one hand, that solving the
Physical Memory Protection (PMP) is a fundamental memory security problem in RISC-V often requires start-
security mechanism introduced by the RISC-V Privileged ing from scratch, with considerable space for technology
Instruction Set Specification (Waterman et al. 2023). It is implementation; On the other hand, it also means that
used to limits the physical addresses which are accessi- to solve the RISC-V memory security problem, it is usu-
ble by software running in hardware threads (harts). PMP ally necessary to comprehensively consider the require-
combines an 8-bit configuration register (pmpcfg) and ments and capabilities of both software and hardware,
an MXLEN bit1 address register (pmpaddr) into a PMP and realize the defense scheme with software-hardware
entry to record and manage the protection properties of a collaboration.
physical memory region. The A field of the configuration Secondly, RISC-V, as a reduced instruction set archi-
register (Fig. 5) specifies a method that determines the tecture, has the advantages of controllable cost, port-
physical memory address range to be protected through ability and customization, and has good applicability in
the corresponding address register. heterogeneous, modular, functional specialization and
PMP is a relatively basic protection measure applied to other application scenarios. Therefore, systems based
physical addresses, so it often requires preparation work on RISC-V architecture may have multiple memory
such as address translation and entering the M privilege environments, that are significantly different in address
level in advance. Moreover, PMP requires that the pro- space size, growth direction, available addressing meth-
tected memory addresses must be physically continuous. ods, segment distribution, address read/write/executable
These all imply the feature that PMP may need to be used characteristics, and supported permission modes, and
in conjunction with other processing logic. are prone to expose a larger attack surface. Due to this
reason, the memory security defense scheme of RISC-V
Memory security in RISC‑V architecture is required to be either universal to a certain extent, or
Main features comprehensive with multiple specific mechanisms for
RISC-V is a new RISC architecture. Compared with other various memory environments, with sufficient hierarchy
existing architectures, RISC-V has its unique features in or multi-dimensional property.
memory security problems. In addition, RISC-V, as a reduced instruction set archi-
First of all, RISC-V, as an emerging architecture, lacks tecture, has stronger regularity in its instructions. It
enough mature hardware and software security solutions has strict specifications in instruction format, instruc-
tion alignment, etc., which greatly reduces the design
complexity of hardware support, improves the decod-
1
A generic representation of the register bit width under M privilege level. ing efficiency, and reduces the time, space, and energy
This can be 32 or 64 bits depending on the actual situation.
costs when completing instruction functions. On the one 1. According to each time that the threats occur and
hand, it simplifies the specific implementation of RISC-V develop, deploy targeted defense mechanisms
defense scheme to a certain extent, while it also implies respectively to form a multi-level comprehensive
a layer of constraint: the implementation of RISC-V software-hardware coordinated defense system. For
defense scheme should be as consistent as possible with example, in the process of program development,
the original style of the system; On the other hand, when use defensive programming and other means, such as
it is really necessary, RISC-V’s energy efficiency features manually checking the array bounds, to avoid obvi-
can be used to balance the negative impact of inefficient ous security risks from the code text level; In the pro-
solutions that other architectures usually can’t bear, mak- cess of program compilation and testing, the relevant
ing the design and implementation of RISC-V defense tool chain can effectively supervise and control, to
solutions more optional. extract the possible overflow threat information and
At the same time, RISC-V will also show some com- deal with it; In the program execution phase, RISC-V
monalities with other architectures in some aspects when security hardware uses a variety of security mecha-
dealing with memory security threats. For example, the nisms to protect the buffer that may be threatened,
basic principle of buffer overflow attack is to illegally and so on.
overwrite other data by breaking the buffer capacity 2. Based on the RISC-V instruction set specification,
limit. Therefore, whether in RISC-V or other architec- use special custom instructions to constrain the spe-
tures, buffer boundary must be one of the key points of cific implementation of RISC-V security hardware.
defense. This commonality makes the design of RISC-V On the one hand, it helps the security defense scheme
memory security defense scheme still able to draw inspi- to make full use of the native features of RISC-V and
ration from similar schemes of existing architectures and maintain the RISC-V system style. On the other
absorb some of their advantages for own use. hand, it provides an unified adaptation method for
different memory environments, which improves the
Basic ideas universality of the security defense scheme. In addi-
According to the main characteristics of the memory tion, the RISC-V tool chain can also be targeted and
security issues in RISC-V architecture above, the RISC-V optimized accordingly to better achieve software-
memory security defense scheme can have the following hardware collaboration.
basic design ideas:
1. Based on the characteristics of RISC-V architecture Xibop instruction set extension for RISC‑V memory
itself, and with RISC-V instruction set as the link, to security
design the defense scheme of software-hardware col- Around the basic ideas proposed in Sect. "Basic ideas",
laboration from both the software and the hardware we hope to build a memory security defense system
sides together. based on the RISC-V instruction set, which combines
2. The method of combining multiple security mecha- hardware and software in the RISC-V architecture, to
nisms is adopted to design and implement a multi- mitigate the threat of buffer overflow attacks on the
dimensional defense scheme from several different RISC-V architecture. To this end, we designed a set of
levels for various memory environments that may RISC-V instructions related to memory security, called
exist in the target system. Xibop instruction set extension. The Xibop extension
3. Prefer the design based on RISC-V’s native character- includes some custom instructions to support various
istics and keeping the original style of the system, and memory defense methods, such as ALSR (Marco-Gisbert
use the existing resources and mechanisms of the et al. 2019), Canary (Krerk et al. 2008), O-CFI (Mohan
system effectively to solve specific security threats. et al. 2015), etc. In this article, we focus on using the
However, when necessary, different external software Xibop extension to implement the defense mechanism of
and hardware modules can also be introduced to non-executable memory.
support it.
4. On some common problems of memory security, Design overview
we can absorb the advantages of similar schemes of Non-executability is a feature of storage media, including
existing architectures and make them fit for RISC-V. memory, which means that the corresponding content
can only be used for data access, not for code execu-
Specifically, the following ideas can be followed tion. When existing systems implement this feature, they
when solving the buffer overflow problem in RISC-V often need to maintain special tag data to control the
architecture: non-executability of certain memory areas. When a
Fig. 6 Encoding format of Xibop extended instructions
Fig. 7 Format of nxset instruction, where opcode is 0101011, indicates that the instruction belongs to Xibop; instruction identify code funct7
is 0000100, means this is a nxset instruction; field funct3 is 000
piece of memory is marked as non-executable, accessing field is used to indicate the type distribution of operands.
to any location within the memory range will throw an However, in the current design, only 32-bit register type
exception. operands are used, so the funct3 field is fixed to 000. As
Therefore, the core behavior of the defense mechanism shown in Fig. 6.
based on non-executable memory is to mark the non Xibop extension is designed as an XLEN independent
executable of a specified memory range. To this end, the extension, that is, it can work in both RV32 and RV64,
Xibop extension introduces the nxset instruction, which or even RV128 in the future. When it comes to 64-bit
is used to set the non-executability of the target mem- or wider architecture, funct3 field should play a role in
ory. Depending on the settings, the target memory will explaining how to use the operands.
become non-executable or revert to an executable state. For the privilege mode of instructions, Xibop exten-
The Xibop extension also introduces the nxcheck sion is designed for the non-privilege level use and works
instruction, which is used to detect the executability of under U mode in principle to achieve the similar effect as
the target memory. This instruction is not necessary, but S privilege level. However, when implementation, the M
it can provide a certain degree of convenience in some privilege level can also be used according to actual needs
scenarios (such as testing). In addition, in order to facili- (for example, in some simple system environments that
tate application, Xibop also introduces some pseudo do not support the U mode).
instructions: setnx and clrnx, which are used to turn
on and off the non-executability of the target memory, Instruction nxset
respectively, and they will eventually be interpreted as The nxset instruction of the Xibop extension is used to
nxset instructions; setnxr and clrnxr, are respectively set the non-executability of the memory at the specified
used to enable and clear the non-executability of a con- address. Its syntax is: nxset rd, rs1, rs2. Among them, rd
tinuous target memory range, and they will eventu- is the destination register for storing instruction execu-
ally be interpreted as a microprogram containing nxset tion results, and you can judge whether the non-execut-
instructions. able is correctly set according to whether the value of rd
In the aspect of instruction coding, Xibop extension is zero. rs1 is an operand register that stores NX values.
uses the R-type encoding format to divide the instruc- A value of 1 indicates that the target memory is set to be
tions into six fields: opcode, rd, funct3, rs1, rs2, and non-executable, and a value of 0 indicates that the tar-
funct7. The Xibop extension uniformly uses 0101011 as get memory is set to restore the executability. rs2 is the
the opcode (opcode field) of instructions to occupy the operand register that stores the target memory address,
minimum space for custom opcodes; The funct7 field is indicating that the corresponding operation will be per-
used as the instruction identifier to distinguish differ- formed on the address. The encoding format of the nxset
ent instruction functions. The rd field, rs1 field and rs2 instruction is shown in Fig. 7.
field represent the parameters of the instruction, which In terms of hardware, the nxset instruction can be
are respectively the destination register, the first oper- implemented as shown in Fig. 8.
and register and the second operand register. The funct3
Fig. 8 An implement method of nxset instruction in RISC-V security hardware. Left part describes the instruction decoding process. opcode, funct3
and funct7 fields are combined to set the enable signal(en). When enabled, nxval from rs1 will be stored to a place corresponding to the value
of addr-reg from rs2. res is set according to the execute result and is returned
Fig. 9 Format of nxcheck instruction. similiar with nxset, the opcode is 0101011, which means the instruction belongs to Xibop extended
instruction set; instruction identify code funct7 is 0000101, to tell this is a nxcheck instruction; field funct3 is 000
Instruction nxcheck of the target memory. This instruction will be interpreted

The nxcheck instruction provided by Xibop is used to as an instruction: nxset rd, 1, addr.
detect the executability of the target memory. Its syn- Similarly, the clrnx pseudo instruction of the Xibop
tax is: nxcheck rd, rs1, rs2. Among them, rd is the tar- extension is used to set the target memory to be execut-
get register for storing the target memory’s executability. able. Its syntax is: clrnx addr, and it will be interpreted as
We can judge whether the target memory is not execut- an instruction: nxset rd, 0, addr.
able according to whether the value of rd is 1. rs1 and rs2
are operand registers used to form the target memory Pseudo instruction setnxr and clrnxr
address. Generally, rs1 stores the base address of the tar- The setnxr pseudo instruction extended by Xibop
get memory address, while rs2 stores the correspond- is used to set a continuous target memory range as
ing address offset. The encoding format of the nxcheck non-executable. Its syntax is: setnxr from, to. Where,
instruction is shown in Fig. 9. from represents the low address boundary of the tar-
In terms of hardware, the nxcheck instruction can be get memory range, and to represents the high address
implemented as shown in Fig. 10. boundary of it. This instruction will be interpreted as a
microprogram in Fig. 11a.
Similarly, the Xibop extended clrnxr pseudo instruc-
Pseudo instruction setnx and clrnx tion is used to set a continuous target memory range
The setnx pseudo instruction of the Xibop extension is as executable. Its syntax is: clrnxr from, to, and will be
used to set the target memory to be non-executable. Its interpreted as the microprogram in Fig. 11b.
syntax is: setnx addr. Where addr represents the address
Fig. 10 An implement method of nxcheck instruction in RISC-V security hardware. Left part describes the instruction decoding process. opcode,
funct3 and funct7 fields are combined to set the enable signal(en). When enabled, read the nxval as result from the store place corresponding
to addr-reg. If error occurs, nxval will be the error code
BOP method and its implementation

Based on the Xibop extension, we propose a buffer
overflow detection and defense system for RISC-V
architecture, called BOP method. This method can be
Fig. 11 Explanation of setnxr and clrnxr pseudo instructions. Here
used to implement specific memory security defense
". L2" is a jump tag, which may have different names in specific mechanisms such as those based on No eXecute Bit
implementations (NX bit).
Fig. 12 Overall architecture of BOP design

Fig. 13 Main workflow of NX bit based non-executable memory defense mechanism
Overall architecture manipulating NX bits, program code can be prohibited

The BOP method is the synergy of several hardware and or allowed to execute from a specific memory range,
software memory defense mechanisms. Its main idea is thus constraining the direction of control flow. Figure 13
to take targeted defense in turn on multiple links that shows the main workflow of this mechanism.
may cause memory security threats, and combine all the
memory defense mechanisms to cooperate with each Support from software side
other. The defense mechanism in each link can not only BOP is a security method based on RISC-V Xibop
complete its own functions, but also provide information extension instruction set. Therefore, compiler, assem-
for subsequent links as much as possible to improve the bler and other tools need to be modified accordingly
overall defense capability and efficiency. Figure 12 out- so that the method can be correctly recognized and
lines the overall design architecture of the BOP method. accepted by the system. A feasible method is to modify
According to actual needs, resource constraints and the official tool chain RISCV-GNU-TOOLCHAIN pro-
other factors, the BOP method can load any security vided by RISC-V, add the description information of
defense mechanism it supports. As for the defense mech- Xibop extension instructions, and insert BOP related
anism of non-executable memory, BOP uses NX bit to instructions according to the code characteristics in the
control the executability of system memory. NX bit is compilation phase, so as to set or clear specific NX bits
a flag bit used to identify the executable permission of as required. Algorithm 1 describes the insertion pro-
memory space, which is mostly in page table entries. By cess of the instruction nxset.
In this process, the BOP first analyzes the control flow the same time, the decoding logic of the processor is
graph of the program in the compile period, finds out the modified to enable it to recognize and execute various
code fragments to be protected, such as function jump functions of Xibop extended instructions. The specific
(involving the stack space), malloc (involving the heap transformation method is consistent with the design idea
space), and determines the memory space associated (Fig. 5, Fig. 7, etc.) mentioned in Chapter 4.
with this fragment that should not be executed. BOP will The BOP module is used to implement various BOP
record the symbols which can describe the boundary of memory security defense mechanisms at the hardware
that memory space first, and update these symbols with side. The non-executable memory defense mechanism
their address value later in the link period. Then, before mainly involves memory access request filtering, NX bit
entering this segment, insert the nxset instruction to management, exception alarm and other functions, as
set the NX bit so that the relevant memory is made not well as some preprocessing operations. The BOP security
executable; After exiting the fragment, insert the nxset component on which it relies provides the basic isolation
instruction to clear the NX bit and restore the executabil- function, and also can be equipped with more self-pro-
ity of the relevant memory. tection schemes, like anti-electromagnetic interference
devices, as needed.
Implement at hardware side During an instruction cycle, the fetching module will
In terms of implementation, the BOP method relies on perform a memory access, take the XLEN bit2 length data
an independent security component which contains from the address pointed to by the PC register, and give it
BOP modules. The BOP security component is a physi- to the decoding module as an instruction; The decoding
cal component used to implement the module, including module will decode this instruction, analyze its operation
related circuit logic such as processing NX mechanisms, code, operand, operand type and other information, and
and related parts such as additional registers. This com- send it to the execution module; The execution module
ponent is connected with both the processor and mem-
ory device to detect the memory status and control the 2
A general representation of the register bit width, which may be 32 or 64
memory access request of the processor at any time. At bits depending on the actual situation; This is used to represent fetching
data of the same length as the register bit width at one time.
executes the corresponding operation according to the access module to complete the normal memory access
operation code, and may access the memory again at process. If the BOP module, by analyzing the mem-
this time. After the introduction of the BOP method, the ory access type, querying NX bit and other processing
memory access requests of the fetching module and the logic, judges that the program has initiated an incorrect
execution module will first be filtered by the BOP mod- memory access request and is about to execute the data
ule. After confirming that there is no exception in the tar- content in the non-executable memory, it will block this
get memory address, they will be passed to the memory memory access request and raise an instruction excep-
tion. This process is shown in Algorithm 2.
In addition, for systems that do not yet support the For systems that have supported NX bit management
NX bit management mechanism, the BOP security com- mechanism, BOP can directly connect with the existing
ponent will maintain a set of NX registers, simulate the management module in the system, and complete the
mapping relationship between memory addresses and setting of NX bit through the interface provided by the
NX bits, and save the values of each NX bit. In this group existing module, instead of being responsible for the spe-
of registers, a NX bit can be uniquely identified by its bit cific logic implementation of related management work.
number b and the register number ln where the register For example, driving PMP entries by doing preparation
is located; While, there is a specific mapping relation- works described in Sect. "PMP" may be a lazy approach.
ship between (b, ln) and the memory address addr cor-
responding to the NX bit: addr = f (b, ln), so that addr and
Key to safety assurance
(b, ln) pair can convert each other. In the specific imple-
According to the BOP design, the BOP module that
mentation, the conversion process can be quickly com-
manipulates NX bits and filters memory access requests
pleted by using a simple preprocessing logic. Figure 14
should be located in an independent security compo-
describes the design structure of the NX register group
nent, with a set of working logic and register groups
organized by several 32-bit registers.
that are not subject to external interference. Even if the
Fig. 14 A design structure of NX register group
instructions or data in the memory have been attacked Therefore, the specific impact on performance is related
and tampered with in advance, the effectiveness and to the program structure, especially the distribution of
accuracy of the BOP module itself will not be affected. the instructions to be protected. Some of same-type
At this time, the BOP module will normally detect the Xibop instructions which work at a same address can be
exception in the memory, and when the instruction pipe- merged by toolchain to simplify work flow and improve
line enters the memory access link, it will stop and cause performance.
the instruction exception in time, and turn to the excep- Preliminary measurement and performance experi-
tion handling process. ments show that the proposed method can averagely
cause around 7% additional assembly file size growth and
Discussion has a slight impact on the program’s running time. How-
Availability experiment ever, such a result is still unstable according to the above
The experiment mainly consists of two parts: software or analysis.
toolchain part, and hardware part.
Future works
1. Experiments in riscv-gcc v11.1.0, riscv-gas v2.37, The BOP method is still being expanded and improved.
riscv-ld v2.37 and riscv-gdb v10.1 show that the mod- This article only focuses on the detection and defense
ified RISC-V toolchain can recognize the category scheme of buffer overflow in RISC-V system by managing
and boundary of target memory address range cor- memory executability. Further work can be done in the
rectly, and generate security information to assembly following aspects in the future:
file and machine instruction file as expected.
2. Experiments in Xilinx ARTY A7 FPGA show that the 1. Add more sufficient support for RV64. Though our
new circuit logic written by verilog can recognize the instruction extension Xibop is designed for all width,
BOP instructions and update the NX value success- current work focus more on RV32. The instruc-
fully. As a result, abnormal memory access is stopped tion formats and the registers will be changed when
and the illegal control flow transfer is terminated. XLEN is extended. More comparative experiments
will be also implemented.
2. Add support for more defense mechanisms. The goal
Performance evaluation of BOP is to form a complete set of detection and
Basically, each instruction that needs to be protected will defense system against buffer overflow and other
bring at least two Xibop instructions: usually one setnx memory security problems. The defense mechanism
instruction, and its corresponding clrnx instruction. based on NX bit management is only one of the func-
tions provided by BOP, with limited scope of applica- Acknowledgements

Here, we would like to express our heartfelt thanks to the review teachers and
tion and single defense effect. In the future, multiple colleagues who have given all kinds of support and valuable suggestions to
defense mechanisms based on Canary, O-CFI and the work of this paper.
other memory security technologies will be added to
Author contributions
form a complete system as shown in Fig. 1 to jointly CL: Propose the theory, design the ISA extension method, run the project,
maintain system memory security. write the paper. Y-JW: Give advices to ISA extension design, support the
3. Combined with more processor platforms. BOP project. J-ZW: Give advices to the theory and the project, help to improve the
paper. CZ: Support the project.
is a general security solution for RISC-V archi-
tecture, not just designed around a single proces- Funding
sor type. For example, BOP can first implement Strategic Priority Research Program of CAS (XDC05040000).
its prototype in simple environments such as Availability of data and materials
tinyriscv(liangkangnan. tinyriscv. 2022), and then Not applicable.
expect to play a role in multiple processor environ-
ments such as HummingBird(Jayden et al. 2019) and Declarations
Rocket (Asanović 2016), explore the general method
Competing interests
of effective combination with RISC-V processor plat- The authors declare that they have no competing interests.
form, and strengthen the universal adaptability in dif-
ferent environments. Author details
1
Intelligent Software Research Center, Institute of Software, Chinese Academy
4. Internal optimization and adjustment. The current of Sciences, Beijing, China. 2 University of Chinese Academy of Sciences, Bei‑
version of the BOP design scheme focuses more on jing, China. 3 State Key Laboratory of Computer Science, Institute of Software,
the realization of functionality, while there is still Chinese Academy of Sciences, Beijing, China.
much room for optimization in terms of perfor- Received: 8 February 2023 Accepted: 5 June 2023
mance, resource consumption, etc. For example, the
instruction encoding format can be optimized to fur-
ther improve the decoding and execution efficiency;
By optimizing the management logic of NX regis- References
ter group, reduce the use of registers or expand the Asanović K, et al (2016) The rocket chip generator. EECS Department, Univer‑
range of managed address space, etc. sity of California, Berkeley Available: https://www2.eecs.berkeley.edu/
Pubs/TechRpts/2016/EECS-2016-17.html
CNNVD (2022) Vulnerability report. Available: https://www.cnnvd.org.cn/
home/report
Cowan C, et al (1998) StackGuard: automatic adaptive detection and preven‑
Conclusion tion of buffer-overflow attacks. In: Proceedings of the 7th USENIX security
In this paper, we first describe the threat of buffer over- symposium
Davis B, et al (2019) CheriABI: enforcing valid pointer provenance and minimiz‑
flow to the operating system, and analyze the features of ing pointer privilege in the POSIX C run-time environment. In: Proceed‑
several existing security schemes. Then, combined with ings of ASPLOS’19
the features and requirements of RISC-V architecture, we Gao Y-C et al (2013) Research on windows DEP data execution protection
technology. Inform Secur Commun Privacy 7:4
analyze the basic idea of implementing memory security Jayden, et al (2019) The ultra-low power RISC-V core. Available: https://tosco
in RISC-V system, and on this basis, propose a RISC-V de.gitee.com/riscv-mcu/e203_hbirdv2
memory security defense system based on instruction Krerk P, et al (2008) Secure bit enhanced canary: hardware enhanced buffer-
overflow protection. In: 2008 IFIP international conference on network
set extension BOP method. We purposefully designed a and parallel computing. pp 125–131. https://doi.org/10.1109/NPC.2008.
set of RISC-V instruction set extension Xibop that can 49
be used to implement the BOP method. Taking buffer liangkangnan. tinyriscv (2022). Available: https://gitee.com/liangkangnan/tinyr
iscv
overflow detection and defense as an example, we dis- Marco-Gisbert H et al (2019) Address space layout randomization next genera‑
cussed the way RISC-V system supports at both the soft- tion. Appl Sci 9(14):2928
ware and hardware side. In the future, we will continue to Mohan V, et al (2015) Opaque control flow integrity. In: Proceedings of the
22nd annual network and distributed system security symposium
improve the BOP system, support more memory security Molnar I (2003) Exec shield, new Linux security feature. Available: https://lwn.
mechanisms, and gradually realize internal optimization net/Articles/31032/
and adjustment. We also plan to carry out research on One A (1996) Smashing the stack for fun and Profit. Available: http://www.
phrack.com/issues.html?issue=49&id=14&mode=txt
more processor platforms, expand the service scope of Pan C-X et al (2021) Method against process control-flow hijacking based on
BOP method, and explore general implementation solu- mimic defense. J Commun 42(1):37–47
tions for RISC-V architecture as a whole. Wang F-F et al (2019) Overview of control-flow hijacking attack and defense
techniques for process. Chin J Netw Inform Secur 5(6):10–20
Waterman A, et al (2023) The RISC-V instruction set manual. volume II: privi‑
leged architecture. Available: https://riscv.org/technical/specifications
Watson RNM, et al (2019) An introduction to CHERI. Computer Laboratory,

University of Cambridge. ISSN 1476-2986. Available: https://www.cl.cam.
ac.uk/techreports/UCAM-CL-TR-941.pdf
Wikipedia (2022) W%5EX. Available: https://en.wikipedia.org/wiki/W%5EX
Zhao C-Y, et al (2021) Research and implementation of real-time detection
method for code reuse attacks. University of Electronic Science and
Technology of China
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in pub‑
lished maps and institutional affiliations.

A Buffer Overflow Detection and Defense

Uploaded by

A Buffer Overflow Detection and Defense

Uploaded by

Liu et al.

Cybersecurity (2023) 6:45 Cybersecurity

RESEARCH Open Access

A buffer overflow detection and defense

Introduction In attack scenarios such as program control flow hijack-

Fig. 1 Memory layout of exec shield process

hardware environment or system features.In addition,

Fig. 3 Extract from the official document of system call mprotect

Fig. 6 Encoding format of Xibop extended instructions

Instruction nxcheck of the target memory. This instruction will be interpreted

BOP method and its implementation

Fig. 12 Overall architecture of BOP design

Fig. 13 Main workflow of NX bit based non-executable memory defense mechanism

Overall architecture manipulating NX bits, program code can be prohibited

Fig. 14 A design structure of NX register group

tions provided by BOP, with limited scope of applica- Acknowledgements

Watson RNM, et al (2019) An introduction to CHERI. Computer Laboratory,

You might also like