Behavior Senpai is an application that supports quantitative behavior observation in video observation methods. It converts video files into time-series coordinate data using keypoint detection AI, enabling quantitative analysis and visualization of human behavior. Behavior Senpai is distinctive in that it permits the utilization of multiple AI models without the necessity for coding.

The following AI image processing frameworks/models are supported by Behavior Senpai:

• YOLO11 Pose
• YOLOv8 Pose
• MediaPipe Holistic
• RTMPose Halpe26 (MMPose)
• RTMPose WholeBody133 (MMPose)

Behavior Senpai performs pose estimation of a person in a video using an AI model selected by the user, and outputs time-series coordinate data. (These are variously referred to as "pose estimation", "markerless motion capture", "landmark detection", and so forth, depending on the intended purpose and application.)

BehaviorSenpai can import inference results (with .h5 extension) from DeepLabCut.

Behavior Senpai is an open source software developed at Faculty of Design, Kyushu University.

In order to use Behavior Senpai, you need a PC that meets the following performance requirements. The functionality has been confirmed on Windows 11 (23H2).

• Disk space: 12GB or more
• RAM: 16GB or more
• Screen resolution: 1920x1080 or higher
• GPU: RTX3060~ (and its drivers)

If you do not have a CUDA-compatible GPU, only MediaPipe Holistic can be used.

• Disk space: 8GB or more
• RAM: 16GB or more
• Screen resolution: 1920x1080 or higher

Download BehaviorSenpai150.zip

Running BehaviorSenpai.exe will start the application; if you want to use CUDA, check the "Enable features using CUDA" checkbox the first time you start the application and click the "OK" button.

BehaviorSenpai.exe is an application to automate the construction of the Python environment by uv and the startup of Behavior Senpai itself. The initial setup by BehaviorSenpai.exe takes some time. Please wait until the terminal (black screen) closes automatically.

To uninstall Behavior Senpai or replace it with the latest version, delete the entire folder containing BehaviorSenpai.exe.

See here for a document with all IDs.

The time-series coordinate data resulting from keypoint detection in app_detect.py is stored in a Pickled Pandas DataFrame. This data is referred to by Behavior Senpai as a "Track file". The Track file is saved in the "trk" folder, which is created in the same directory as the video file where the keypoint detection was performed. The Track file holds time-series coordinate data in a 3-level-multi-index format. The indexes are designated as "frame" "member", and "keypoint", starting from level 0. "Frame" is an integer, starting from 0, corresponding to the frame number of the video. "Member" and "keypoint" are the identifiers of keypoints detected by the model. The Track file always contains three columns: "x," "y," and "timestamp." "X" and "y" are in pixels, while "timestamp" is in milliseconds.

An illustrative example of a DataFrame stored in the Track file is presented below. It should be noted that the columns may include additional columns such as 'z' and 'conf', contingent on the specifications of the AI model.

BehaviorSenpai saves calculated features based on Track file data to Feature files. Feature files are created in HDF5 format with the .feat extension. Feature files store data calculated by app_points_calc.py, app_trajplot.py, and app_feat_mix.py in the format shown in the table below:

Additionally, data calculated by app_dimredu.py is stored in the format shown in the table below. All these data are handled as Pandas DataFrames, with time-series data in 2-level-multi-index format, with the indices designated as "frame" and "member", respectively, and the columns including a "timestamp".

As mentioned above, Behavior Senpai handles pickle format files, and because of the security risks associated with pickle format files, please only open files that you trust (e.g., do not open files from unknown sources that are available on the Internet). (For example, do not try to open files of unknown origin published on the Internet). See here for more information.

Behavior Senpai can output videos in mp4 format with detected keypoints drawn on them.

This section explains the default locations for data output by Behavior Senpai. Track files are saved in the "trk" folder, Feature files in the "calc" folder, and videos with keypoints drawn are saved in the "mp4" folder. If a Track file is edited and overwritten, the old Track file is saved in the "backup" folder (only one backup is kept). These folders are automatically generated at the time of file saving.

Below is an example of the folder structure when there are files named "ABC.MP4" and "XYZ.MOV" in a folder. Output file names include suffixes according to the model or type of calculation. To avoid file read/write failures, use alphanumeric characters for folder and file names, especially when the file path contains Japanese characters.

Videos
├── ABC.MP4
├── XYZ.MOV
├── calc
│   ├── case1
│   │   ├── ABC.feat
│   │   └── XYZ.feat
│   └── case2
│       └── XYZ.feat
├── mp4
│   └── ABC_mediapipe.mp4
└── trk
    ├── ABC.pkl
    ├── XYZ.pkl
    └── backup
        └── ABC.pkl

When Behavior Senpai loads a Track file, if a video file exists in the parent folder, it also loads that video file. The file name of the video to be loaded is referred from the "video_name" value in Attributes of Track file. If the video file is not found, a black background is used as a substitute.

The application's settings and the path of the most recently loaded Track file are saved as a Pickled dictionary. The file name is "temp.pkl". If this file does not exist, the application automatically generates it (using default values). To reset the settings, delete the "temp.pkl" file. The Temporary file is managed by gui_parts.py.

Please acknowledge and cite the use of this software and its authors when results are used in publications or published elsewhere.

Nishimura, E. (2025). Behavior Senpai (Version 1.5.0) [Computer software]. Kyushu University, https://doi.org/10.48708/7160651

@misc{behavior-senpai-software,
  title = {Behavior Senpai},
  author = {Nishimura, Eigo},
  year = {2024},
  publisher = {Kyushu University},
  doi = {10.48708/7160651},
  note = {Available at: \url{https://hdl.handle.net/2324/7160651}},
}

Sample Videos for Behavioral Observation Using Keypoint Detection Technology

Quantitative Behavioral Observation Using Keypoint Detection Technology: Towards the Development of a New Behavioral Observation Method through Video Imagery