The software manages all scanning, ball processing, and trajectory prediction. Everything was written in Python using a variety of libraries. Due to the processing limitations of a Raspberry Pi and real-time speed demands, this code was designed to run on a laptop running Linux. The code is written in four main files with a plotting file for debugging. There is the main script to run the program, the BallScanner class, to manage the Kinect connection and the processing, and two processing libraries. The architecture section provides a more descriptive overview of the files, and the processing pipeline section explains how the ball is processed and trajectory is fit.
The Run Scanner script is the script that brings everything together. It connects to the Arduino over a Serial connection, initializes a BallScanner object, gets scans, and sends coordinates to the Arduino when the trajectory is predicted.
The BallScanner class does the core of the processing. It uses the PyLibFreenect2 python library to connect to the Xbox Kinect. With this connection, it reads the scans, processes them into image and point cloud information, uses the Ball Process Library and Calibration Library to predict the trajectory, and converts it to the hoop's coordinate system.
When initializing the class, there are three options. The first is whether or not to save the trajectory histories when the BallScanner is closed. The second is whether to calibrate from scratch upon initialization or pull from the calibration file. Finally, you can set whether or not it will reset the trajectories after inactivity to prepare for a new scan.
The Ball Process Library manages all ball processing. It finds the ball from point cloud data, fits a sphere to it, and calculates trajectories.
The calibration library manages the initial calibration for the basketball hoop depth and coordinate system. It prompts the user using Matplotlib to select a rectangle and returns those coordinates and points.
The plotting script enables 3D visualization of the reference scan, predicted trajectories, mid-trajectory points, and the fitted spheres. Each option can be toggled on and off at the bottom of the script. This script is designed to work with the saved data when the BallScanner's save history option is turned on.
Example Reference Image
The system begins by capturing a reference image and point scan as baseline data. It then either loads pre-existing calibration data for depth and hoop positioning or guides the user through a calibration process to establish the reference frame coordinates.
Ball Mask
The PyLibFreenect2 library provides RGB image and infrared (IR) data streams from the Kinect, which are precisely aligned and registered. The system then compares the current frame against the reference image to create a mask highlighting changed pixels, enabling accurate motion detection.
Plot of 3D point cloud data
Using the _calculate_depth method, the system projects the masked 2D pixel coordinates into three-dimensional space. This process creates an RGB point cloud containing the (x, y, z) coordinates of all detected changes, effectively isolating moving objects in 3D space.
Sphere Fitting to Ball Point Cloud
The DBSCAN algorithm analyzes the point cloud to identify distinct clusters of points. The system selects the most isolated cluster (comparing against the reference scan) and verifies its orange color to confirm it's the basketball. Once confirmed, a sphere-fitting algorithm determines the ball's center coordinates.
Predicted Ball Trajectory
Early in the trajectory (≤5 points), the system calculates velocity components and applies kinematic equations to predict the ball's path. Later in the trajectory (>5 points), it employs regression analysis, fitting linear functions to the x/y coordinates and quadratic to the z coordinates, to model the ball's trajectory.
Math to Convert Coordinate Frames
Using the fitted trajectory equations, the system calculates the time when the ball will reach the hoop's y-coordinate. This information, combined with the trajectory equations, determines the predicted landing location, which is then transformed into the hoop's coordinate frame.
Motor Actuation to the Ball
The system accumulates trajectory data and, once it has collected four consistent trajectories showing the ball landing within the target frame, sends the coordinate data to an Arduino for motor actuation. If the ball passes the hoop or if 7 seconds elapse without new trajectory data, the system automatically resets to prepare for the next shot.
Free AI Website Builder