Visual Synchronome

The Visual Synchronome project goal is to prove the viability of Linux as a soft real-time system by synchronizing to an external clock. The design consists of a Logitech C270 camera connected to a Raspberry Pi 4 monitoring an online clock presented on a monitor running on a separate computer. Clear images must be captured after each state change of the clock to show successful synchronization. This means the image capture must happen within the major period of the clock (either 1Hz or 10Hz) and cannot overlap with the clock transition or the image will be blurry. To meet the requirements, the design has four primary services spread across three cores to capture, process, and save the image to flash memory. Two services complete the image processing sharing a single core. The system requires predictable service release and execution time to ensure image clarity. Performance is tested at 1Hz to ensure system feasibility, and then at 10 Hz to test near the limit of the camera frame rate.

Functional Requirements

The minimum requirements are:

2 or more real-time services running on one core.
1 Hz, non-blurry, unique, monotonic seconds while monitoring an online analog clock.
PPM or PGM images with a timestamp.
180+1 frames at 1 Hz accomplished at least twice.

The target requirements are:

Runs at either a 10 Hz or 1Hz frame acquisition rate.
Can save 1800+1 image files
Simple transformation that can be enabled or disabled
Achieve minimum requirements with the transformation running

The stretch requirements are:

Run at 10 Hz, non-blurry, unique, monotonic seconds while monitoring an online digital clock.
Tested and proven design with no observable errors and predictable response.

The extra requirements are:

Add additional workload to get more than 50% CPU utility.

The minimum and target requirements have been met. The stretch requirements have been partially met, but still have the occasional blurry image. The extra requirement has not been implemented yet.

Figure 1: A depiction of the visual synchronome application. Tasks, which are broken out into threads in the application, are depicted as circles and data buffers are depicted as squares. Timing is coordinated by the sequencer task. The solid lines represent data flow. Each task is assigned a specific core which are represented by the columns separated by dotted lines.

Real-Time Requirements

The system must identify each time the external clock ticks and capture a single frame showing the new time. This means the full duration of the capture must be within the major period. Overlapping with the transition will result in a blurry image. The 1Hz problem involves capturing each tick of the second hand on an analog clock. Since both clocks are predictable, the primary challenge is to find a frame with a stationary second hand, and then capture a new frame every second after that. This is possible so long as the initial frame has sufficient buffer from the clock transition to account for service release jitter and variations in computation time. To maximize the reliability of this service, it will run on its own core, independent from the other services.

Once a frame is captured, it needs to be compared to the previous frame to determine if a clock transition was detected. For each frame, the image differencing service needs to compute the difference between the current frame and the previous frame to determine if there was significant motion between the two captures. This needs to happen at least as fast as the frame acquisition rate to ensure the buffer doesn’t get backed up. Since this service has the shortest period, it will have the highest priority on this core to align with rate-monotonic scheduling. Once every major period, the selection service decides which frame to use for synchronization and writeback. This will involve reviewing all of the available images provided by the differencing service and picking the clearest frame that shows the transition. This will be the second highest priority service. Optionally, the transformation service will run on this core and cycle once per major period to prepare the selected image for writeback. Based on the frequencies of these services, the differencing service will be the highest priority. The remaining two services will run at the same rate, but since the transformation is optional and could run at a slower rate, the selection service will be the second-highest priority. If necessary, to better align with rate-monotonic theory, these periods could be extended to artificially lower their priority.

The writeback service will run by itself on a core due to the potential for long computation times. This will keep other services from blocking it and help ensure writeback happens on time. Additionally, there is a JSON logger service that monitors a data buffer for analytical data from all of the services and writes it to a JSON file. This service helps with debugging and analyzing real-time characteristics. An optional CPU Burner service is yet to be implemented but will be a low-priority, long computation time service that increases CPU utility beyond the 50% required.

Functional Design Overview

A depiction of the function design is presented in Figure 1. Accomplishing the project requirements is possible through one of several methods: start the capture at a random time, manually trigger the initial capture, or auto-trigger the initial capture. Starting the capture at a random time will work as long as the first captured frame is fortunate enough to be during a moment when the second hand is motionless and with enough buffer to overcome the release jitter and variation in computation time; however, this is not reliable. If the captured frame happens to be during a transition, all of the follow-on frames may be repeats, skips, or blurry. Even if it is close to the transition, process jitter may cause the occasional problem frames. This method cannot reliably be used to meet even the minimum requirements.

Manually triggering the initial capture is a more reliable solution for the 1Hz problem. The operator can watch the clock and signal the start of the capture as soon as the second hand moves. Since there will be a delay between when the clock transitions and the operator recognizes the motion, the captured frame will have a reasonable buffer from the transitioning frame. This method can further be automated by capturing a group of photos, identifying the transition frame, and using the frame that is as far from the transitioning frame as possible. This is the auto-trigger method and is the method used for this design.

The services are controlled by the Sequencer which keeps track of when to release the different services. It uses an interval timer for each service set to its release period. The callback for each timer releases its associated service by incrementing the related semaphore. Each service is blocked on its semaphore until this timer goes off, allowing it to complete a single cycle.

The auto-trigger requires a group of frames covering a single period. For the 1Hz solution, 10 frames are captured each second, giving 10 frames to choose from when determining the motion frame. For the 10Hz solution, frames are captured at the maximum rate for the camera, 30Hz, which gives 3 frames to review. The Capture Service first acquired these frames on CPU 1 and then moved into the Image Ring Buffer. The capture service attempts to grab a non-empty frame up to a user-defined timeout to minimize the potential for missed frames, particularly when running at the maximum frame rate. There is also a Simulate Capture Service used for development that simply alternates between two images across a user-defined period. This allows for faster development and testing without needing the hardware.

Figure 2: Shows the data flow through the Differencing Service. The Current and Previous Image pixel-wise difference is computed to create the Difference Image. This image is thresholded to reduce noise so that small variations in the pixels are not considered. Finally, the pixel values for the Threshold Difference Image are summed to create the Image Score which is attached to the Current Image and pushed to the Selection Ring Buffer.

The Differencing Service operates at the same rate as the capture service, either 10 or 30Hz. This service retrieves frames from the Image Ring Buffer and compares them to the previous frame using a differencing algorithm to detect motion subject to a threshold (Berrios, n.d.; Ghode, 2019). If the difference between the corresponding pixels in the two frames exceeds the threshold, the value is stored in a new image. Next, the sum of pixel values in the new image is computed and attached to the current image being analyzed as the Image Score. A high Image Score indicates a large difference from the previous frame and is an indication that the image captured a clock transition. The analyzed image with its attached Image Score is then moved into the Selection Ring Buffer.

The Selection Service runs on CPU 2 at the same rate as the major period for the system, either 1Hz or 10Hz. This service first empties the Selection Ring Buffer to create a set of photos to be analyzed for the period. When the service first starts, it waits for a full buffer (i.e., either 10 or 3 frames for 1Hz or 10 Hz operation, respectively) before analyzing the set. With a full set, the image with the highest Image Score is identified as the frame that captured the clock transition. For 1Hz operation, the frame index as far from this frame as possible is stored as the default image index to use going forward. In other words, if the second image is identified as the motion image, then the seventh frame is used to show a clear picture of the clock. Then the seventh image is used for all subsequent releases of the Selection Service. The 10Hz mode is more likely to miss a frame or have multiple high Image Scores, making finding a clear frame more challenging. The auto-trigger 1Hz operation is still used to find the frame that provides a clear image of a tenth of a second, and the same index is used moving forward. However, more work needs to be done to find a clear frame during outlier scenarios. Once the selection is done, the selected frame is moved into the Writeback Ring Buffer, or the Transform Ring Buffer if the Transform Service is active.

Figure 3: shows a selection example for 10 and 3 images. The left image shows an example of ten images being pulled from the Selection Ring Buffer during 1Hz operation, with the M marking the highest Image Score from the set. The X marks the selected index to use for writeback. Similarly, the right image shows an example of the three images being used for 10Hz operation.

Optionally, the Transform Service also runs on CPU 2 at the same rate as the Selection Service, either 1 or 10 Hz. This service takes an image from the Transform Queue and creates a new image with inverted pixel values from the original. This image is then pushed into the Writeback Ring Buffer or the CPU Burner Ring Buffer if the CPU Burner Service is active.

Another optional service, the CPU Burner Service, will also run on CPU 2. When active, this service will process images from the Selection Ring Buffer, or the Transform Ring Buffer if the Transform Service is active. This service first converts the image to grayscale and then determines the mean pixel value and standard deviation. These values are then written to the image by the Writeback service. The purpose of this service is to achieve greater than 50% CPU utilization. To this end, there is also a user-defined minimum duration for this service to run allows for exploring the effects of different CPU loads on the RMA core. This is the lowest priority service on CPU 2.

The final step in the data pipeline is the Writeback Service. This service writes the images from the Writeback Ring Buffer to FLASH. The Writeback Service runs at the same rate as the major period, 1 or 10 Hz. Because this service has a high potential for variable execution time, it runs alone on CPU 3. The application requirements are to save a certain number of images, so this service is also responsible for signaling the end of the program. Once the requisite images are saved, this service will return success and the other services will be aborted.

Additionally, the JSON Logger Service collects service data from all of the other services for use in debugging and real-time analysis. This service needs to keep up with, at most, a single data log for each service during its service period. To keep it from interfering with the primary services, it runs on CPU 0. Data collected includes the service ID, start and stop time, along with metrics for the various services. The Differencing Service reports the Image Score, the Selection Service reports the number of images reviewed, and the Writeback service reports how many total images have been saved after each service’s release.

Table 1: Service (S) analytics including computation times (C), periods (T), and deadlines (D). The computation time includes the worst case and typical processing time (C: worst / typical).

Real-Time Analysis

The project currently has six services, with computation times (C), periods (T), and deadlines (D) described below. The computation times were determined by logging the start and stop time of each service and then finding the worst-case and typical time the service ran.

The RMA feasibility analysis is shown in Figure 4. This analysis includes the differencing, selection, and transform service running at 10 Hz on the same core. Because this configuration is the most demanding on the system, running without the transform service or at 1Hz will both be feasible. The utility for the 10Hz system assuming worst-case computation times with the transformation service running is 4.99% and the LUB for 3 services is 77.97%. This means the design is also feasible according to the LUB with a safety margin of 72.98% CPU. This also indicates that the CPU Burner Service will need to fill at least approximately 45% of the CPU up to approximately 73% to remain within the LUB.

Figure 4: RMA feasibility shows the visual synchronome design can be scheduled at the 10Hz rate. S2 is the differencing service, S3 is the selection service, and S6 is the transformation service. S3 is shown only as a thin line because its computation time is small compared to the other two. This analysis does not fully show the longest period because S3 and S6 finish by the third period at approximately 7 milliseconds, leaving the remaining time free on the CPU.

Figure 5: The Capture Service is running alone on CPU 1. For the 1Hz requirement, the typical computation time of approximately 5.5 milliseconds is more than adequate. For the 10Hz requirement, and based on the 30 Hz frame rate of the camera, the system can expect to run this service at most every 33.3 milliseconds. Assuming the worst-case performance of 6.5 milliseconds of computation, the utility of the CPU is below 20%. This service will be able to reliably provide frames at the maximum frame rate. The release jitter is on the order of tens of microseconds which is very good for a predictable response of the capture service.

Figure 6: The Differencing Service runs on CPU 2 and must keep up with the buffered data from the capture service. The typical computation time is approximately 3.4 milliseconds. For the 1Hz requirement frames are arriving every 100 milliseconds, which means this computation time is more than adequate to keep up, even with the worst-case scenarios. This service will also be able to keep up with frames coming in at 30Hz (i.e., 33.3 milliseconds) for the 10Hz requirement. Again, the service release jitter is on the order of tens of microseconds and provides excellent reliability.

Figure 7: The Selection Service runs on CPU 2 on the major period, either 1 or 10 Hz, and is extremely fast. It reliably completes within 70 microseconds and also has release jitter on the order of tens of microseconds. This service is fast enough and infrequent enough that it has a negligible impact on the overall performance of the system.

Figure 8: The Transform Service runs on CPU 2 on the major period, either 1 or 10 Hz, and transforms the selected image to the negative image. This service needs to be completed within the major period. The WCET for this service was 2.8 milliseconds which allows plenty of margin. Again, the service release jitter is on the order of tens of microseconds and provides excellent reliability.

Figure 9: The Writeback Service service runs alone on CPU 3 and needs to keep up with the frames coming in on the major periods, either 1 or 10 Hz. Since the computation time can vary widely for this service, and take a significant amount of time, keeping it on its own core helps ensure it doesn’t fall behind. The service typically takes less than 100 milliseconds, which means it will be able to keep up with the 10Hz requirement. When it occasionally falls behind, there is plenty of margin to allow it to catch back up. Again, the service release jitter is on the order of tens of microseconds and provides excellent reliability.

Figure 10: The JSON Service runs on CPU 0 at a faster rate than the other services so that it can keep up with the analytical data coming in. The computation time is typically about one millisecond which is plenty fast to keep up with services publishing data at most every 33.3 milliseconds. The release period for this service cycles between 100 and 200 milliseconds because of a feature of the logger where releases without data to process will not generate a log message. So the 200 millisecond release periods occur when the JSON logger has nothing to log during one cycle.

Proof-of-Concept

To prove that the synchronome works, collections of images were gathered for the different modes of operation. Figures 11 through 17 show the first and last images of the 1Hz operation for the captured image, a transformation to a negative image at 1Hz, and a 10Hz collection, respectively. Additionally, 1801 frames were successfully captured and checked for image clarity at 1Hz. The 10Hz mode cannot currently capture exclusively clear images. Work will continue for this mode before the final project is submitted.

Figure 11: shows the first and last frame of the system before collecting 181 frames over three minutes. The intermediate frames were verified to have clear images. Additionally, the timestamp for the image is the first number on the bottom right. The next two numbers are the mean and standard deviation of the pixels, respectively, provided by the CPU Burner Service.

Figure 12: shows the last frame of the system after collecting 181 frames over three minutes. The intermediate frames were verified to have clear images. Additionally, the timestamp for the image is the first number on the bottom right. The next two numbers are the mean and standard deviation of the pixels, respectively, provided by the CPU Burner Service.

Figure 14: shows the first frame of the system before collecting 181 transformed frames over three minutes. The intermediate frames were verified to have clear images.

Figure 15: shows the first frame of the system before collecting 181 transformed frames over three minutes. The intermediate frames were verified to have clear images.

Figure 16: shows the first frame of the system before collecting 1801 frames over three minutes at 10Hz. The intermediate frames had at least one blurry image.

Figure 17: shows the last frame of the system after collecting 1801 frames over three minutes at 10Hz. The intermediate frames had at least one blurry image.

Conclusion

The visual synchronome project successfully shows that Linux can be used to meet soft real-time deadlines. The 1Hz mode has plenty of margin for error and can maintain synchronization without any glitches. The 10Hz operation pushes the system to the limit of the camera’s capabilities and still requires additional work to get fully functioning. However, the RMA analysis and measured service performance indicate that the 10Hz operation is within the limits of the system and should be achievable.

Page updated

Google Sites

Report abuse