Maintaining open-source software usually involves juggling many possible improvements with only so much time to work on them.
FastTrack, the open-source tool I started for tracking multiple deformable objects in videos during my PhD thesis, is no different. The project has grown over the years, but improving performance is especially tough. Finding bottlenecks is just the first step, and every possible optimization needs to be implemented, tested, benchmarked, reviewed, and then either kept or dropped.
Many potential improvements are therefore difficult to justify. An optimization may sound promising but ultimately produce little measurable benefit. Investigating it manually can consume several hours or days, with no guarantee that the resulting code will be faster.
For the development of FastTrack 6.6.0, I decided to test a different workflow: use AI coding tools to accelerate the implementation and evaluation of potential optimization paths, while relying on profiling, automated tests, reproducible benchmarks, and manual code review to retain control over the final result.
The results were impressive. Every dataset in the benchmark suite ran faster on both systems I tested. Depending on the workload and processor, speedups ranged from about 1.2× to 6×. Overall, the total benchmark time dropped by 48% on an Intel Core i3-12100F and by 56% on an AMD Ryzen AI 9 365, which means overall speedups of 1.93× and 2.25×.
This experiment also taught me something important: the main value of AI here wasn’t in writing software on its own, but in making it much cheaper to try out new ideas.
About FastTrack
FastTrack is a free, open-source desktop application for tracking multiple objects in video recordings. It combines automatic object detection and identity-preserving tracking with tools for manual review and correction. You can use it on Linux, macOS, and Windows, and its tracking core can be integrated into other C++ and Qt apps through a public API.
The software was first built for scientific work, especially experiments with multiple animals or other deformable objects. In these cases, processing time is important for several reasons:
- Recordings may contain many thousands of frames.
- Multiple recordings may need to be processed as a batch.
- Tracking parameters are often adjusted iteratively.
- Real-time tracking should be performed.
So, even a moderate speedup can save much more time than just one run. Faster processing shortens the whole cycle of picking parameters, tracking, checking results, making corrections, and doing the final analysis.
The maintenance constraint
FastTrack isn’t a new prototype. It’s a well-established project. It already works, has users, and reflects years of design choices.
When you’re building a new algorithm, big changes to the structure are often fine. But in a mature application, every change has to keep things working the same way across different platforms and real datasets. A faster version isn’t helpful if it causes numerical errors, data races, platform-specific bugs, database issues, or subtle changes in how objects are identified.
Performance work also competes with other development priorities:
- Compatibility with new operating systems.
- Packaging and distribution.
- Bug fixes.
- Feature requests.
- Dependency upgrades.
- Continuous-integration maintenance.
As the main maintainer, I can’t look into every possible optimization myself. Before this experiment, I knew some parts of the tracking pipeline were likely slow, but rewriting each one would have taken more time than I could justify without proof that it would really help.
Why FastTrack was a suitable candidate for AI-assisted optimization
FastTrack already had two essential components:
- An automated test suite.
- A benchmark and accuracy-testing workflow based on representative datasets.
The tests helped catch any behavior changes after each update. The benchmark suite checked whether the modification was faster across several heterogeneous datasets.
A code change might pass all the tests but still slow down the app. Or it might look great in benchmarks but quietly change the tracking results.
Because this infrastructure was already in place, FastTrack was a good fit for trying out AI-assisted development. I didn’t have to trust AI-generated patches just by looking at them. I could compile, test, benchmark, inspect, tweak, or reject them based on real evidence.
The working hypothesis
The experiment started with a pretty simple idea:
AI coding tools could reduce the implementation cost of testing optimization ideas, including ideas whose benefits were too uncertain to justify implementing manually.
The goal wasn’t to just tell an AI to “make FastTrack twice as fast” and accept whatever it gave me.
Instead, the process was more like supervised performance engineering:
- Identify or profile an expensive part of the program.
- Define the expected behavior and relevant constraints.
- Ask the AI system to analyze or rewrite a bounded section.
- Inspect the proposed implementation.
- Compile it with the project’s normal warning and quality settings.
- Run the functional and accuracy tests.
- Benchmark the change.
- Refine it manually or with additional AI iterations.
- Reject it if the improvement was absent, too small, or obtained at an unacceptable complexity cost.
- Approve and commit only the final reviewed version.
AI generated most of the first versions. I guided the process, reviewed the code, fixed and simplified things as needed, checked how it worked and performed, and took full responsibility for every change that was committed.
Profiling before optimization
When optimizing performance, it’s best to start by measuring, not just guessing. The first step was therefore to inspect the tracking pipeline and identify operations that were likely to dominate execution time or scale poorly with the number of objects, pixels, or frames.
Replacing the object-assignment implementation
One of the most substantial algorithmic changes concerned FastTrack’s implementation of the Hungarian assignment algorithm.
Object tracking requires associating objects detected in the current frame with objects identified in previous frames. FastTrack constructs a cost matrix describing the plausibility of each possible association and solves the resulting assignment problem.
The previous implementation (the best I found five years ago; I benchmarked many available implementations at the time) repeatedly scanned or copied complete matrices and created large temporary allocations. The replacement uses a shortest-augmenting-path formulation with contiguous row-major storage. It also handles square, wide, tall, and empty matrices, avoids physically transposing tall matrices, removes repeated large temporary allocations, and validates malformed input matrices.
This shows why changing the algorithm itself can matter more than just tweaking small parts. Better memory layout and less repeated matrix work can improve both speed and how well the code uses the computer’s cache.
However, this was also one of the areas where validation was most important. Assignment code influences object identities directly. A faster implementation would not be acceptable if it changed assignments unexpectedly or mishandled rectangular or degenerate matrices.
Avoiding unnecessary image rotations
Another important change removed part of the image-processing work previously used to determine object orientation.
The earlier implementation physically rotated an image region so that an object’s main axis became horizontal. It then divided the rotated image into two regions and calculated properties used to distinguish the head and tail.
The new implementation derives the required information from projected pixel coordinates and statistical moments without first constructing a rotated image. This optimization is interesting because it skips a costly image transformation and instead calculates the needed information directly.
This kind of reformulation is a particularly useful target for AI-assisted exploration. The transformation is mathematically bounded, but implementing it correctly involves enough indexing, coordinate conversion, and statistical calculation that trying it manually may not have been an attractive use of limited maintenance time.
Improving database writes
Tracking does not end with image analysis. Results must also be stored. The optimization work included changes to the database-insertion path, reducing repeated query setup and reorganizing how values are bound and written within the existing transaction flow.
It’s easy to miss database overhead when profiling a computer vision app. You might assume most of the time goes to image processing, but in reality, repeated setup, allocation, conversion, and insertion inside a frame-by-frame loop can add up.
Reducing repeated work in object selection
A smaller optimization simplified the object-selection logic used during tracking. Although the corresponding patch changed only a few lines, operations inside a nested frame-and-object loop can accumulate substantial cost over an entire recording.
This is another plus of using AI in the workflow: it makes it easier to try out even small improvements.
As a maintainer, I might skip spending an hour rewriting and testing a few lines if I don’t know the impact. But if a patch can be generated quickly and tested automatically, even small ideas are worth trying.
Balancing parallelism across heterogeneous workloads
The latest changes add parallel processing to key parts of the tracking pipeline, such as contour processing and cost-matrix calculations. However, parallelism is only used when it actually speeds things up. For smaller workloads, the overhead of managing threads can outweigh the benefits, so the code runs sequentially unless the workload is large enough to justify parallel execution. Thresholds for switching between sequential and parallel processing were set based on benchmarking, not fixed values.
FastTrack handles many kinds of datasets, some with a few large objects, others with many small ones, or complex combinations. Each type stresses different parts of the pipeline. For example, large objects require more image processing, while many small objects make assignment problems bigger. Because of this variety, indiscriminate parallelism can actually slow things down for some cases.
To address this, the implementation uses conditional parallelism: only workloads above a certain size are parallelized, and the best threshold or scheduling strategy can vary depending on the data. AI assistance made it much faster to try, test, and refine these parameters. Instead of finding a single "best" setup, the process focused on quickly evaluating different options to find what works well across a range of real datasets.
As a result, the overall speedup varies by workload and processor, since each dataset stresses a different mix of tasks. The main goal was robust performance for many scenarios, not just maximizing one benchmark.
Changes that were not retained
Not every proposed optimization was useful. Some changes failed to improve performance, made one dataset faster but another slower, altered output values, added too much complexity, duplicated existing library functions, created unsafe parallel access, increased memory use, relied on invalid assumptions, or passed tests while failing broader accuracy checks. One of the main benefits was being able to discard these failed experiments quickly and at low cost. The AI system was valuable not just for generating patches that worked, but also for making it affordable to test and rule out ideas that were not worth pursuing.
Interpreting the results
The benchmark results show that the optimizations improve performance across every tested dataset, but the magnitude of the gain varies substantially between workloads and processors. This variation is expected because each dataset stresses a different combination of image processing, contour analysis, cost-matrix construction, object assignment, database access, and parallel execution. Datasets with many objects can benefit more from improvements to assignment and pairwise-cost calculations, while datasets containing fewer but larger or more complex objects may remain dominated by image-level operations. Hardware characteristics also influence the results: the 20-thread Ryzen AI 9 365 provides more opportunities for parallel execution, whereas the 8-thread Intel Core i3-12100F places a lower ceiling on parallel throughput. Nevertheless, the optimized version completed the full benchmark suite approximately 1.93× faster on the Intel system and 2.25× faster on the AMD system, reducing cumulative runtime by about 48% and 56%, respectively. The individual speedups ranged from approximately 1.2× to 6×, demonstrating that the improvements are broadly effective while remaining strongly dependent on workload composition. The appropriate conclusion is therefore not that FastTrack is uniformly six times faster, but that it now completes a diverse set of representative tracking workloads in roughly half the previous time.
What AI implemented and what remained human work
It is worth being transparent about the division of work.
AI tools generated most of the initial implementation for the optimization patches. This included algorithmic rewrites, local simplifications, candidate parallelizations, and associated test changes.
My work included:
- Identifying the performance objective.
- Directing the profiling and optimization process.
- Deciding which parts of the code could be changed safely.
- Providing constraints and feedback.
- Reviewing generated code.
- Independently running and interpreting tests.
- Rejecting unsuitable approaches.
- Evaluating maintainability.
- Approving and committing the final code.
- Retaining responsibility for the released behavior.
AI as a tool for reducing experimental cost
The most useful effect of the workflow was economic rather than magical.
Before AI-assisted implementation, each optimization idea had an implementation cost. Even a plausible idea could remain unexplored because that cost was too high relative to its uncertain benefit.
The new workflow changed that balance.
A candidate optimization could be generated, compiled, tested, and benchmarked quickly. If it failed, the loss was limited. If it worked, it could be reviewed and refined. This made it reasonable to explore more paths, including those with uncertain returns.
That is a meaningful change for open-source maintenance, where developer attention is often the scarcest resource.
Conclusion
FastTrack 6.6.0 demonstrates a practical use of AI-assisted software development in an existing open-source scientific application.
Overall, this work shows that AI-assisted development can deliver meaningful improvements in a mature open-source project when it is combined with strong engineering safeguards. The main benefit was not a single optimization or an isolated benchmark result, but the ability to explore, implement, test, and reject performance ideas much more quickly than would otherwise have been practical. Profiling identified where effort should be focused, automated tests protected correctness, heterogeneous benchmarks revealed workload-dependent effects, and manual review ensured that only maintainable changes were retained. The resulting performance gains demonstrate the value of this approach, but the broader lesson is methodological: AI is most effective when it operates inside a disciplined development process rather than replacing one.
The methodological result is more interesting.
AI made it feasible to implement and evaluate optimization paths that would previously have been too expensive or uncertain to investigate with the available maintenance time. Yet the useful outcome depended on infrastructure and oversight that already existed around the generated code:
- profiling identified meaningful targets;
- automated tests guarded behavior;
- benchmark datasets measured real performance;
- accuracy checks evaluated scientific output;
- manual review protected code quality;
- final approval remained human.
The experiment did not show that AI could autonomously optimize a mature application. It showed that, in a well-tested project, AI can make disciplined software experimentation considerably cheaper. That may be one of its most useful roles for open-source maintainers.