I’m trying to to detect actions/tasks from surgical videos. The normal object detection is straightforward which I used to the draw bounding boxes around surgical instruments but I’ve not much experience of tasks detection. How videos are labelled for tasks/actions detections and other challenges.
Does anyone know about an example tutorial for actions/tasks detection from videos? I prefer if tutorial is using FastAI but any help or guidance will be appreciated.
Medical video annotation involves marking and extracting objects of interest frame-by-frame. This enables the development of AI applications, such as surgical robots, automated patient monitoring systems, diagnostic tools, etc. Medical video annotation includes:
Endoscopic Footage, Surgical Recordings: Annotating video data from endoscopic procedures or operating-room recordings. This includes labeling surgical instruments in use, anatomical structures, and tissue interactions to train AI models for video-based surgical assistance.
Surgical Step Detection: Identifying and tagging key procedural steps in surgical videos — for example, incision, dissection, resection, and closure — to create structured datasets that support surgical training, automation, and intraoperative guidance.
Event Detection: Identifying and flagging significant events or adverse incidents happening within a video, such as accidental injury, bleeding, or device malfunctions. This supports the development of predictive models for complication prevention and enhances surgical safety.