DETAILED ACTION
Applicant's amendment of July 20, 2026 overcomes the following:
Specification objections
Claim 1-11 and 14-17 objections
Rejection of claims 16-17 under 35 U.S.C. 112(b), pre-AIA 35 U.S.C. 112, second paragraph
Applicant has amended claims 1-17. Claims 1-17 are pending.
Response to Arguments
Applicant’s arguments filed on July 20, 2026 with respect to claims 1-17 have been considered but are moot in view of the new ground(s) of rejection. The amended claims resulted in changes to the scope and contents; therefore, the grounds of rejection are modified accordingly.
Claim Objections
Claim 12 is objected to because of the following informalities:
Claim 12 recites the limitation “method according to claim 1 including… outputting the motion performance metric” in lines 1-2 of the claim. However, the claimed “motion performance metric” recited in line 2 of claim 12 was previously recited in line 3 of claim 11.
Therefore, based on above, for examination purposes, the claimed “method according to claim 1 including… outputting the motion performance metric” recited in lines 1-2 of claim 12 will be interpreted as “method according to claim 11 including… outputting the motion performance metric”.
Appropriate correction is still required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 15 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 15 now recites the limitation “visual data of at least two subjects as they move between the first and second distance markers” in lines 3-5 of the claim. However, it is not clear if any the claimed “at least two subjects” recited in line 4 of claim 15 encompass embodiments corresponding to the claimed “subject” previously recited in line 1 of claim 1, or if the claimed “at least two subjects” recited in line 4 of claim 15 encompass embodiments corresponding to “at least two subjects” different from the claimed “subject” previously recited in line 1 of claim 1, for example.
Therefore, based on above, the metes and bounds of the claim are not clearly set forth and the examiner cannot clearly determine which elements are encompassed by the claim language, which renders the claim indefinite.
Claim 15 further recites the limitation “recognising human pose points on the subject… recognising human pose points on the subject to extract kinematic data of the subject… recognising a plurality of predefined anatomical components of the subject to construct the biomechanical model of the subject” in lines 9-13 of the claim. However, examiner cannot clearly ascertain which one of the claimed “at least two subjects” recited in lines of claim 15 corresponds to the claimed “the subject” recited in lines 9-13 of claim 15, for example, or if the claimed “the subject” recited in lines 9-13 of claim 15 encompass embodiments corresponding to the claimed “subject” previously recited in line 1 of claim 1 instead of the claimed “at least two subjects” recited in lines of claim 15, for example. Furthermore, it is not clear if the claimed “a plurality of predefined anatomical components of the subject” recited in lines 11-12 of claim 15 encompass embodiments corresponding to the claimed “plurality of predefined anatomical components of the subject” recited in lines 8-9 of claim 1, or if the claimed “a plurality of predefined anatomical components of the subject” recited in lines 11-12 of claim 15 encompass embodiments corresponding to another “plurality of predefined anatomical components of the subject” different from the claimed “plurality of predefined anatomical components of the subject” recited in lines 8-9 of claim 1, for example.
Therefore, based on above, the metes and bounds of the claim are not clearly set forth and the examiner cannot clearly determine which elements are encompassed by the claim language, which renders the claim indefinite.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-6, 9, and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Martinetz et al. (US PG Publication No. 2014/0328519 A1), hereafter referred to as Martinetz, in view of Kord et al. (US PG Publication No. 2015/0097937 A1), hereafter referred to as Kord, Applicant cited prior art furnished via IDS, in further in view of McGibbon et al. (US PG Publication No. 2002/0009222 A1), hereafter referred to as McGibbon, and in further view of MALHOTRA et al. (US PG Publication No. 2016/0027325 A1), hereafter referred to as MALHOTRA.
Regarding claim 1, Martinetz discloses a method for generating a model of a subject in motion (Par. [0001-5]: method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, i.e. estimating the pose of the movable object… apparatus mentioned for example provides the 3D coordinates of predetermined body parts that change over time--during the motion; Par. [0024-26]: pose estimation by means of SOM training is proposed that is able to work with a skeleton model that is modelled according to the anatomy of the object observed… the anatomy of the object whose pose is to be detected is modelled as a skeleton model… A skeleton model that is described only by a small number of points ("nodes" below) in the 3D space represents a good information compression of the image information if the coordinates of the nodes at all times describe the position of predetermined parts of the moving object… a method for pose estimation of a moving object (e.g. a person or a robot) by computer calculation of displacements of 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds. The node coordinates are present in table form in an electronic memory and the 3D point clouds are determined from electronically recorded images from a depth sensor camera that represent the moving person; Par. [0116-120]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person, using image segmentation that is known per se… A depth sensor camera that comprises an arithmetic unit for determining the distance coordinate can in particular also be engineered directly to carry out the inventive method. Such a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts; a method for generating a model of a subject in motion (e.g. method for pose estimation of a moving object (i.e. a subject), such as a person, for example, includes three dimensional (3D) point clouds that are determined (i.e. generated, calculated, computed, estimated, etc.) by computer calculation of displacements (i.e. movements) of 3D position coordinates of nodes of a skeleton model (i.e. a model) that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (i.e. a method for generating a model of a subject in motion), as indicated above), for example) including the steps of:
capturing, by a single motion capture device, visual data of the subject as it moves in a field of vision of the motion capture device (Par. [0001-5]: method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, i.e. estimating the pose of the movable object… If the motion of the person is imaged by an image sequence, the apparatus mentioned for example provides the 3D coordinates of predetermined body parts that change over time--during the motion; Par. [0024-53]: the anatomy of the object whose pose is to be detected is modelled as a skeleton model… A skeleton model that is described only by a small number of points ("nodes" below) in the 3D space represents a good information compression of the image information if the coordinates of the nodes at all times describe the position of predetermined parts of the moving object… The skeleton model is to be fitted fast and precisely into the image information that corresponds to the object. The fitting is effected between two images of an image sequence by continuously displacing the nodes and updating the skeleton model in step with the image sequence… a method for pose estimation of a moving object (e.g. a person or a robot) by computer calculation of displacements of 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds. The node coordinates are present in table form in an electronic memory and the 3D point clouds are determined from electronically recorded images from a depth sensor camera that represent the moving person… an apparatus for pose estimation of a moving object. This apparatus comprises a depth sensor camera, an electronic memory… the memory storing the electronic images of the depth sensor camera… determine from the electronic images 3D point clouds representing the object, temporally in step with the image recording by the camera. The memory further stores a list of 3D coordinates for the nodes of a skeleton model… A skeleton model having only a low node count serves to effectively compress the image information when the coordinates of the nodes at any time describe the position of predetermined parts of the moving object; Par. [0116-120]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person, using image segmentation that is known per se… A depth sensor camera that comprises an arithmetic unit for determining the distance coordinate can in particular also be engineered directly to carry out the inventive method. Such a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts; capturing, by a single motion capture device, visual data of the subject as it moves in a field of vision of the motion capture device (e.g. method for pose estimation of a moving object (i.e. subject), such as a person, for example, includes three dimensional (3D) point clouds that are determined from electronically recorded (i.e. captured, acquired, etc.) images (i.e. capturing visual data of the subject) by a depth sensor camera (i.e. by a single motion capture device), that represent a person (i.e. visual data of the subject as it moves in a field of vision of the motion capture device), for example, to provide 3D coordinates of predetermined body parts of the person that change over time during motion (i.e. capturing, by a single motion capture device, visual data of the subject as it moves in a field of vision of the motion capture device), as indicated above), for example);
from the captured visual data, recognising [recognizing] human pose points on the subject to extract data of the subject (Par. [0001]: method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, i.e. estimating the pose of the movable object; Par. [0024-26]: pose estimation by means of SOM training is proposed that is able to work with a skeleton model that is modelled according to the anatomy of the object observed and exhibits a reduced node count, it being possible to reliably and consistently assign each model node to a predetermined body part. Here the anatomy of the object whose pose is to be detected is modelled as a skeleton model… embodiment of the invention refers to a method for pose estimation of a moving object (e.g. a person or a robot) by computer calculation of displacements of 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds; Par. [0068-73]: the learning rules are formed according to the following concept. Starting from the existence of an image sequence for which continuous pose estimation is to be carried out, the skeleton model that is a list of node positions (described in a 3D coordinate system), node pairs, and optionally node triplets, is in each case updated when a new image of the sequence exists. As soon as the new image is recorded by the depth sensor camera and has been converted by means of image segmentation and projection into a list of 3D coordinates (3D point cloud) for the points of the surface of the object observed--e.g. the entire person, his torso, his hand etc.… determination of the closest topology element is explained in more detail below, and the specific displacement rules for the topology elements are named… At first the crossing points of X in terms of all topology elements are at first determined for a point X of the 3D point cloud--that is to say for a coordinate point in the 3D space that, after imaging of the object observed using a depth sensor camera and subsequent image segmenting and projection, represents a point of the body surface of the person; Par. [0116-120]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person, using image segmentation that is known per se. The anatomic skeleton model is fitted in real time using the learning rules described above, and the association of the model nodes with the different body parts of the person remains correct throughout as can be clearly gathered from FIG. 4… A depth sensor camera that comprises an arithmetic unit for determining the distance coordinate can in particular also be engineered directly to carry out the inventive method. Such a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts; from the captured visual data, recognizing human pose points on the subject to extract data of the subject (e.g. method for pose estimation of a moving object (i.e. subject), such as a person (i.e. human), for example, includes three dimensional (3D) point clouds that are determined from electronically recorded (i.e. captured, acquired, etc.) images (i.e. from the captured visual data), including analysis of a sequence of electronic images for estimating positions and orientations of a movable object captured in the image sequence, for example, to estimate (i.e. recognize, identify, determine, etc.) a pose of the movable object (i.e. from the captured visual data, recognizing human pose points on the subject), for example, including images recorded by the depth sensor camera that have been converted by image segmentation (i.e. to extract data of the subject) and projected into a list of 3D coordinates (3D point cloud) for points of a surface of an object observed (i.e. from the captured visual data, recognizing human pose points on the subject to extract data of the subject), for example, including coordinate points in the 3D space that, after imaging of the object observed using the depth sensor camera and subsequent image segmenting and projection, represent points of the body surface of a person, as indicated above), for example); and
based on the extracted data, recognising [recognizing] a plurality of predefined anatomical components of the subject to construct the model of the subject (Par. [0056-57]: an anatomically motivated skeleton model is now used… The model from FIG. 1 b) is particularly suited that unambiguously associates each node with a distinctive point of human anatomy (e.g. head, shoulder, elbow, hand, hip, pelvis, knee, foot). FIG. 1 c) represents a variant of the model from FIG. 1 b), where the torso is represented by triangles (in each case defined by three nodes that form the corners); Par. [0068-73]: the learning rules are formed according to the following concept. Starting from the existence of an image sequence for which continuous pose estimation is to be carried out, the skeleton model that is a list of node positions (described in a 3D coordinate system), node pairs, and optionally node triplets, is in each case updated when a new image of the sequence exists. As soon as the new image is recorded by the depth sensor camera and has been converted by means of image segmentation and projection into a list of 3D coordinates (3D point cloud) for the points of the surface of the object observed--e.g. the entire person, his torso, his hand etc.… determination of the closest topology element is explained in more detail below, and the specific displacement rules for the topology elements are named… At first the crossing points of X in terms of all topology elements are at first determined for a point X of the 3D point cloud--that is to say for a coordinate point in the 3D space that, after imaging of the object observed using a depth sensor camera and subsequent image segmenting and projection, represents a point of the body surface of the person; Par. [0116-120]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person, using image segmentation that is known per se. The anatomic skeleton model is fitted in real time using the learning rules described above, and the association of the model nodes with the different body parts of the person remains correct throughout as can be clearly gathered from FIG. 4… A depth sensor camera that comprises an arithmetic unit for determining the distance coordinate can in particular also be engineered directly to carry out the inventive method. Such a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts; and based on the extracted data, recognizing a plurality of predefined anatomical components of the subject to construct the model of the subject (e.g. method for pose estimation of a moving object (i.e. subject), such as a person (i.e. human), for example, includes three dimensional (3D) point clouds that are determined (i.e. generated, calculated, computed, estimated, etc.) from electronically recorded images (i.e. captured visual data), including images recorded by a depth sensor camera that have been converted by image segmentation (i.e. based on the extracted data) and projected into a list of 3D coordinates (3D point cloud) for points of a surface of an object observed (i.e. recognized, analyzed, etc.), for example, by computer calculation of displacements (i.e. movements) of 3D position coordinates of nodes of a skeleton model (i.e. the model of the subject) that is modelled according to an anatomy of the object observed, including 3D point clouds that represent a moving person (i.e. recognizing a plurality of components of the subject to construct the model of the subject), for example, including coordinate points in 3D space that, after imaging of the object observed using the depth sensor camera and subsequent image segmenting and projection, represent points of the body surface of a person, for example, by directly translating images of a moving person into 3D coordinates of essential body parts and by assigning each model node to a predetermined body part (i.e. based on the extracted data, recognizing a plurality of predefined anatomical components of the subject to construct the model of the subject), for example, in order to provide 3D coordinates of predetermined body parts of a person that change over time during motion, as indicated above), for example), but fails to teach the following as further recited in claim 1.
However, Kord teaches a biomechanical model of a subject (Par. [0002]: motion capture system records the movements of a motion capture subject in a sequence of digital images, converting the recorded images to a mathematical model… The mathematical model may represent a biomechanical skeleton having rigid links connected to one another by rotatable joints; Par. [0013-22]: FIG. 2 is a pictorial view toward the front of a person acting as a motion capture subject, with the subject standing in an example of a scale frame, the scale frame positioned relative to a camera as in the example of FIG. 1, the subject's right hand at the left side of the figure, and an example of a biomechanical skeleton model overlaid on the motion capture subject showing examples of correspondence between links and joints in the biomechanical skeleton with the person's joints, limbs, head, neck, and torso… A single digital camera records images of a motion capture subject. The images are analyzed to assign values to parameters for a biomechanical skeleton capable of accurately emulating the motion capture subject's movements and body positions; a biomechanical model of a subject (e.g. motion capture system records movements of a motion capture subject (i.e. a subject) in a sequence of digital images, converts the recorded images of the subject to a mathematical model representing a biomechanical skeleton (i.e. a biomechanical model of a subject) having rigid links connected to one another by rotatable joints, such as motion capture subject 148 shown in Fig. 2 below:
PNG
media_image1.png
734
564
media_image1.png
Greyscale
, for example);
capturing, by a single supported motion capture device from a capture position, visual data of the subject (Par. [0009-27]: example of an apparatus embodiment includes a digital camera, a central processing unit in data communication with the digital camera, a memory in data communication with the central processing unit, and an image calibration tool… method embodiment includes the capturing a sequence of digital images of a motion capture subject… FIG. 1 shows a pictorial view of an example of an apparatus for creating accurately calibrated motion capture images from a single stationary digital camera… FIG. 2 is a pictorial view toward the front of a person acting as a motion capture subject, with the subject standing in an example of a scale frame, the scale frame positioned relative to a camera as in the example of FIG. 1, the subject's right hand at the left side of the figure, and an example of a biomechanical skeleton model overlaid on the motion capture subject showing examples of correspondence between links and joints in the biomechanical skeleton with the person's joints, limbs, head, neck, and torso… A single digital camera records images of a motion capture subject… the single camera in an embodiment may remain entirely stationary in a fixed position for capturing images used for determining accurate positions, angles, and distances along all three mutually perpendicular spatial directions. Captured images may quickly and easily be recalibrated should a camera be located to a new viewing location, for example to record a motion capture subject from a new viewing angle… Turning to FIG. 1, an example of an apparatus embodiment 100 includes a camera 114… The camera may optionally be mounted on an adjustable-height tripod 116 or another stable camera support; capturing, by a single supported motion capture device from a capture position, visual data of the subject (e.g. motion capture system that records (i.e. captures, acquires, etc.) movements of a motion capture subject in a sequence of digital images (i.e. capturing visual data of the subject) includes capturing a sequence of digital images of the motion capture subject, including a person (i.e. a human) acting as the motion capture subject No. 148 shown in Fig. 2 below:
PNG
media_image1.png
734
564
media_image1.png
Greyscale
, for example, by using a single digital camera (Fig. 1, No. 114) that remains entirely stationary in a fixed position for capturing (i.e. a capture position) images of the motion capture subject (i.e. capturing, by a single supported motion capture device from a capture position, visual data of the subject), as shown in Fig. 1 below:
PNG
media_image2.png
536
702
media_image2.png
Greyscale
, for example) as it moves between a first distance marker and a second distance marker, the first and second distance markers being disposed at a distance apart from each other and in a field of vision of the motion capture device (Par. [0009-11]: apparatus embodiment of the invention further includes a camera and a computer implemented in hardware, wherein the camera is in data communication with the computer and the computer is adapted to receive an image from the camera, convert the image to a silhouette, and extract parameters for an actor file from the image… method embodiment includes the capturing a sequence of digital images of a motion capture subject, each of the sequence of digital images including an image of an image calibration tool… comparing a separation distance between two calibration markers on the image calibration tool to a corresponding distance between the same two calibration markers a digital image, and overlaying a biomechanical skeleton over each of the sequence of digital images, with a biomechanical reference location superimposed over a movable joint of the motion capture subject and at least one link rotatably coupled to the biomechanical reference location; Par. [0024-33]: image calibration tool enables accurate determination of angles, positions, and lengths in captured images without the use of multiple cameras or multiple camera positions for a single camera… A mathematical model in accord with an embodiment may be referred to as an actor file. An actor file represents a motion capture subject, for example a person engaged in a sports activity or an actor in a video game or motion picture, as an articulated biomechanical skeleton comprising rigid links joined to one another at biomechanical reference locations… image calibration tool 102 optionally includes at least two calibration markers 106… The image calibration tool 102 in the examples of FIGS. 1-2 comprises 12 struts and eight calibration markers 106, including an upper right front ball 132, an upper left front ball 134, a lower right front ball 136, and a lower left front ball 138, where left and right have been labeled with respect a viewing direction along the optical axis 128 toward the image calibration tool 102. Continuing on the back side of the image calibration 102, an upper right back ball 140, an upper left back ball 142, a lower right back ball 144, and a lower left back ball 146 are joined to one another and to the front balls by struts… A digital image captured by the digital camera 114 may be processed by a CPU included in some embodiments to extract parameters for an actor file… An image of a motion capture subject 148 may optionally be processed by the computer (ref. FIG. 1) to form a silhouette 150 corresponding to an outline of the person 148 from the camera's viewing direction. An example of a silhouette is shown in FIG. 3. As the motion capture subject moves, the camera collects images, for example a sequence of video images recorded at 30 frames per second or a sequence of still images recorded at selected positions of the subject 148; Par. [0042-51]: method… includes the steps of capturing a sequence of digital images of a motion capture subject, each of the sequence of digital images including an image of an image calibration tool, determining a separate scale factor for each of three mutually orthogonal spatial directions by comparing a separation distance between two calibration markers on the image calibration tool to a corresponding distance between the same two calibration markers in each of said sequence of digital images, and overlaying a biomechanical skeleton over each of the sequence of digital images with a biomechanical reference location superimposed over a movable joint of the motion capture subject and at least one link rotatably coupled to the biomechanical reference location… positioning the camera 114 along the y-axis at a distance 118 selected to fit the motion capture subject 148 into the camera's field of view; as it moves between a first distance marker and a second distance marker, the first and second distance markers being disposed at a distance apart from each other and in a field of vision of the motion capture device (e.g. motion capture system that records (i.e. captures, acquires, etc.) movements of a motion capture subject in a sequence of digital images (i.e. capturing visual data of the subject) includes capturing a sequence of digital images of the motion capture subject, including a person (i.e. a human) acting as the motion capture subject No. 148 shown in Fig. 2 below:
PNG
media_image1.png
734
564
media_image1.png
Greyscale
, for example, by using a single digital camera (Fig. 1, No. 114) that remains entirely stationary in a fixed position for capturing (i.e. a capture position) images of the motion capture subject (i.e. capturing, by a single supported motion capture device from a capture position, visual data of the subject), as shown in Fig. 1 below:
PNG
media_image2.png
536
702
media_image2.png
Greyscale
, for example, in which each of the sequence of digital images includes of a calibration tool captured by using the single digital camera, for example, including two calibration markers (i.e. a first distance marker and a second distance marker) separated by a distance measured between the first and second calibration markers and a corresponding distance (i.e. the first and second distance markers being disposed at a distance apart from each other) in the captured images within the camera's field of view (i.e. and in a field of vision of the motion capture device), for example, and as the motion capture subject moves (i.e. as it moves in a field of vision of the motion capture device), the camera collects images of the motion capture subject to perform measurements of the motion capture subject (i.e. capturing, by a single supported motion capture device from a capture position, visual data of the subject as it moves between a first distance marker and a second distance marker, the first and second distance markers being disposed at a distance apart from each other and in a field of vision of the motion capture device), as indicated above), for example).
Martinetz and Kord are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with a biomechanical model of a subject and capturing, by a single supported motion capture device from a capture position, visual data of the subject as it moves in a field of vision of the motion capture device (as taught by Kord, Abstract, Par. [0009-51]) to create an articulated, movable mathematical model of a person from position, size, and angle information extracted from digital camera images, to accurately emulate a motion capture subject's movements and body positions, to create accurately calibrated motion capture images from a single stationary digital camera, and to enable accurate determination of angles, positions, and lengths in captured images (Kord, Abstract, Par. [0001-51]).
The combination of Martinetz and Kord, as a whole, teaches the method, as indicated above, but fails to teach the following as further recited in claim 1.
However, McGibbon teaches kinematic data (Par. [0001-7]: system and method for analyzing kinetic and kinematic information of human motion, and for viewing the information… present invention addresses these drawbacks by providing a full four-dimensional analysis (three space dimensions, one time dimension) of human movement data captured by a motion analysis system. The invention enables detailed biomechanical analysis of human movement data, as well as the visualization of data… a system for displaying kinematic and kinetic information of a subject is provided. The system includes an image input stage for acquiring image data of the subject, a transformation stage for transforming the image data into three dimensional coordinates corresponding to one or more body segments of the subject, and an output data stage for calculating the kinematic and kinetic information of the subject from the three dimensional coordinates… method comprises the steps of acquiring image data of the subject, transforming the image data into three dimensional coordinates corresponding to one or more body segments of the subject, and calculating the kinematic and kinetic information of the subject from the three dimensional coordinates; Par. [0020-28]: system and method, and a software facility, for the analysis of kinematics and kinetics of human movement. The present invention utilizes an eleven segment three-dimensional model of human movement analysis… FIG. 1 is a schematic block diagram of a movement analysis system… present invention relies on the acquisition of image data to provide an accurate estimation of movement. The image input stage 2 is utilized for acquiring, obtaining or receiving image data needed for the movement analysis system. The image input stage 2 can be any device or structure suitable for receiving, obtaining or acquiring image data… The image data acquired by the image input stage is converged to the transformation stage 4 which utilizes the acquired image data to track and build a 3-D model of the human body. In particular, the transformation stage 4 performs the coordinate transformation needed to calculate the various kinetics and kinematics discussed in more detail below. The output data stage 6 generates output containing an array of information used in modeling human movement. In particular, the output data stage 6 provides output analysis for the various kinematic and kinetic parameters… FIG. 7 is a schematic block diagram illustration of the output data stage 6 of FIG. 1. The output data stage 6 generates numerous output files containing a variety of useful biomechanical measures. In general, the output data stage 6 provides the kinematic output information and kinetic output information. The illustrated kinematic analysis module 64 provides for kinematic analysis on all of the eleven segmented body parts mentioned above. In particular, the kinematic analysis module 64 provides for a greater understanding of how the body of the subject 30 move relative to one another (coordination), as well as the rates at which they move (velocities). Thus, the kinematic analysis module 64 includes analysis information regarding the subject's bodily motions; kinematic data (e.g. system and method for analyzing kinetic and kinematic information (i.e. kinematic data) of human motion and for providing detailed biomechanical analysis of human movement data includes acquiring image data of a subject, transforming the image data into three dimensional coordinates (i.e. 3D point cloud) corresponding to one or more body segments of the subject (i.e. predefined anatomical components of the subject), for example, and calculating (i.e. extracting, computing, etc.) the kinematic information of the subject from the three dimensional coordinates, as indicated above), for example).
Martinetz, Kord, and McGibbon are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, and McGibbon, as a whole, would have rendered obvious the invention recited in claim 1 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with kinematic data (as taught by McGibbon, Abstract, Par. [0001-7, 20-28]) to provide a software facility for computing and displaying kinematic and kinetic information of a subject to a user, to analyze various human movements, to provide output analysis for various kinematic and kinetic parameters, thus allowing a more detail output of various modeled segments acquired, to provide an accurate estimation of movement, and to analyze aspects of motion of a body and body segments (McGibbon, Abstract, Par. [0001-7, 20-29]).
The combination of Martinetz, Kord, and McGibbon, as a whole, teaches the method, as indicated above, but fails to teach the object “moves from a first distance marker to a second distance marker”, as now recited in claim 1.
However, MALHOTRA teaches moves from a first distance marker to a second distance marker (Par. [0006-11]: system and methods configured to improve the athlete's ability to synchronize the movement of their body in time… an image capturing device may capture images at different or variable frame rates… determinations of movement (and/or movement quality) may utilize rate of capture information… landmarks/distance calibrations may be utilized from time-stamped image data to allow for precise measuring of performance. For example, objects represented by image data may be utilized to determine whether movement thresholds are met. For example, markings on a field (such as yard lines) may be used to calibrate distance measurements… image data may indicate that the athlete is performing a second or additional features of a first athletic movement; Par. [0029]: determining athletic attributes of an athlete from image data… an athlete's performance of a physical activity, such as for example, a sprint or agility drill, or battery of field-based tests, may be analyzed according to image data… landmarks/distance calibrations may be utilized from time-stamped image data to allow for precise measuring of performance (including, but not limited to: sprint or agility times, flight time for vertical jump, distance for throws). Data retrieved or derived from the image data may be used in scoring and/or ranking athletes; Par. [0070-77]: processing image data, such as the image data captured… a location of a body portion of the athlete, or a location of a sporting device within the image data (absolute location or relative to another object, such as the athlete any other object represented by image data) may be used to detect the performance of one or more features of specific movements… landmarks/distance calibrations may be utilized from time-stamped image data to allow for precise measuring of performance. For example, objects represented by image data may be utilized to determine whether movement thresholds are met (e.g., block 804b). For example, markings on a field (such as yard lines) may be used to calibrate distance measurements… dimensions may be used to identify thresholds and/or determine whether certain thresholds have been met, including but not limited to: flight time for vertical jump, distance for throws, kick distance and power, among others, sprint times between two distances; Par. [0087-88]: identifying image data (such as but not limited to specific images) that correlate to a specific physical movement or activity of the athlete (e.g., block 810). For example, certain embodiments may detect initiation of the athletic activity, performance of an athletic movement, and/or the occurrence of a movement feature. As non-limiting examples, image data may be used to identify one or more actions, including: initiation of the activity, levels of acceleration, velocity, reaction, and/or tempo, distance traveled by an object, completion of the activity, among others… identify image data (including specific frames or images) such as including, but not limited to: an initiation image, a termination image, or any other image comprising motion data that can be identified based upon the systems or methods disclosed herein; Par. [0118-120]: 9A and 9B show example activity spaces comprising illustrative test elements. With reference to FIG. 9A, activity space 900 illustrates example test elements. Activity space 900 is associated with an Arrowhead Agility performance test, which test may be various configured with respect to test elements. As to test elements, activity space 900 may include area 902, equipment 906A-F, and a start-finish line 908… Within area 902, the equipment 906A-F may be positioned at prescribed locations… The equipment 906A-F may indicate to the athlete the start-stop line 908… the equipment 906A-F includes markers, such as cones, arranged in a predetermined formation: (i) four cones 906A, B, C, E are positioned to form a square with ten-meter sides… and (iii) cone 906D is positioned on a line perpendicular to the line formed by cones 906C, 906F and 906E, at a distance, e.g., five meters, from 906F, distally from cones 906A, B… an activity may contemplate various constituents, and each constituent may contemplate various components performed in an activity space… constituents or components may include one or more of the following… athlete's time from the start-stop line 908 to initiation of rounding of cone 906F… athlete's time from cone 906F to cone 906E… athlete's time from cone 906E to cone 906D… athlete's time from cone 906D to start-stop line 908; moves from a first distance marker to a second distance marker (e.g. image capturing device captures images, including image data captured of an object (i.e. subject), such as an athlete or any other object represented by image data, for example, to perform measurements by using landmarks/distance calibrations utilized from time-stamped image data to allow for precise measuring of performance, including markers, such as cones, for example, arranged in a predetermined formation, including cones placed (i.e. disposed, located, etc.) at a distance from other cones, such as five meters (i.e. distance markers being disposed at a distance apart from each other and in a field of vision of the motion capture device), for example, and identifying image data that correlates to a specific physical movement or activity of the object in a field of vision of the motion capture device, including a distance traveled by an object, sprint times between two distances, etc., for example, including determining an object’s (i.e. athlete's) movement time from an initial cone (i.e. first distance marker), such as cone 906A, for example, to another cone (i.e. second, third… Nth distance marker), such as cones 906C-F, for example, as the object moves from one cone to another (i.e. moves from a first distance marker to a second distance marker), as shown in Fig. 9A below:
PNG
media_image3.png
744
914
media_image3.png
Greyscale
, for example, and detecting the performance of one or more features of specific movements of the object in order to determine athletic attributes the object (i.e. an athlete) from image data, as indicated above), for example).
Martinetz, Kord, McGibbon, and MALHOTRA are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, would have rendered obvious the invention recited in claim 1 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with moves from a first distance marker to a second distance marker (as taught by MALHOTRA, Abstract, Par. [0006-11, 29, 70-77, 87-88, 118-120]) to improve a feedback system that allows an athlete (or another individual, such as a trainer) to improve performance for a particular sport or athletic activity, to provide feedback to an athlete regarding the athlete's performance of a physical activity, to comprehend and optimize the timing of one or more components or features of an athletic movement, and to determine athletic attributes the object from image data (MALHOTRA, Abstract, Par. [0002-11, 29, 70-77, 87-88, 118-120]).
Regarding claim 2, claim 1 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]) wherein the captured visual data includes a plurality of video frames (Martinetz, Par. [0116]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person), and the constructed biomechanical model is based on at least one frame of the plurality of video frames capturing subject in a predefined stance (Kord, par. [0033-54]: As the motion capture subject moves, the camera collects images, for example a sequence of video images recorded at 30 frames per second or a sequence of still images recorded at selected positions of the subject 148… By measuring the positions of biomechanical reference locations in images of the subject and compensating the measured values with scaling information derived from images of the image calibration tool, the spatial coordinates may be determined for each biomechanical reference location on the biomechanical skeleton… positioning the motion capture subject 148 in the center of the camera's field of view, facing the camera 114 in a relaxed posture and close enough to the camera to achieve desired image resolution, also referred to as an initialization pose or alternately as a master pose; and the constructed biomechanical model is based on at least one frame of the plurality of video frames capturing subject in a predefined stance (e.g. motion capture system records movements of a motion capture subject in a sequence of digital images (i.e. capturing visual data of the subject) includes measuring positions of biomechanical reference (i.e. a predefined) locations in images of the subject, including images recorded at selected (i.e. predefined, predetermined, etc.) positions of the subject (i.e. and the constructed biomechanical model is based on at least one frame of the plurality of video frames capturing subject in a predefined stance), for example, and by measuring positions of biomechanical reference locations in images of the subject and compensating the measured values with scaling information derived from images of the image calibration tool, spatial coordinates are determined for each biomechanical reference location on the biomechanical skeleton, as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 4, claim 1 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]) wherein the motion capture device is substantially stationarily supported (Par. [0009-27]: example of an apparatus embodiment includes a digital camera, a central processing unit in data communication with the digital camera, a memory in data communication with the central processing unit, and an image calibration tool… method embodiment includes the capturing a sequence of digital images of a motion capture subject… FIG. 1 shows a pictorial view of an example of an apparatus for creating accurately calibrated motion capture images from a single stationary digital camera… FIG. 2 is a pictorial view toward the front of a person acting as a motion capture subject, with the subject standing in an example of a scale frame, the scale frame positioned relative to a camera as in the example of FIG. 1, the subject's right hand at the left side of the figure, and an example of a biomechanical skeleton model overlaid on the motion capture subject showing examples of correspondence between links and joints in the biomechanical skeleton with the person's joints, limbs, head, neck, and torso… A single digital camera records images of a motion capture subject… the single camera in an embodiment may remain entirely stationary in a fixed position for capturing images used for determining accurate positions, angles, and distances along all three mutually perpendicular spatial directions. Captured images may quickly and easily be recalibrated should a camera be located to a new viewing location, for example to record a motion capture subject from a new viewing angle… Turning to FIG. 1, an example of an apparatus embodiment 100 includes a camera 114… The camera may optionally be mounted on an adjustable-height tripod 116 or another stable camera support; wherein the motion capture device is substantially stationarily supported (e.g. motion capture system that records movements of a motion capture subject in a sequence of digital images (i.e. capturing visual data of the subject) includes capturing a sequence of digital images of the motion capture subject, including a person (i.e. a human) acting as the motion capture subject, for example, by using a single digital camera (Fig. 1, No. 114) that remains entirely stationary in a fixed position for capturing (i.e. a capture position) images of the motion capture subject (i.e. wherein the motion capture device is substantially stationarily supported), as shown in Fig. 1 below:
PNG
media_image2.png
536
702
media_image2.png
Greyscale
, for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 5, claim 1 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]) wherein the motion capture device is a camera (Martinetz, par. [0026-38]: method for pose estimation of a moving object (e.g. a person or a robot) by computer calculation of displacements of 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds. The node coordinates are present in table form in an electronic memory and the 3D point clouds are determined from electronically recorded images from a depth sensor camera that represent the moving person… an apparatus for pose estimation of a moving object. This apparatus comprises a depth sensor camera, an electronic memory, and a programmable arithmetic unit, the memory storing the electronic images of the depth sensor camera and the arithmetic unit being designed to determine from the electronic images 3D point clouds representing the object, temporally in step with the image recording by the camera).
Regarding claim 6, claim 5 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches a the method (Martinetz, Par. [0001-5]) wherein the camera is a smartphone camera (Kord, Par. [0040]: In the example of FIG. 1, the digital camera 114 and computer 122 are provided as separated devices. In alternative embodiments, the camera and computer may be combined into a single device, for example the portable computer 182 with an integrated camera in FIG. 6. Examples of a portable computer with an integrated camera suitable for use with an embodiment include, but are not limited to, a laptop computer, a tablet computer, a personal digital assistant, and a smart phone having cellular telephone combined with a CPU, data and program memory).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 9, claim 1 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]) wherein the visual data of the subject is captured without the use of wearable subject makers on the subject (Martinetz, Par. [00119-120]: the method described for pose estimation can be carried out by an apparatus… a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts. This is comparable to a motion capture apparatus where, however, the markers on the body of the person that until now were common, can be dispensed with).
Regarding claim 11, claim 1 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]) including the further step of formulating based on the constructed biomechanical model, a motion performance metric (McGibbon, Par. [0005-30]: providing a full four-dimensional analysis (three space dimensions, one time dimension) of human movement data captured by a motion analysis system. The invention enables detailed biomechanical analysis of human movement data, as well as the visualization of data… a system and method, and a software facility, for the analysis of kinematics and kinetics of human movement… output data stage 6 generates numerous output files containing a variety of useful biomechanical measures… the output data stage 6 provides the kinematic output information and kinetic output information… determine the forces that interact among the various body segments of the subject 30… body forces and torques 80 are useful in evaluating athletic performance; formulating based on the constructed biomechanical model, a motion performance metric (e.g. system and method for analyzing kinetic and kinematic information (i.e. kinematic data) of human motion and for providing detailed biomechanical analysis of human movement data includes acquiring image data of a subject, transforming the image data into three dimensional coordinates (i.e. 3D point cloud) corresponding to one or more body segments of the subject (i.e. the constructed biomechanical model), for example, and provides kinematic output information that determines forces that interact among various body segments of human movement of a subject, for example, including body forces and torques that are useful in evaluating (i.e. measure, metric, appraise, etc.) athletic performance (i.e. formulating based on the constructed biomechanical model, a motion performance metric), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 12, claim 1 [11] incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]) including the further step of outputting the motion performance metric for visual display on a display device (McGibbon, Par. [0005-31]: providing a full four-dimensional analysis (three space dimensions, one time dimension) of human movement data captured by a motion analysis system. The invention enables detailed biomechanical analysis of human movement data, as well as the visualization of data… providing a software facility for computing and displaying kinematic and kinetic information to a user… a system for displaying kinematic and kinetic information of a subject is provided. The system includes an image input stage for acquiring image data of the subject, a transformation stage for transforming the image data into three dimensional coordinates corresponding to one or more body segments of the subject, and an output data stage for calculating the kinematic and kinetic information of the subject from the three dimensional coordinates. The system can also include a user interface for displaying the calculated kinematic and kinetic information of the subject… a method for displaying kinematic and kinetic information of a subject is also provided… a system and method, and a software facility, for the analysis of kinematics and kinetics of human movement… user interface 8 displays in various formats the calculated outputs of the output data stage 6… output data stage 6 generates numerous output files containing a variety of useful biomechanical measures… the output data stage 6 provides the kinematic output information and kinetic output information… determine the forces that interact among the various body segments of the subject 30… body forces and torques 80 are useful in evaluating athletic performance… FIG. 10 is a detailed depiction of the user interface 8. The user interface 8 is a flexible tool for analyzing and displaying the output data stage 6 information; outputting the motion performance metric for visual display on a display device (e.g. system and method for analyzing kinetic and kinematic information (i.e. kinematic data) of human motion and for providing detailed biomechanical analysis of human movement data includes acquiring image data of a subject, transforming the image data into three dimensional coordinates (i.e. 3D point cloud) corresponding to one or more body segments of the subject (i.e. the constructed biomechanical model), for example, and provides kinematic output information that determines forces that interact among various body segments of human movement of a subject, for example, including body forces and torques that are useful in evaluating (i.e. measure, metric, appraise, etc.) athletic performance (i.e. motion performance metric), for example, and displaying the output data via a user interface display (i.e. outputting the motion performance metric for visual display on a display device), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 13, is a corresponding apparatus claim rejected as applied to the method claim 1 above.
Regarding claim 14, claim 1 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]) including the further steps of:
recognising [recognizing] a captured length of a component of the subject having a known real-world length (Kord, Par. [0011-32]: method embodiment includes the capturing a sequence of digital images of a motion capture subject… overlaying a biomechanical skeleton over each of the sequence of digital images, with a biomechanical reference location superimposed over a movable joint of the motion capture subject and at least one link rotatably coupled to the biomechanical reference location. A length of the link in each image corresponds to a projected length. A true length of each link is determined… A joint in the biomechanical skeleton is positioned at the biomechanical reference location and a link having a true length is coupled to the joint... A single digital camera records images of a motion capture subject. The images are analyzed to assign values to parameters for a biomechanical skeleton… Image scale information for creating the biomechanical skeleton may be determined along each of the three mutually perpendicular spatial axes by comparing known dimensions for an image calibration tool with corresponding dimensions of the tool's image recorded with images of the subject. After image scale has been determined for all three spatial axes in recorded images, the true length and true position of each link and joint in the biomechanical skeleton may be determined accurately from recorded images so that motions and positions of the motion capture subject may be accurately reproduced by the biomechanical skeleton… By comparing known sizes and positions of components on the image scale tool to sizes and positions of the same components measured in captured images, true dimensions, true angles, and true positions of objects in captured images may be determined accurately from the images. Examples of parameters which may be determined accurately from calibrated images of a motion capture subject include, but are not limited to, limb length, joint position, limb and joint positions with respect to a position reference, limb angles, and distances traversed by the motion capture subject. The image calibration tool enables accurate determination of angles, positions, and lengths in captured images… Known dimensions, shapes, and positions of the calibration markers with respect to one another are used to create calibrated digital images of the image calibration tool… The known lengths of each strut and the known diameter of each ball in the image calibration tool may be compared to their dimensions in a camera image of the image calibration tool to determine dimensions, angles, and positions for other objects in the image; recognizing a captured length of a component of the subject having a known real-world length (e.g. motion capture system records movements of a motion capture subject (i.e. a subject) in a sequence of digital images, converts the recorded images of the subject to a mathematical model representing a biomechanical skeleton, for example, includes determination (i.e. recognition, identification, calculation, etc.) of angles, positions, and lengths in captured images of a motion capture subject (i.e. captured length of a component) based on known dimensions (i.e. a known real-world length), shapes, and positions of calibration markers, with respect to one another (i.e. recognizing a captured length of a component of the subject having a known real-world length), for example, which are used to create calibrated digital images of the image calibration tool, as indicated above), for example);
mapping the known real-world length of the component to the captured length of the component (Kord, Par. [0002-32]: motion capture system records the movements of a motion capture subject in a sequence of digital images, converting the recorded images to a mathematical model that may be manipulated in a computer system. The mathematical model may represent a biomechanical skeleton having rigid links connected to one another by rotatable joints. A computer graphics system may map images of a character in a motion picture or video game onto the links and joints of the biomechanical skeleton to cause the character's movements to emulate the motion capture subject's movements… method embodiment includes the capturing a sequence of digital images of a motion capture subject… overlaying a biomechanical skeleton over each of the sequence of digital images, with a biomechanical reference location superimposed over a movable joint of the motion capture subject and at least one link rotatably coupled to the biomechanical reference location. A length of the link in each image corresponds to a projected length. A true length of each link is determined… A joint in the biomechanical skeleton is positioned at the biomechanical reference location and a link having a true length is coupled to the joint... A single digital camera records images of a motion capture subject. The images are analyzed to assign values to parameters for a biomechanical skeleton… Image scale information for creating the biomechanical skeleton may be determined along each of the three mutually perpendicular spatial axes by comparing known dimensions for an image calibration tool with corresponding dimensions of the tool's image recorded with images of the subject. After image scale has been determined for all three spatial axes in recorded images, the true length and true position of each link and joint in the biomechanical skeleton may be determined accurately from recorded images so that motions and positions of the motion capture subject may be accurately reproduced by the biomechanical skeleton… By comparing known sizes and positions of components on the image scale tool to sizes and positions of the same components measured in captured images, true dimensions, true angles, and true positions of objects in captured images may be determined accurately from the images. Examples of parameters which may be determined accurately from calibrated images of a motion capture subject include, but are not limited to, limb length, joint position, limb and joint positions with respect to a position reference, limb angles, and distances traversed by the motion capture subject. The image calibration tool enables accurate determination of angles, positions, and lengths in captured images… Known dimensions, shapes, and positions of the calibration markers with respect to one another are used to create calibrated digital images of the image calibration tool… The known lengths of each strut and the known diameter of each ball in the image calibration tool may be compared to their dimensions in a camera image of the image calibration tool to determine dimensions, angles, and positions for other objects in the image; mapping the known real-world length of the component to the captured length of the component (e.g. motion capture system records movements of a motion capture subject (i.e. a subject) in a sequence of digital images, converts the recorded images of the subject to a mathematical model representing a biomechanical skeleton, for example, includes determination (i.e. recognition, identification, calculation, etc.) of angles, positions, and lengths in captured images of a motion capture subject (i.e. captured length of the component) based on known dimensions (i.e. known real-world length), shapes, and positions of calibration markers, with respect to one another, for example, which are used to create calibrated digital images of the image calibration tool, for example and map images of a character in a motion picture or video game onto the links and joints of the biomechanical skeleton to cause the character's movements to emulate the motion capture subject's movements (i.e. mapping the known real-world length of the component to the captured length of the component), as indicated above), for example); and
based on the biomechanical model, formulating a motion performance metric (McGibbon, Par. [0005-30]: providing a full four-dimensional analysis (three space dimensions, one time dimension) of human movement data captured by a motion analysis system. The invention enables detailed biomechanical analysis of human movement data, as well as the visualization of data… a system and method, and a software facility, for the analysis of kinematics and kinetics of human movement… output data stage 6 generates numerous output files containing a variety of useful biomechanical measures… the output data stage 6 provides the kinematic output information and kinetic output information… determine the forces that interact among the various body segments of the subject 30… body forces and torques 80 are useful in evaluating athletic performance; based on the biomechanical model, formulating a motion performance metric (e.g. system and method for analyzing kinetic and kinematic information (i.e. kinematic data) of human motion and for providing detailed biomechanical analysis of human movement data includes acquiring image data of a subject, transforming the image data into three dimensional coordinates (i.e. 3D point cloud) corresponding to one or more body segments of the subject (i.e. the constructed biomechanical model), for example, and provides kinematic output information that determines forces that interact among various body segments of human movement of a subject, for example, including body forces and torques that are useful in evaluating (i.e. measure, metric, appraise, etc.) athletic performance (i.e. based on the biomechanical model, formulating a motion performance), as indicated above), for example), wherein the biomechanical model includes real-world lengths based on the mapping (Kord, Par. [0002-32]: motion capture system records the movements of a motion capture subject in a sequence of digital images, converting the recorded images to a mathematical model that may be manipulated in a computer system. The mathematical model may represent a biomechanical skeleton having rigid links connected to one another by rotatable joints. A computer graphics system may map images of a character in a motion picture or video game onto the links and joints of the biomechanical skeleton to cause the character's movements to emulate the motion capture subject's movements… method embodiment includes the capturing a sequence of digital images of a motion capture subject… overlaying a biomechanical skeleton over each of the sequence of digital images, with a biomechanical reference location superimposed over a movable joint of the motion capture subject and at least one link rotatably coupled to the biomechanical reference location. A length of the link in each image corresponds to a projected length. A true length of each link is determined… A joint in the biomechanical skeleton is positioned at the biomechanical reference location and a link having a true length is coupled to the joint... A single digital camera records images of a motion capture subject. The images are analyzed to assign values to parameters for a biomechanical skeleton… Image scale information for creating the biomechanical skeleton may be determined along each of the three mutually perpendicular spatial axes by comparing known dimensions for an image calibration tool with corresponding dimensions of the tool's image recorded with images of the subject. After image scale has been determined for all three spatial axes in recorded images, the true length and true position of each link and joint in the biomechanical skeleton may be determined accurately from recorded images so that motions and positions of the motion capture subject may be accurately reproduced by the biomechanical skeleton… By comparing known sizes and positions of components on the image scale tool to sizes and positions of the same components measured in captured images, true dimensions, true angles, and true positions of objects in captured images may be determined accurately from the images. Examples of parameters which may be determined accurately from calibrated images of a motion capture subject include, but are not limited to, limb length, joint position, limb and joint positions with respect to a position reference, limb angles, and distances traversed by the motion capture subject. The image calibration tool enables accurate determination of angles, positions, and lengths in captured images… Known dimensions, shapes, and positions of the calibration markers with respect to one another are used to create calibrated digital images of the image calibration tool… The known lengths of each strut and the known diameter of each ball in the image calibration tool may be compared to their dimensions in a camera image of the image calibration tool to determine dimensions, angles, and positions for other objects in the image; wherein the biomechanical model includes real-world lengths based on the mapping (e.g. motion capture system records movements of a motion capture subject (i.e. a subject) in a sequence of digital images, converts the recorded images of the subject to a mathematical model representing a biomechanical skeleton, for example, includes determination (i.e. recognition, identification, calculation, etc.) of angles, positions, and lengths in captured images of a motion capture subject (i.e. captured length of the component) based on known dimensions (i.e. real-world lengths), shapes, and positions of calibration markers, with respect to one another, for example, which are used to create calibrated digital images of the image calibration tool, for example and map images of a character in a motion picture or video game onto the links and joints of the biomechanical skeleton to cause the character's movements to emulate the motion capture subject's movements (i.e. wherein the biomechanical model includes real-world lengths based on the mapping), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Martinetz, in view of Kord, in further view of McGibbon, in further view of MALHOTRA, as applied to claim 1 above, and in further view of Folland et al. (US PG Publication No. 2018/0279916 A1), hereafter referred to Folland.
Regarding claim 3, claim 2 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]), but fails to teach the following as further recited in claim 3.
However, Folland teaches wherein the predefined stance is one or more of:
a toe-off stance whereby a back foot of the subject is lifted off a ground push-off point;
a touch down stance whereby a front foot of the subject is about to contact a ground drop point; and
a full support stance whereby the front foot flattens on the ground drop point and whereby hip and heel human pose points of the subject are vertically aligned (Par. [0009-58]: system for monitoring the running technique of a user undertaking a physical activity… sensors may also detect at least three parameters relating to the movement of the user… It has also been found that the analysis of the parameters relating to ground contact, stride pattern and centre of mass of the user can have utility in the assessment of running technique… the one or more sensors may detect at least one parameter relating to the ground contact of the user, such as relating to: ground contact time (GCT); flight time (FLT); duty factor (DF); touch down to centre of mass (CM) distance, such as the anterior-posterior distance between the CM and toe at touch down… take-off to centre of mass distance, such as the anterior-posterior distance between the CM and toe at take-off… or ground contact distance, such as the sum of the anterior-posterior distance between the CM and toe at touch down and take-off… the one or more sensors may detect for example: a parameter relating to duty factor and a parameter relating to take-off to centre of mass distance, such as the anterior-posterior distance between the CM and toe at take-off; or a parameter relating to the ground contact distance… and a parameter relating to lower spine angle, preferably relative to the angle during a standing stance… the one or more sensors may detect at least one parameter relating to the stride pattern of the user, such as relating to: stride rate (SR) or stride length, such as the anterior-posterior distance covered by the CM during a complete stride (e.g. right foot touch down to next right foot touch down)… change in velocity of the centre of mass of the user, such as the difference in anterior-posterior velocity of the CM between the minimum and maximum, for example during stance, such as around take-off; or change in vertical position of the centre of mass of the user, such as the difference between the highest and lowest vertical position of the CM during each step (right foot touch down to left foot touch down)… comparison with the biomechanical model may comprise an analysis of: the velocity of the pelvis, preferably the minimum forward pelvic velocity, and ground contact time… duty factor and take-off to centre of mass distance, such as the anterior-posterior distance between the CM and toe at take-off… anterior-posterior velocity of the CM between the minimum and maximum, for example during stance, such as around take-off; or ground contact distance… and lower spine angle, preferably relative to the angle during a standing stance… utilise at least one sensor attached to the body of the individual, attached to a garment worn by the individual, attached to or incorporated in a shoe or shoes worn by the individual, incorporated in a device carried by or attached to the individual, such as a portable electronic device or watch, or may utilise other methodology such as video recording and analysis… the method may comprise the measurement of at least one parameter relating to: the ground contact of the individual, such as relating to ground contact time (GCT); flight time (FLT); duty factor (DF); touch down to centre of mass distance (CM), such as the anterior-posterior distance between the CM and toe at touch down… take-off to centre of mass distance, such as the anterior-posterior distance between the CM and toe at take-off… or ground contact distance, such as the sum of the anterior-posterior distance between the CM and toe at touch down and take-off… method may comprise the measurement of at least one parameter relating to: the stride pattern of the individual, such as relating to stride rate (SR); or stride length, such as the anterior-posterior distance covered by the CM during a complete stride (e.g. right foot touch down to next right foot touch down); wherein the predefined stance is one or more of: a toe-off stance whereby a back foot of the subject is lifted off a ground push-off point; a touch down stance whereby a front foot of the subject is about to contact a ground drop point (e.g. system for monitoring the running technique of a user undertaking a physical activity includes video recording and analysis to detect parameters relating to the user’s stride pattern, such as a toe at take-off parameter of the user (i.e. predefined stance is one or more of: a toe-off stance whereby a back foot of the subject is lifted off a ground push-off point), for example, and a toe at touch down parameter of the user (i.e. a touch down stance whereby a front foot of the subject is about to contact a ground drop point), as indicated above), for example).
Martinetz, Kord, McGibbon, MALHOTRA, and Folland are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, McGibbon, MALHOTRA, and Folland, as a whole, would have rendered obvious the invention recited in claim 3 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with wherein the predefined stance is one or more of: a toe-off stance whereby a back foot of the subject is lifted off a ground push-off point; a touch down stance whereby a front foot of the subject is about to contact a ground drop point (as taught by Folland, Abstract, Par. [0009-58]) by using kinematic variables and biomechanical models for monitoring, assessing and improving a running technique of an individual undertaking a physical activity, to improve running performance and to prevent and/or to reduce injury, and to monitor the form and technique of the user and provide improvement feedback (Folland, Abstract, Par. [0001-10, 52, 110]).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Martinetz, in view of Kord, in further view of McGibbon, in further veiew of MALHOTRA, as applied to claim 1 above, and in further view of Aginsky et al. (US PG Publication No. 2019/0274614 A1), hereafter referred to Aginsky.
Regarding claim 7, claim 5 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches a [the] method (Martinetz, Par. [0001-5]), but fails to teach the following as further recited in claim 7.
However, Aginsky teaches wherein the camera is an internet protocol (IP) camera (par. [0077-81]: Computing device 204 receives images (e.g., video) and/or body part locations of the target individual captured by one or more sensors(s) 212. The body part locations correspond to the captured images. The body part locations may be computed based on an analysis of the captured images (e.g., based on visual markers denoting the body part locations, and/or by code that performs a 3D analysis of the images), and/or the body part locations may be computed by a kinetic sensor component of sensor(s) 212 that captures depth data… sensor(s) 212 includes video camera(s) that captures the video and code for analyzing the video for computing the body location data… Sensor(s) 212 may be implemented as an external device and/or integrated into a computing device, for example, an IP camera and/or camera of a smartphone).
Martinetz, Kord, McGibbon, MALHOTRA, and Aginsky are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, McGibbon, MALHOTRA, and Aginsky, as a whole, would have rendered obvious the invention recited in claim 7 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with wherein the camera is an internet protocol (IP) camera (as taught by Aginsky, Abstract, Par. [0077-81]) by using systems and methods for assessment of a musculoskeletal profile of a target individual to improve athletic performance, to estimate a value indicative of muscle strength ratio of at least one target muscle of a target individual based on an analysis of a plurality of digital images, to convert image-parameter to an estimate of a measured-parameter indicative of strength measurement ratios of at least one target muscle obtained, according to correlation code that correlates between image-parameters and measured-parameters obtained, to identify a plurality of image state-frames, each image state-frame corresponding to a certain state of a plurality of states of the certain physical movement test, and identifying the first and second defined movements according to corresponding image state-frames based on a predefined order of the corresponding plurality of states of the physical movement test, to compute respective image-metrics (Aginsky, Abstract, Par. [0001-7, 33, 47, 51, 54, 82]).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Martinetz, in view of Kord, in further view of McGibbon, in further view of MALHOTRA, as applied to claim 1 above, and in further view of Sullivan et al. (US PG Publication No. 2010/0164862 A1), hereafter referred to Sullivan.
Regarding claim 8, claim 1 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]), but fails to teach the following as further recited in claim 8.
However, Sullivan teaches wherein the motion capture device includes two synchronised cameras (Par. [0005-28]: system includes a visual data collector for collecting visual information from a first image of at least a first feature of an object… determine the position of a representation of the first feature in a virtual representation of the object from the combined visual information… motion data combiner may use the visual information to define a pose of the object… The motion data combiner may be configured to synchronize the visual information… While the one camera 206 is illustrated as being included in the visual motion sensing system 104, two or more cameras (which are typically synchronized for image capture) may also be included for capturing visual information from different perspectives).
Martinetz, Kord, McGibbon, MALHOTRA, and Sullivan are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, McGibbon, MALHOTRA, and Sullivan, as a whole, would have rendered obvious the invention recited in claim 8 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with wherein the camera is an internet protocol (IP) camera (as taught by Sullivan, Abstract, Par. [0005-28]) to allow motion data to be generated based on tracking and recording the movement of real objects, to increase accuracy for determining poses (e.g., position, orientation) of an object, to determine the position of a representation of the first feature in a virtual representation of the object from combined visual information, and to improve the accuracy of visual information (Sullivan, Abstract, Par. [0001-9, 17-28]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Martinetz, in view of Kord, in further view of McGibbon, in further view of MALHOTRA, as applied to claim 1 above, and in further view of Homsi et al. (US PG Publication No. 2012/0130514 A1), hereafter referred to Homsi.
Regarding claim 10, claim 1 incorporated and the combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method (Martinetz, Par. [0001-5]), but fails to teach the following as further recited in claim 10.
However, Homsi teaches wherein the distance between the two distance markers is a predetermined distance of 20 metres [meters] (Par. [0043-140]: methods of rating the performance of an athlete… different tests may be administered to determine an athlete's athleticism… the athletic performance tests may include… measuring sprint time of the athlete over a 20-meter distance, and Yo Yo Intermittent Recovery Test (YIRT)… The 20-meter dash is described above… In the Yo Yo Intermittent Recovery Test (YIRT) measures the "start-stop-recover-start" nature of soccer. With reference to FIG. 25, the athletes starts at a starting line 250 located between a pair of cones 252A and 252B, and completes pairs of 20-meter sprints to an intermediate line 254 positioned between a pair of cones 256A and 256B, at a distance of 20-meters from the starting line 250, until failure of the athlete. From a recorded CD, a first beep initiates the first 20-meter sprint, the second beep ends the first 20-meter sprint and initiates the second 20-meter sprint).
Martinetz, Kord, McGibbon, MALHOTRA, and Homsi are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, McGibbon, MALHOTRA, and Homsi, as a whole, would have rendered obvious the invention recited in claim 10 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with wherein the distance between the two distance markers is a predetermined distance of 20 meters (as taught by Homsi, Abstract, Par. [0043-140], Fig. 25) to evaluate and rate the performance of an athlete, to measure a different athletic skill that is needed to compete effectively in a defined sport, to map the range of performance and establish percentile rankings and thresholds for each test value observed during testing of the athletes, to yield an overall athleticism rating, and to assess the athletic ability and/or performance of a given athlete and measures an athletic performance skill related to a particular sport or physical activity (Homsi, Abstract, Par. [0003-12, 43-45, 63-64, 107, 139-140]).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Martinetz, in view of Kord, in further view of McGibbon, in further view of MALHOTRA, as applied to claim 1 above, and in further view of SHAVIT et al. (US PG Publication No. S 2017/0368413 A1), hereafter referred to SHAVIT.
Regarding claim 16, Martinetz discloses a method (Par. [0001]: method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, i.e. estimating the pose of the movable object… implementing the invention in software and in this context to a computer-readable medium that stores commands the execution of which causes the method according to the invention to be carried out) for providing a visual comparison of a first model and a second model (Par. [0019-20]: using skeleton models… Anatomically motivated skeleton models are additionally suited for falling back on stored movement patterns (templates) for detecting fast and complex movements; Par. [0049-65]: FIG. 1: shows sketches of usable skeleton models… an anatomically motivated skeleton model is now used… FIG. 4: shows exemplary images having 3D point clouds and skeleton models fitted into these point clouds… that is seen as a comparison in FIG. 1 a). The model from FIG. 1 b) is particularly suited that unambiguously associates each node with a distinctive point of human anatomy (e.g. head, shoulder, elbow, hand, hip, pelvis, knee, foot). FIG. 1 c) represents a variant of the model from FIG. 1 b), where the torso is represented by triangles (in each case defined by three nodes that form the corners)… By identifying the nodes with human body parts, also the edges of the models are given an anatomic interpretation… the anatomical model is advantageous in particular for modelling objects (that for example correspond to body regions); method for providing a visual comparison of a first model and a second model (e.g. method for pose estimation of a moving object (i.e. a subject), such as a person, for example, includes three dimensional (3D) point clouds that are determined (i.e. generated, calculated, computed, estimated, etc.) by computer calculation of displacements (i.e. movements) of 3D position coordinates of nodes of a skeleton model (i.e. a model) that is modelled according to an anatomy of an object observed, including 3D point clouds that represent moving persons and anatomically motivated skeleton models (i.e. a first, second… Nth model) that are continuously fitted into these point clouds, for example, and providing a visual comparison of a first model and a second model, as shown in Fig. 1 below:
PNG
media_image4.png
490
510
media_image4.png
Greyscale
, for example), the method including the steps of:
generating the first and second models, wherein each of the first and second models is generated, respectively (Par. [0019-26]: using skeleton models… Anatomically motivated skeleton models are additionally suited for falling back on stored movement patterns (templates) for detecting fast and complex movements… pose estimation by means of SOM training is proposed that is able to work with a skeleton model that is modelled according to the anatomy of the object observed… the anatomy of the object whose pose is to be detected is modelled as a skeleton model… A skeleton model that is described only by a small number of points ("nodes" below) in the 3D space represents a good information compression of the image information if the coordinates of the nodes at all times describe the position of predetermined parts of the moving object… a method for pose estimation of a moving object (e.g. a person or a robot) by computer calculation of displacements of 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds. The node coordinates are present in table form in an electronic memory and the 3D point clouds are determined from electronically recorded images from a depth sensor camera that represent the moving person; Par. [0116-120]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person, using image segmentation that is known per se… A depth sensor camera that comprises an arithmetic unit for determining the distance coordinate can in particular also be engineered directly to carry out the inventive method. Such a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts; generating the first and second models (e.g. method for pose estimation of a moving object (i.e. a subject), such as a person, for example, includes three dimensional (3D) point clouds that are determined (i.e. generated, calculated, computed, estimated, etc.) by computer calculation of displacements (i.e. movements) of 3D position coordinates of nodes of a skeleton model (i.e. a model) that is modelled according to an anatomy of an object observed, including 3D point clouds that represent moving persons and anatomically motivated skeleton models (i.e. a first, second… Nth model) that are continuously fitted into these point clouds (i.e. generating the first and second models), as indicated above), for example), wherein each of the first and second models is generated, respectively, by;
capturing, by a single motion capture device, visual data of the subject as it moves in a field of vision of the motion capture device (Par. [0001-5]: method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, i.e. estimating the pose of the movable object… If the motion of the person is imaged by an image sequence, the apparatus mentioned for example provides the 3D coordinates of predetermined body parts that change over time--during the motion; Par. [0024-53]: the anatomy of the object whose pose is to be detected is modelled as a skeleton model… A skeleton model that is described only by a small number of points ("nodes" below) in the 3D space represents a good information compression of the image information if the coordinates of the nodes at all times describe the position of predetermined parts of the moving object… The skeleton model is to be fitted fast and precisely into the image information that corresponds to the object. The fitting is effected between two images of an image sequence by continuously displacing the nodes and updating the skeleton model in step with the image sequence… a method for pose estimation of a moving object (e.g. a person or a robot) by computer calculation of displacements of 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds. The node coordinates are present in table form in an electronic memory and the 3D point clouds are determined from electronically recorded images from a depth sensor camera that represent the moving person… an apparatus for pose estimation of a moving object. This apparatus comprises a depth sensor camera, an electronic memory… the memory storing the electronic images of the depth sensor camera… determine from the electronic images 3D point clouds representing the object, temporally in step with the image recording by the camera. The memory further stores a list of 3D coordinates for the nodes of a skeleton model… A skeleton model having only a low node count serves to effectively compress the image information when the coordinates of the nodes at any time describe the position of predetermined parts of the moving object; Par. [0116-120]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person, using image segmentation that is known per se… A depth sensor camera that comprises an arithmetic unit for determining the distance coordinate can in particular also be engineered directly to carry out the inventive method. Such a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts; capturing, by a single motion capture device, visual data of the subject as it moves in a field of vision of the motion capture device (e.g. method for pose estimation of a moving object (i.e. subject), such as a person, for example, includes three dimensional (3D) point clouds that are determined from electronically recorded (i.e. captured, acquired, etc.) images (i.e. capturing visual data of the subject) by a depth sensor camera (i.e. by a single motion capture device), that represent a person (i.e. visual data of the subject as it moves in a field of vision of the motion capture device), for example, to provide 3D coordinates of predetermined body parts of the person that change over time during motion (i.e. capturing, by a single motion capture device, visual data of the subject as it moves in a field of vision of the motion capture device), as indicated above), for example);
from the captured visual data, recognising [recognizing] human pose points on the subject to extract data of the subject (Par. [0001]: method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, i.e. estimating the pose of the movable object; Par. [0024-26]: pose estimation by means of SOM training is proposed that is able to work with a skeleton model that is modelled according to the anatomy of the object observed and exhibits a reduced node count, it being possible to reliably and consistently assign each model node to a predetermined body part. Here the anatomy of the object whose pose is to be detected is modelled as a skeleton model… embodiment of the invention refers to a method for pose estimation of a moving object (e.g. a person or a robot) by computer calculation of displacements of 3D position coordinates of the nodes of a skeleton model, that is continuously fitted into a sequence of 3D point clouds; Par. [0068-73]: the learning rules are formed according to the following concept. Starting from the existence of an image sequence for which continuous pose estimation is to be carried out, the skeleton model that is a list of node positions (described in a 3D coordinate system), node pairs, and optionally node triplets, is in each case updated when a new image of the sequence exists. As soon as the new image is recorded by the depth sensor camera and has been converted by means of image segmentation and projection into a list of 3D coordinates (3D point cloud) for the points of the surface of the object observed--e.g. the entire person, his torso, his hand etc.… determination of the closest topology element is explained in more detail below, and the specific displacement rules for the topology elements are named… At first the crossing points of X in terms of all topology elements are at first determined for a point X of the 3D point cloud--that is to say for a coordinate point in the 3D space that, after imaging of the object observed using a depth sensor camera and subsequent image segmenting and projection, represents a point of the body surface of the person; Par. [0116-120]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person, using image segmentation that is known per se. The anatomic skeleton model is fitted in real time using the learning rules described above, and the association of the model nodes with the different body parts of the person remains correct throughout as can be clearly gathered from FIG. 4… A depth sensor camera that comprises an arithmetic unit for determining the distance coordinate can in particular also be engineered directly to carry out the inventive method. Such a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts; from the captured visual data, recognizing human pose points on the subject to extract data of the subject (e.g. method for pose estimation of a moving object (i.e. subject), such as a person (i.e. human), for example, includes three dimensional (3D) point clouds that are determined from electronically recorded (i.e. captured, acquired, etc.) images (i.e. from the captured visual data), including analysis of a sequence of electronic images for estimating positions and orientations of a movable object captured in the image sequence, for example, to estimate (i.e. recognize, identify, determine, etc.) a pose of the movable object (i.e. from the captured visual data, recognizing human pose points on the subject), for example, including images recorded by the depth sensor camera that have been converted by image segmentation (i.e. to extract data of the subject) and projected into a list of 3D coordinates (3D point cloud) for points of a surface of an object observed (i.e. from the captured visual data, recognizing human pose points on the subject to extract data of the subject), for example, including coordinate points in the 3D space that, after imaging of the object observed using the depth sensor camera and subsequent image segmenting and projection, represent points of the body surface of a person, as indicated above), for example); and
based on the extracted data, recognising [recognizing] a plurality of predefined anatomical components of the subject to construct the model of the subject (Par. [0056-57]: an anatomically motivated skeleton model is now used… The model from FIG. 1 b) is particularly suited that unambiguously associates each node with a distinctive point of human anatomy (e.g. head, shoulder, elbow, hand, hip, pelvis, knee, foot). FIG. 1 c) represents a variant of the model from FIG. 1 b), where the torso is represented by triangles (in each case defined by three nodes that form the corners); Par. [0068-73]: the learning rules are formed according to the following concept. Starting from the existence of an image sequence for which continuous pose estimation is to be carried out, the skeleton model that is a list of node positions (described in a 3D coordinate system), node pairs, and optionally node triplets, is in each case updated when a new image of the sequence exists. As soon as the new image is recorded by the depth sensor camera and has been converted by means of image segmentation and projection into a list of 3D coordinates (3D point cloud) for the points of the surface of the object observed--e.g. the entire person, his torso, his hand etc.… determination of the closest topology element is explained in more detail below, and the specific displacement rules for the topology elements are named… At first the crossing points of X in terms of all topology elements are at first determined for a point X of the 3D point cloud--that is to say for a coordinate point in the 3D space that, after imaging of the object observed using a depth sensor camera and subsequent image segmenting and projection, represents a point of the body surface of the person; Par. [0116-120]: exemplary embodiment for the inventive pose estimation method with the skeleton model from FIG. 1 b) is presented in FIG. 4 using exemplary images from video sequences. The video images of a depth sensor camera constantly provide 3D point clouds that represent a moving person, using image segmentation that is known per se. The anatomic skeleton model is fitted in real time using the learning rules described above, and the association of the model nodes with the different body parts of the person remains correct throughout as can be clearly gathered from FIG. 4… A depth sensor camera that comprises an arithmetic unit for determining the distance coordinate can in particular also be engineered directly to carry out the inventive method. Such a camera as a constructional unit having at least one arithmetic unit designed according to the invention is correspondingly suited to directly translate the image of a moving person into 3D coordinates of his essential body parts; and based on the extracted data, recognizing a plurality of predefined anatomical components of the subject to construct the model of the subject (e.g. method for pose estimation of a moving object (i.e. subject), such as a person (i.e. human), for example, includes three dimensional (3D) point clouds that are determined (i.e. generated, calculated, computed, estimated, etc.) from electronically recorded images (i.e. captured visual data), including images recorded by a depth sensor camera that have been converted by image segmentation (i.e. based on the extracted data) and projected into a list of 3D coordinates (3D point cloud) for points of a surface of an object observed (i.e. recognized, analyzed, etc.), for example, by computer calculation of displacements (i.e. movements) of 3D position coordinates of nodes of a skeleton model (i.e. the model of the subject) that is modelled according to an anatomy of the object observed, including 3D point clouds that represent a moving person (i.e. recognizing a plurality of components of the subject to construct the model of the subject), for example, including coordinate points in 3D space that, after imaging of the object observed using the depth sensor camera and subsequent image segmenting and projection, represent points of the body surface of a person, for example, by directly translating images of a moving person into 3D coordinates of essential body parts and by assigning each model node to a predetermined body part (i.e. based on the extracted data, recognizing a plurality of predefined anatomical components of the subject to construct the model of the subject), for example, in order to provide 3D coordinates of predetermined body parts of a person that change over time during motion, as indicated above), for example), but fails to teach the following as further recited in claim 16.
However, Kord teaches a biomechanical model of a subject (Par. [0002]: motion capture system records the movements of a motion capture subject in a sequence of digital images, converting the recorded images to a mathematical model… The mathematical model may represent a biomechanical skeleton having rigid links connected to one another by rotatable joints; Par. [0013-22]: FIG. 2 is a pictorial view toward the front of a person acting as a motion capture subject, with the subject standing in an example of a scale frame, the scale frame positioned relative to a camera as in the example of FIG. 1, the subject's right hand at the left side of the figure, and an example of a biomechanical skeleton model overlaid on the motion capture subject showing examples of correspondence between links and joints in the biomechanical skeleton with the person's joints, limbs, head, neck, and torso… A single digital camera records images of a motion capture subject. The images are analyzed to assign values to parameters for a biomechanical skeleton capable of accurately emulating the motion capture subject's movements and body positions; a biomechanical model of a subject (e.g. motion capture system records movements of a motion capture subject (i.e. a subject) in a sequence of digital images, converts the recorded images of the subject to a mathematical model representing a biomechanical skeleton (i.e. a biomechanical model of a subject) having rigid links connected to one another by rotatable joints, such as motion capture subject 148 shown in Fig. 2 below:
PNG
media_image1.png
734
564
media_image1.png
Greyscale
, for example);
capturing, by a single supported motion capture device from a capture position, visual data of the subject (Par. [0009-27]: example of an apparatus embodiment includes a digital camera, a central processing unit in data communication with the digital camera, a memory in data communication with the central processing unit, and an image calibration tool… method embodiment includes the capturing a sequence of digital images of a motion capture subject… FIG. 1 shows a pictorial view of an example of an apparatus for creating accurately calibrated motion capture images from a single stationary digital camera… FIG. 2 is a pictorial view toward the front of a person acting as a motion capture subject, with the subject standing in an example of a scale frame, the scale frame positioned relative to a camera as in the example of FIG. 1, the subject's right hand at the left side of the figure, and an example of a biomechanical skeleton model overlaid on the motion capture subject showing examples of correspondence between links and joints in the biomechanical skeleton with the person's joints, limbs, head, neck, and torso… A single digital camera records images of a motion capture subject… the single camera in an embodiment may remain entirely stationary in a fixed position for capturing images used for determining accurate positions, angles, and distances along all three mutually perpendicular spatial directions. Captured images may quickly and easily be recalibrated should a camera be located to a new viewing location, for example to record a motion capture subject from a new viewing angle… Turning to FIG. 1, an example of an apparatus embodiment 100 includes a camera 114… The camera may optionally be mounted on an adjustable-height tripod 116 or another stable camera support; capturing, by a single supported motion capture device from a capture position, visual data of the subject (e.g. motion capture system that records (i.e. captures, acquires, etc.) movements of a motion capture subject in a sequence of digital images (i.e. capturing visual data of the subject) includes capturing a sequence of digital images of the motion capture subject, including a person (i.e. a human) acting as the motion capture subject No. 148 shown in Fig. 2 below:
PNG
media_image1.png
734
564
media_image1.png
Greyscale
, for example, by using a single digital camera (Fig. 1, No. 114) that remains entirely stationary in a fixed position for capturing (i.e. a capture position) images of the motion capture subject (i.e. capturing, by a single supported motion capture device from a capture position, visual data of the subject), as shown in Fig. 1 below:
PNG
media_image2.png
536
702
media_image2.png
Greyscale
, for example) as it moves between a first distance marker and a second distance marker, the first and second distance markers being disposed at a distance apart from each other and in a field of vision of the motion capture device (Par. [0009-11]: apparatus embodiment of the invention further includes a camera and a computer implemented in hardware, wherein the camera is in data communication with the computer and the computer is adapted to receive an image from the camera, convert the image to a silhouette, and extract parameters for an actor file from the image… method embodiment includes the capturing a sequence of digital images of a motion capture subject, each of the sequence of digital images including an image of an image calibration tool… comparing a separation distance between two calibration markers on the image calibration tool to a corresponding distance between the same two calibration markers a digital image, and overlaying a biomechanical skeleton over each of the sequence of digital images, with a biomechanical reference location superimposed over a movable joint of the motion capture subject and at least one link rotatably coupled to the biomechanical reference location; Par. [0024-33]: image calibration tool enables accurate determination of angles, positions, and lengths in captured images without the use of multiple cameras or multiple camera positions for a single camera… A mathematical model in accord with an embodiment may be referred to as an actor file. An actor file represents a motion capture subject, for example a person engaged in a sports activity or an actor in a video game or motion picture, as an articulated biomechanical skeleton comprising rigid links joined to one another at biomechanical reference locations… image calibration tool 102 optionally includes at least two calibration markers 106… The image calibration tool 102 in the examples of FIGS. 1-2 comprises 12 struts and eight calibration markers 106, including an upper right front ball 132, an upper left front ball 134, a lower right front ball 136, and a lower left front ball 138, where left and right have been labeled with respect a viewing direction along the optical axis 128 toward the image calibration tool 102. Continuing on the back side of the image calibration 102, an upper right back ball 140, an upper left back ball 142, a lower right back ball 144, and a lower left back ball 146 are joined to one another and to the front balls by struts… A digital image captured by the digital camera 114 may be processed by a CPU included in some embodiments to extract parameters for an actor file… An image of a motion capture subject 148 may optionally be processed by the computer (ref. FIG. 1) to form a silhouette 150 corresponding to an outline of the person 148 from the camera's viewing direction. An example of a silhouette is shown in FIG. 3. As the motion capture subject moves, the camera collects images, for example a sequence of video images recorded at 30 frames per second or a sequence of still images recorded at selected positions of the subject 148; Par. [0042-51]: method… includes the steps of capturing a sequence of digital images of a motion capture subject, each of the sequence of digital images including an image of an image calibration tool, determining a separate scale factor for each of three mutually orthogonal spatial directions by comparing a separation distance between two calibration markers on the image calibration tool to a corresponding distance between the same two calibration markers in each of said sequence of digital images, and overlaying a biomechanical skeleton over each of the sequence of digital images with a biomechanical reference location superimposed over a movable joint of the motion capture subject and at least one link rotatably coupled to the biomechanical reference location… positioning the camera 114 along the y-axis at a distance 118 selected to fit the motion capture subject 148 into the camera's field of view; as it moves between a first distance marker and a second distance marker, the first and second distance markers being disposed at a distance apart from each other and in a field of vision of the motion capture device (e.g. motion capture system that records (i.e. captures, acquires, etc.) movements of a motion capture subject in a sequence of digital images (i.e. capturing visual data of the subject) includes capturing a sequence of digital images of the motion capture subject, including a person (i.e. a human) acting as the motion capture subject No. 148 shown in Fig. 2 below:
PNG
media_image1.png
734
564
media_image1.png
Greyscale
, for example, by using a single digital camera (Fig. 1, No. 114) that remains entirely stationary in a fixed position for capturing (i.e. a capture position) images of the motion capture subject (i.e. capturing, by a single supported motion capture device from a capture position, visual data of the subject), as shown in Fig. 1 below:
PNG
media_image2.png
536
702
media_image2.png
Greyscale
, for example, in which each of the sequence of digital images includes of a calibration tool captured by using the single digital camera, for example, including two calibration markers (i.e. a first distance marker and a second distance marker) separated by a distance measured between the first and second calibration markers and a corresponding distance (i.e. the first and second distance markers being disposed at a distance apart from each other) in the captured images within the camera's field of view (i.e. and in a field of vision of the motion capture device), for example, and as the motion capture subject moves (i.e. as it moves in a field of vision of the motion capture device), the camera collects images of the motion capture subject to perform measurements of the motion capture subject (i.e. capturing, by a single supported motion capture device from a capture position, visual data of the subject as it moves between a first distance marker and a second distance marker, the first and second distance markers being disposed at a distance apart from each other and in a field of vision of the motion capture device), as indicated above), for example).
Martinetz and Kord are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with a biomechanical model of a subject and capturing, by a single supported motion capture device from a capture position, visual data of the subject as it moves in a field of vision of the motion capture device (as taught by Kord, Abstract, Par. [0009-51]) to create an articulated, movable mathematical model of a person from position, size, and angle information extracted from digital camera images, to accurately emulate a motion capture subject's movements and body positions, to create accurately calibrated motion capture images from a single stationary digital camera, and to enable accurate determination of angles, positions, and lengths in captured images (Kord, Abstract, Par. [0001-51]).
The combination of Martinetz and Kord, as a whole, teaches the method, as indicated above, but fails to teach the following as further recited in claim 16.
However, McGibbon teaches kinematic data (Par. [0001-7]: system and method for analyzing kinetic and kinematic information of human motion, and for viewing the information… present invention addresses these drawbacks by providing a full four-dimensional analysis (three space dimensions, one time dimension) of human movement data captured by a motion analysis system. The invention enables detailed biomechanical analysis of human movement data, as well as the visualization of data… a system for displaying kinematic and kinetic information of a subject is provided. The system includes an image input stage for acquiring image data of the subject, a transformation stage for transforming the image data into three dimensional coordinates corresponding to one or more body segments of the subject, and an output data stage for calculating the kinematic and kinetic information of the subject from the three dimensional coordinates… method comprises the steps of acquiring image data of the subject, transforming the image data into three dimensional coordinates corresponding to one or more body segments of the subject, and calculating the kinematic and kinetic information of the subject from the three dimensional coordinates; Par. [0020-28]: system and method, and a software facility, for the analysis of kinematics and kinetics of human movement. The present invention utilizes an eleven segment three-dimensional model of human movement analysis… FIG. 1 is a schematic block diagram of a movement analysis system… present invention relies on the acquisition of image data to provide an accurate estimation of movement. The image input stage 2 is utilized for acquiring, obtaining or receiving image data needed for the movement analysis system. The image input stage 2 can be any device or structure suitable for receiving, obtaining or acquiring image data… The image data acquired by the image input stage is converged to the transformation stage 4 which utilizes the acquired image data to track and build a 3-D model of the human body. In particular, the transformation stage 4 performs the coordinate transformation needed to calculate the various kinetics and kinematics discussed in more detail below. The output data stage 6 generates output containing an array of information used in modeling human movement. In particular, the output data stage 6 provides output analysis for the various kinematic and kinetic parameters… FIG. 7 is a schematic block diagram illustration of the output data stage 6 of FIG. 1. The output data stage 6 generates numerous output files containing a variety of useful biomechanical measures. In general, the output data stage 6 provides the kinematic output information and kinetic output information. The illustrated kinematic analysis module 64 provides for kinematic analysis on all of the eleven segmented body parts mentioned above. In particular, the kinematic analysis module 64 provides for a greater understanding of how the body of the subject 30 move relative to one another (coordination), as well as the rates at which they move (velocities). Thus, the kinematic analysis module 64 includes analysis information regarding the subject's bodily motions; kinematic data (e.g. system and method for analyzing kinetic and kinematic information (i.e. kinematic data) of human motion and for providing detailed biomechanical analysis of human movement data includes acquiring image data of a subject, transforming the image data into three dimensional coordinates (i.e. 3D point cloud) corresponding to one or more body segments of the subject (i.e. predefined anatomical components of the subject), for example, and calculating (i.e. extracting, computing, etc.) the kinematic information of the subject from the three dimensional coordinates, as indicated above), for example).
Martinetz, Kord, and McGibbon are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, and McGibbon, as a whole, would have rendered obvious the invention recited in claim 16 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with kinematic data (as taught by McGibbon, Abstract, Par. [0001-7, 20-28]) to provide a software facility for computing and displaying kinematic and kinetic information of a subject to a user, to analyze various human movements, to provide output analysis for various kinematic and kinetic parameters, thus allowing a more detail output of various modeled segments acquired, to provide an accurate estimation of movement, and to analyze aspects of motion of a body and body segments (McGibbon, Abstract, Par. [0001-7, 20-29]).
The combination of Martinetz, Kord, and McGibbon, as a whole, teaches the method, as indicated above, but fails to teach the object “moves from a first distance marker to a second distance marker”, as now recited in claim 16.
However, MALHOTRA teaches moves from a first distance marker to a second distance marker (Par. [0006-11]: system and methods configured to improve the athlete's ability to synchronize the movement of their body in time… an image capturing device may capture images at different or variable frame rates… determinations of movement (and/or movement quality) may utilize rate of capture information… landmarks/distance calibrations may be utilized from time-stamped image data to allow for precise measuring of performance. For example, objects represented by image data may be utilized to determine whether movement thresholds are met. For example, markings on a field (such as yard lines) may be used to calibrate distance measurements… image data may indicate that the athlete is performing a second or additional features of a first athletic movement; Par. [0029]: determining athletic attributes of an athlete from image data… an athlete's performance of a physical activity, such as for example, a sprint or agility drill, or battery of field-based tests, may be analyzed according to image data… landmarks/distance calibrations may be utilized from time-stamped image data to allow for precise measuring of performance (including, but not limited to: sprint or agility times, flight time for vertical jump, distance for throws). Data retrieved or derived from the image data may be used in scoring and/or ranking athletes; Par. [0070-77]: processing image data, such as the image data captured… a location of a body portion of the athlete, or a location of a sporting device within the image data (absolute location or relative to another object, such as the athlete any other object represented by image data) may be used to detect the performance of one or more features of specific movements… landmarks/distance calibrations may be utilized from time-stamped image data to allow for precise measuring of performance. For example, objects represented by image data may be utilized to determine whether movement thresholds are met (e.g., block 804b). For example, markings on a field (such as yard lines) may be used to calibrate distance measurements… dimensions may be used to identify thresholds and/or determine whether certain thresholds have been met, including but not limited to: flight time for vertical jump, distance for throws, kick distance and power, among others, sprint times between two distances; Par. [0087-88]: identifying image data (such as but not limited to specific images) that correlate to a specific physical movement or activity of the athlete (e.g., block 810). For example, certain embodiments may detect initiation of the athletic activity, performance of an athletic movement, and/or the occurrence of a movement feature. As non-limiting examples, image data may be used to identify one or more actions, including: initiation of the activity, levels of acceleration, velocity, reaction, and/or tempo, distance traveled by an object, completion of the activity, among others… identify image data (including specific frames or images) such as including, but not limited to: an initiation image, a termination image, or any other image comprising motion data that can be identified based upon the systems or methods disclosed herein; Par. [0118-120]: 9A and 9B show example activity spaces comprising illustrative test elements. With reference to FIG. 9A, activity space 900 illustrates example test elements. Activity space 900 is associated with an Arrowhead Agility performance test, which test may be various configured with respect to test elements. As to test elements, activity space 900 may include area 902, equipment 906A-F, and a start-finish line 908… Within area 902, the equipment 906A-F may be positioned at prescribed locations… The equipment 906A-F may indicate to the athlete the start-stop line 908… the equipment 906A-F includes markers, such as cones, arranged in a predetermined formation: (i) four cones 906A, B, C, E are positioned to form a square with ten-meter sides… and (iii) cone 906D is positioned on a line perpendicular to the line formed by cones 906C, 906F and 906E, at a distance, e.g., five meters, from 906F, distally from cones 906A, B… an activity may contemplate various constituents, and each constituent may contemplate various components performed in an activity space… constituents or components may include one or more of the following… athlete's time from the start-stop line 908 to initiation of rounding of cone 906F… athlete's time from cone 906F to cone 906E… athlete's time from cone 906E to cone 906D… athlete's time from cone 906D to start-stop line 908; moves from a first distance marker to a second distance marker (e.g. image capturing device captures images, including image data captured of an object (i.e. subject), such as an athlete or any other object represented by image data, for example, to perform measurements by using landmarks/distance calibrations utilized from time-stamped image data to allow for precise measuring of performance, including markers, such as cones, for example, arranged in a predetermined formation, including cones placed (i.e. disposed, located, etc.) at a distance from other cones, such as five meters (i.e. distance markers being disposed at a distance apart from each other and in a field of vision of the motion capture device), for example, and identifying image data that correlates to a specific physical movement or activity of the object in a field of vision of the motion capture device, including a distance traveled by an object, sprint times between two distances, etc., for example, including determining an object’s (i.e. athlete's) movement time from an initial cone (i.e. first distance marker), such as cone 906A, for example, to another cone (i.e. second, third… Nth distance marker), such as cones 906C-F, for example, as the object moves from one cone to another (i.e. moves from a first distance marker to a second distance marker), as shown in Fig. 9A below:
PNG
media_image3.png
744
914
media_image3.png
Greyscale
, for example, and detecting the performance of one or more features of specific movements of the object in order to determine athletic attributes the object (i.e. an athlete) from image data, as indicated above), for example).
Martinetz, Kord, McGibbon, and MALHOTRA are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, would have rendered obvious the invention recited in claim 16 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with moves from a first distance marker to a second distance marker (as taught by MALHOTRA, Abstract, Par. [0006-11, 29, 70-77, 87-88, 118-120]) to improve a feedback system that allows an athlete (or another individual, such as a trainer) to improve performance for a particular sport or athletic activity, to provide feedback to an athlete regarding the athlete's performance of a physical activity, to comprehend and optimize the timing of one or more components or features of an athletic movement, and to determine athletic attributes the object from image data (MALHOTRA, Abstract, Par. [0002-11, 29, 70-77, 87-88, 118-120]).
The combination of Martinetz, Kord, McGibbon, and MALHOTRA, as a whole, teaches the method, as indicated above, but fails to teach the object “moves from a first distance marker to a second distance marker”, as now recited in claim 16.
However, SHAVIT teaches identifying corresponding central points of reference on each of the first and second biomechanical models (Par. [0002]: systems and techniques for planning, presenting, monitoring, feed-backing and improving training in various disciplines; Par. [0052-53]: FIG. 1 shows an exemplary and non-limiting visual representation of a system 100 utilized to describe the various embodiments disclosed herein… generating sensory information that may include the position and movements of a user performing the exercise routine, and the like. The sensors may be a camera or an optical sensor that can produce 2D or 3D still or video images. The sensors may further include acoustical LIDAR or RADAR sensors that can produce 2D or 3D still or moving images or mapping; Par. [0344-382]: examples for Body Models are given in the references, for example US20100303289A1 by Polzin et al. incorporated here by reference, body model 406 of FIG. 12 of that disclosure. Body model 406 is exemplary in that a body model may contain any number of body parts, each of which may be any machine-understandable representation of the corresponding part of the modeled target… Machine Representation model or Body model can be a collection of points in 3D or 2D space and/or a collection of 3D and/or 2D shapes… The evolution in time from step to step or frame to frame can be viewed as movement in space of the collection of shape and/or points. The shape may be complex and may, for example, represent a part of the target body. For example, if the target is a person, the shape maybe a scaled model of a body part in machine representation… If a location of a certain point or a collection of points have been identified on each one of the different sensor 2D/3D images, they can be adjoined in 3D space… images 2719 and 2723 can then be projected in space, using the data from the common points 2742… This process can be generalized to any number of sensors, or to a group or collection of common points between the images… it is possible not to generate possibilities for new points, but only use the collection of known 3D points attained for example from identifying common points between the sensors… Existing 3D points that become occluded or not possible to obtain their 3D location from the sensors, can be interpolated or extrapolated using motion estimation methods known in the art. Some of these motion estimation methods may give more than one possibility for the new location of such points. They may also give probability estimation for every such point. Based on these possibilities a plurality of possible 3D models of the objects may be possible. All the possible models may be saved in memory or only those with a function of the probabilities higher than a certain threshold. Thus, the 3D mapping is evolving and improving with time; identifying corresponding central points of reference on each of the first and second biomechanical models (e.g. systems and techniques for planning, presenting, monitoring, feed-backing and improving training in various disciplines include body models (i.e. each of the first and second biomechanical models) contain any number of body parts, each of which may be any machine-understandable representation of the corresponding part of the modeled target, for example, including a collection of points in 3D or 2D space and/or a collection of 3D and/or 2D shapes generated from different sensor 2D/3D images, for example, and if a location of a certain point or a collection of points have been identified on each one of the different sensor 2D/3D images (i.e. identifying corresponding central points of reference on each of the first and second biomechanical models), they are adjoined in 3D space by using a collection of known 3D points attained from identifying common points between the sensors, as indicated above), for example);
scaling each of the first and second biomechanical models such that relative heights of the first and second biomechanical models are identical (Par. [0344-373]: examples for Body Models are given in the references, for example US20100303289A1 by Polzin et al. incorporated here by reference, body model 406 of FIG. 12 of that disclosure. Body model 406 is exemplary in that a body model may contain any number of body parts, each of which may be any machine-understandable representation of the corresponding part of the modeled target… Machine Representation model or Body model can be a collection of points in 3D or 2D space and/or a collection of 3D and/or 2D shapes… The evolution in time from step to step or frame to frame can be viewed as movement in space of the collection of shape and/or points. The shape may be complex and may, for example, represent a part of the target body. For example, if the target is a person, the shape maybe a scaled model of a body part in machine representation… If a location of a certain point or a collection of points have been identified on each one of the different sensor 2D/3D images, they can be adjoined in 3D space in the following way: for example, the images 2719 and 2723 seen from the sensors 145 and 140 can be adjoined: assuming 2742 is the up most (with the largest Z coordinate), left most (lowest Y coordinate) point on the object. And assuming its location is known on both images 2719 and 2723. If the location of 2742 is known in space or relative to the coordinate system 2730, and the location of the sensors 145 and 140 is known, it is then possible using known methods to scale the images 2719 and 2723. The scaling purpose is to give the images 2719 and 2723 their actual size in space at point 2742… once a point is identified as common between two sensors or more, its location in space can be found by triangulation (using the two or more sensors location) or other methods known in the art. The vice versa direction is also possible if the exact location in space of a point visible in the two sensors or more is known, it becomes a common point and can match the sensors images; Par. [0398-409]: possibilities of skeletal or body models of the known posture can be projected on the 2D or 3D images from the sensors and correlation can be performed to filter out the non-resembling ones and to keep the most resembling model/models or machine representations… creating the first frame of body model of the objects to track, will be performed where the object/objects are coerced into a known position and/or posture After creating the first frame of body model, the system can continue to track the subject/s. It can continue to track to the next exercises and throughout the training facility or environment… a full and/or detailed 3D model of the subject/s can be attained at a certain previous time and known conditions… This initiated full and or detailed 3D model of the subject/s can be used in later times as a reference model or starting point for creating the first frame of body image and or skeleton mapping and or body mapping and alike, and tracking it as it evolves to the next frames… This model can be used to create possibilities for body mapping of the subject… Before using it, the initiated model can be scaled to fit the current state and location of the trainee. For example, if the user has gained weight or lost weight his model can be modified accordingly. If when the model was attained the user was weighted, he can be weighted again (For example using the gym floor embodiment previously discussed), and the model can be scaled according to the weight proportion. Additional measurements in this case, such as body fat percentage change, can help make the scaling more accurate. Similar scaling can be done as a response to other user attribute changes such as for example height (As a result of child growth for example), body fat percentage… possible skeleton or body maps saved on the database 120 are scaled to the user's dimensions. For example, if the user is 1.90 meters high and the images saved in the database 120 represent a 1.80 meter high model, the model dimensions can be enlarged by the ratio of 1.90 to 1.80. The dimensions of certain body parts or directions in space can be scaled in a different ratio which can be predefined or calculated. As a non-limiting example, the height can be enlarged by the ratio of 1.90 to 1.80 and the width enlarged by this ratio multiplied by a predefined constant. Such constants can be based on known proportions of the human or object body… If the proportions of the user's body are known the possible skeletons or body maps can be scaled accordingly. Alternatively, the required scaling can be attained from an object found in the frame for which the dimensions or relative dimensions are known, for example, a feature of the exercise device which is adjacent to the user's body. The feature's measured dimensions in the frame may be compared to the features known dimensions to enable the training module 110 to calculate the required scaling for the candidate skeleton or body maps; scaling each of the first and second biomechanical models such that relative heights of the first and second biomechanical models are identical (e.g. systems and techniques for planning, presenting, monitoring, feed-backing and improving training in various disciplines include body models (i.e. each of the first and second biomechanical models) contain any number of body parts, each of which may be any machine-understandable representation of the corresponding part of the modeled target, for example, for example, including a collection of points in 3D or 2D space and/or a collection of 3D and/or 2D shapes generated from different sensor 2D/3D images, for example, and if a location of a certain point or a collection of points (i.e. central points of reference) have been identified on each one of the different sensor 2D/3D images (i.e. identifying corresponding central points of reference on each of the first and second biomechanical models), they are adjoined in 3D space by using a collection of known 3D points attained from identifying common (i.e. identical, matching, same, etc.) points between the sensors, for example, and scaling for each body model is scaled to fit a current state and location of a body of a trainee (i.e. scaling each of the first and second biomechanical models), for example, including scaling done as a response to user attribute changes, such as height (i.e. scaling each of the first and second biomechanical models such that relative heights of the first and second biomechanical models are identical), for example, including possible skeleton or body maps scaled to a user's dimensions, such as an initiated model scaled to fit a current state and location of the trainee, as indicated above), for example); and
overlaying the scaled first and second biomechanical models at their respective corresponding central points of reference to provide the visual comparison of the first and second biomechanical models (Par. [0373-409]: once a point is identified as common between two sensors or more, its location in space can be found by triangulation (using the two or more sensors location) or other methods known in the art. The vice versa direction is also possible if the exact location in space of a point visible in the two sensors or more is known, it becomes a common point and can match the sensors images… Possibilities for identifying pixels as common can also emerge from comparison of pixels and their nearest neighbors in one 2D image to pixels and their nearest neighbors in another sensor 2D or 3D image. A grade for comparison can be calculated for each comparison where the grade can be a weighted sum of the differences between the pixels in the matrices of the pixel and its nearest neighbors. Each pixel in the metrics get a grade in the sum for example according to its distance from the center pixel or according to other properties of this pixel. Many methods are known in the art of image processing for making such comparisons… possibilities of skeletal or body models of the known posture can be projected on the 2D or 3D images from the sensors and correlation can be performed to filter out the non-resembling ones and to keep the most resembling model/models or machine representations… creating the first frame of body model of the objects to track, will be performed where the object/objects are coerced into a known position and/or posture After creating the first frame of body model, the system can continue to track the subject/s. It can continue to track to the next exercises and throughout the training facility or environment… a full and/or detailed 3D model of the subject/s can be attained at a certain previous time and known conditions… This initiated full and or detailed 3D model of the subject/s can be used in later times as a reference model or starting point for creating the first frame of body image and or skeleton mapping and or body mapping and alike, and tracking it as it evolves to the next frames… This model can be used to create possibilities for body mapping of the subject… Before using it, the initiated model can be scaled to fit the current state and location of the trainee. For example, if the user has gained weight or lost weight his model can be modified accordingly. If when the model was attained the user was weighted, he can be weighted again (For example using the gym floor embodiment previously discussed), and the model can be scaled according to the weight proportion. Additional measurements in this case, such as body fat percentage change, can help make the scaling more accurate. Similar scaling can be done as a response to other user attribute changes such as for example height (As a result of child growth for example), body fat percentage… possible skeleton or body maps saved on the database 120 are scaled to the user's dimensions. For example, if the user is 1.90 meters high and the images saved in the database 120 represent a 1.80 meter high model, the model dimensions can be enlarged by the ratio of 1.90 to 1.80. The dimensions of certain body parts or directions in space can be scaled in a different ratio which can be predefined or calculated. As a non-limiting example, the height can be enlarged by the ratio of 1.90 to 1.80 and the width enlarged by this ratio multiplied by a predefined constant. Such constants can be based on known proportions of the human or object body… If the proportions of the user's body are known the possible skeletons or body maps can be scaled accordingly. Alternatively, the required scaling can be attained from an object found in the frame for which the dimensions or relative dimensions are known, for example, a feature of the exercise device which is adjacent to the user's body. The feature's measured dimensions in the frame may be compared to the features known dimensions to enable the training module 110 to calculate the required scaling for the candidate skeleton or body maps; Par. [0414-419]: series of such comparisons to different sensors images can leave only the most resembling or correlated reference body models… Comparison/Correlation and alike can be performed to data on the same points, features or shapes and alike from 2D or 3D sensors. Comparison/Correlation and alike can leave only the most resembling or correlated reference body models… the 3D/2D representation/s of the body are compared with the candidate filtered possible skeleton or body maps… data about direction and/or orientation and/or location and/or speed and alike is compared between the database and the sensor/s output… measuring the degree of overlap between the body and the possible skeleton or candidate body map/s, for example, giving a grade to every non overlapping pixel which is dependent upon its distance from the nearest body point (grades can be negative for pixels inside the body). The grade can be the sum of these distances. In this case the best match is the lowest grade; overlaying the scaled first and second biomechanical models at their respective corresponding central points of reference to provide the visual comparison of the first and second biomechanical models (e.g. systems and techniques for planning, presenting, monitoring, feed-backing and improving training in various disciplines include body models (i.e. each of the first and second biomechanical models) contain any number of body parts, each of which may be any machine-understandable representation of the corresponding part of the modeled target, for example, for example, including a collection of points in 3D or 2D space and/or a collection of 3D and/or 2D shapes generated from different sensor 2D/3D images, for example, and if a location of a certain point or a collection of points (i.e. central points of reference) have been identified on each one of the different sensor 2D/3D images (i.e. identifying corresponding central points of reference on each of the first and second biomechanical models), they are adjoined in 3D space by using a collection of known 3D points attained from identifying common (i.e. identical, matching, same, etc.) points between the sensors, for example, and scaling for each body model is scaled to fit a current state and location of a body of a trainee (i.e. the scaled first and second biomechanical models at their respective corresponding central points of reference), for example, including feature's measured dimensions in the frame compared to features known dimensions (i.e. provide the visual comparison of the first and second biomechanical models) used to calculate the required scaling for the candidate skeleton or body maps leave only the most resembling or correlated reference body models, for example, by measuring a degree of overlap (i.e. overlaying) between the body and the possible skeleton or candidate body maps (i.e. overlaying the scaled first and second biomechanical models at their respective corresponding central points of reference to provide the visual comparison of the first and second biomechanical models),as indicated above), for example).
Martinetz, Kord, McGibbon, MALHOTRA, and SHAVIT are considered to be analogous art because they pertain to image processing applications related to generating biomechanical skeletal models of subjects. Therefore, the combined teachings of Martinetz, Kord, McGibbon, MALHOTRA, and SHAVIT, as a whole, would have rendered obvious the invention recited in claim 16 with a reasonable expectation of success in order to modify the method and an apparatus for real time-capable analysis of a sequence of electronic images for estimating the positions and orientations of a movable object captured in the image sequence, to generate a skeleton model that is modelled according to an anatomy of an object observed, including 3D point clouds that represent a moving person (as disclosed by Martinetz) with identifying corresponding central points of reference on each of the first and second biomechanical models; scaling each of the first and second biomechanical models such that relative heights of the first and second biomechanical models are identical; and overlaying the scaled first and second biomechanical models at their respective corresponding central points of reference to provide the visual comparison of the first and second biomechanical models (as taught by SHAVIT, Abstract, Par. [0002, 52-53, 344-382, 399-409, 419-420]) to provide a solution for planning, presenting, monitoring, feed-backing and improving training in various disciplines, to identify and monitor training routines performed by a user, to improve training programs, monitor the training and give feedback, and to provide 3D mapping evolving and improving with time (SHAVIT, Abstract, Par. [0001-10, 52-53, 344-382]).
Allowable Subject Matter
Claims 17 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant’s amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GUILLERMO RIVERA-MARTINEZ whose telephone number is 571-272-4979. The examiner can normally be reached on Monday-Friday (8am - 5pm Eastern Time). If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GUILLERMO M RIVERA-MARTINEZ/ Primary Examiner, Art Unit 2677
