DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
Claim1-20 of this application is patentably indistinct from claim 1-20 of Application No. 18/749,898. Pursuant to 37 CFR 1.78(f), when two or more applications filed by the same applicant or assignee contain patentably indistinct claims, elimination of such claims from all but one application may be required in the absence of good and sufficient reason for their retention during pendency in more than one application. Applicant is required to either cancel the patentably indistinct claims from all but one application or maintain a clear line of demarcation between the applications. See MPEP § 822.
A rejection based on double patenting of the “same invention” type finds its support in the language of 35 U.S.C. 101 which states that “whoever invents or discovers any new and useful process... may obtain a patent therefor...” (Emphasis added). Thus, the term “same invention,” in this context, means an invention drawn to identical subject matter. See Miller v. Eagle Mfg. Co., 151 U.S. 186 (1894); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Ockert, 245 F.2d 467, 114 USPQ 330 (CCPA 1957).
A statutory type (35 U.S.C. 101) double patenting rejection can be overcome by canceling or amending the claims that are directed to the same invention so they are no longer coextensive in scope. The filing of a terminal disclaimer cannot overcome a double patenting rejection based upon 35 U.S.C. 101.
Claim 1-20 provisionally rejected under 35 U.S.C. 101 as claiming the same invention as that of claim 1-20 of copending Application No. 18/749,898 (reference application). This is a provisional statutory double patenting rejection since the claims directed to the same invention have not in fact been patented.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1-12,14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kumar et al (US 20240057892 A1) in view of Fu et al (US 20230126178 A1)
Regarding claim 1, Kumar discloses a computer-implemented method ([0037] method for processing video data to first track body parts of a subject) comprising:
receiving a real image that includes a human, the real image comprising a photograph or a frame of a video ([0041] image capture device 101 may be part of, included in, or connected to another device (e.g., device 1600), and may be a camera, a high speed video camera, or other types of devices capable of capturing images and videos, [0085] subject is a human);
creating a synthetic image corresponding to the real image, the synthetic image including a synthetic environment and a humanoid shape that correspond to the human ([0044] point data 112 may represent data tracking movements of a set of subject body parts over a time period represented in the video data);
predicting, using a trained viewpoint network and based on the synthetic image, a predicted viewpoint heatmap ([0029] heat maps, curves and plots. FIG. 16A is a heat map summarizing the effect sizes and q-values obtained from model M2), the trained viewpoint network comprising a first trained convolutional neural network ([0040] neural network may provide multiple two-dimensional markers (in some embodiments, twelve such markers) of a subject's anatomical location (also referred to as “keypoints”), for each frame of video describing the pose of the subject at each time point.);
predicting, using a trained pose network and based on the synthetic image, a predicted pose heatmap, the trained pose network comprising a second trained convolutional neural network ([0117] performing pose estimation after object detection, full resolution pose keypoint heatmaps were used to infer the posture of a single mouse at every frame);
providing, as input to a random synthetic environment, the predicted viewpoint heatmap and the predicted pose heatmap ([0125] multiple repeated measurements have been taken at two different ages giving rise to a nested hierarchical data structure. The models (M1, M2 M3) follow the standard LMM notation with (Genotype, BodyLength, Speed, TestAge) denoting the fixed effects and (MouseID/TestAge) (test age nested within the animal) denoting the random effect.);
classifying the reconstructed three-dimensional pose as a particular type of pose of the human in the real image ([0152], , SVM is a supervised learning model with associated learning algorithms that analyze data and recognize patterns in the data, and which are commonly used for classification and regression analysis).
Fu discloses creating a reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment (0073] applies a backward loop to reconstruct a previous pose estimation from current frames to improve robustness and minimize inconsistent estimation)
Kumar and Fu are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify method for tracking body parts of subject of Kumar to include creating a reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment as described by Fu.
The motivation for doing so would have been to generate an interpretable low-dimension representation of bodies in images (Fu, [0003]).
Therefore, it would have been obvious to combine Kumar and Fu to obtain the invention as specified in claim 1.
Regarding claim 2, Kumar discloses determining that at least one of the predicted viewpoint heatmap or the predicted pose heatmap specify a Gaussian heatmap that warps around a vertical edge or a horizontal edge of the synthetic image ([0050] output of the neural network of the point tracker component 110 may be compared with the keypoint-centered Gaussian distribution, and the loss may be calculated as the mean squared difference between the respective keypoint and the heatmap generated by the point tracker component)
Regarding claim 3, Kumar discloses determining, that at least one of the predicted viewpoint heatmap or the predicted pose heatmap include more than three dimensions ([0045] point tracker component 110 may generate a first heatmap, where each cell in the heatmap may correspond to a pixel within the video frame, and may represent a likelihood of a first subject body part (e.g., a right forepaw) being located at the respective pixel).
Regarding claim 4, Kumar discloses determining, that at least one of the predicted viewpoint heatmap or the predicted pose heatmap include a time dimension ([0045] point tracker component 110 may generate a first heatmap, where each cell in the heatmap may correspond to a pixel within the video frame, and may represent a likelihood of a first subject body part (e.g., a right forepaw) being located at the respective pixel).
Regarding claim 5, Kumar discloses wherein creating the reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment comprises:
decomposing a human pose of the human into bone vectors and bone lengths that are relative to a parent joint ([0046] set of subject body parts may be pre-defined and may be based on which keypoints are visually salient, such as ears or nose, and/or which keypoints capture important information for analyzing the gait and posture of the subject, such as limb joints or paw); and
transforming a camera position of the synthetic image from subject-centered coordinates to world coordinates ([0046] The point tracker component 110 may be configured to locate two-dimensional coordinates of a set of subject body parts, identified as keypoints).
Regarding claim 6, Kumar discloses wrapping a matrix in a geometric formation including defining an encoding in which a seam line is at a back of the humanoid shape and opposite a forward vector ([0040] neural network may provide multiple two-dimensional markers (in some embodiments, twelve such markers) of a subject's anatomical location (also referred to as “keypoints”), for each frame of video describing the pose of the subject at each time point.).
Regarding claim 7, Kumar discloses wherein creating the reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment comprises:
transforming a camera’s position from subject-centered coordinates to world coordinates ([0046] The point tracker component 110 may be configured to locate two-dimensional coordinates of a set of subject body parts, identified as keypoints).
Regarding claim 8, Kumar discloses a computing device ([0037] method for processing video data to first track body parts of a subject) comprising:
one or more processor ([0156] include one or more controllers/processors (1604/1704), which may each include a central processing unit (CPU) for processing data and computer-readable instructions); and
a non-transitory memory device to store instructions executable by the one or more processors to perform operations ([0156] memories (1606/1706) may individually include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magnetoresistive memory (MRAM), and/or other types of memory) comprising:
receiving a real image that includes a human, the real image comprising a photograph or a frame of a video ([0041] image capture device 101 may be part of, included in, or connected to another device (e.g., device 1600), and may be a camera, a high speed video camera, or other types of devices capable of capturing images and videos, [0085] subject is a human);
creating a synthetic image corresponding to the real image, the synthetic image including a synthetic environment and a humanoid shape that correspond to the human ([0044] point data 112 may represent data tracking movements of a set of subject body parts over a time period represented in the video data);
predicting, using a trained viewpoint network and based on the synthetic image, a predicted viewpoint heatmap, the trained viewpoint network comprising a first trained convolutional neural network ([0029] heat maps, curves and plots. FIG. 16A is a heat map summarizing the effect sizes and q-values obtained from model M2), the trained viewpoint network comprising a first trained convolutional neural network ([0040] neural network may provide multiple two-dimensional markers (in some embodiments, twelve such markers) of a subject's anatomical location (also referred to as “keypoints”), for each frame of video describing the pose of the subject at each time point.);
predicting, using a trained pose network and based on the synthetic image, a predicted pose heatmap, the trained pose network comprising a second trained convolutional neural network ([0117] performing pose estimation after object detection, full resolution pose keypoint heatmaps were used to infer the posture of a single mouse at every frame);
providing, as input to a random synthetic environment, the predicted viewpoint heatmap and the predicted pose heatmap ([0125] multiple repeated measurements have been taken at two different ages giving rise to a nested hierarchical data structure. The models (M1, M2 M3) follow the standard LMM notation with (Genotype, BodyLength, Speed, TestAge) denoting the fixed effects and (MouseID/TestAge) (test age nested within the animal) denoting the random effect.);
classifying the reconstructed three-dimensional pose as a particular type of pose of the human in the real image ([0152], , SVM is a supervised learning model with associated learning algorithms that analyze data and recognize patterns in the data, and which are commonly used for classification and regression analysis).
Fu discloses creating a reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment (0073] applies a backward loop to reconstruct a previous pose estimation from current frames to improve robustness and minimize inconsistent estimation)
Kumar and Fu are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify method for tracking body parts of subject of Kumar to include creating a reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment as described by Fu.
The motivation for doing so would have been to generate an interpretable low-dimension representation of bodies in images (Fu, [0003]).
Therefore, it would have been obvious to combine Kumar and Fu to obtain the invention as specified in claim 8.
Regarding claim 9, Kumar discloses wherein the trained pose network and the trained viewpoint network are created by:
randomly selecting a pose from a set of poses; randomly selecting a viewpoint from a set of viewpoints ([0050] point tracker component 110 may be trained using training video data of subjects having varying physical characteristics, such as, different coat color, different body lengths, different body sizes, etc.);
generating the synthetic environment based at least in part on the pose and the viewpoint ([0050] the point tracker component 110 may be trained using an optimization algorithm, for example, a stochastic gradient descent optimization algorithm); and
deriving, from the synthetic environment, an abstract representation, a viewpoint heatmap, and a pose heatmap, wherein the viewpoint heatmap and the pose heatmap are used as supervised training targets ([0049] , the point tracker component 110 may generate multiple heatmaps, each heatmap representing an inference of where one keypoint representing one subject body part is located within a frame of the video data).
Regarding claim 10, Kumar discloses the operations further comprising:
extracting, using a first feature extraction neural network to extract first features from the synthetic environment and the abstract representation ([0040] neural network (e.g., a deep convolutional neural network) that has been trained to perform pose estimation using top-down videos of an open field.);
training a viewpoint network using the first features to create the trained viewpoint network ([0040] neural network may provide multiple two-dimensional markers (in some embodiments, twelve such markers) of a subject's anatomical location (also referred to as “keypoints”), for each frame of video describing the pose of the subject at each time point.);
extracting, using a second feature extraction neural network to extract second features from the synthetic environment and the abstract representation ([0040] another modular component may be capable of extracting several posture metrics); and
training a pose network using the second features to create the trained pose network ([0040] is a neural network (e.g., a deep convolutional neural network) that has been trained to perform pose estimation using top-down videos).
Regarding claim 11, Kumar discloses the operations further comprising:
minimizing a viewpoint L2 loss for a viewpoint output of the viewpoint network ([0050] the point tracker component 110 may be trained for a loss function, for example, a Gaussian distribution centered on the respective keypoint.); and
minimizing a pose L2 loss for a pose output of the pose network ([0103] training loss curves (FIG. 8C) show a fast convergence of the training loss without an overfitting of the validation loss.).
Regarding claim 12, Kumar discloses creating multiple tiles based on the synthetic image, wherein the multiple tiles include:
a limb tile for each limb of the humanoid shape ([0108] selected pose keypoints were either visually salient, such as ears or nose, or capture important information for understanding pose, such as limb joints or paw); and
a torso tile for a torso of the humanoid shape ([0046] point tracker component 110 may be configured to locate two-dimensional coordinates of a set of subject body parts, identified as keypoints, in an image or video).
Regarding claim 13, Kumar discloses A non-transitory computer-readable memory device configured to store instructions executable by one or more processors to perform operations ([0156] memories (1606/1706) may individually include volatile random access memory (RAM), non-volatile read only memory (ROM), non-volatile magnetoresistive memory (MRAM), and/or other types of memory) comprising:
receiving a real image that includes a human, the real image comprising a photograph or a frame of a video ([0041] image capture device 101 may be part of, included in, or connected to another device (e.g., device 1600), and may be a camera, a high speed video camera, or other types of devices capable of capturing images and videos, [0085] subject is a human);
creating a synthetic image corresponding to the real image, the synthetic image including a synthetic environment and a humanoid shape that correspond to the human ([0044] point data 112 may represent data tracking movements of a set of subject body parts over a time period represented in the video data);
predicting, using a trained viewpoint network and based on the synthetic image, a predicted viewpoint heatmap ([0029] heat maps, curves and plots. FIG. 16A is a heat map summarizing the effect sizes and q-values obtained from model M2), the trained viewpoint network comprising a first trained convolutional neural network ([0040] neural network may provide multiple two-dimensional markers (in some embodiments, twelve such markers) of a subject's anatomical location (also referred to as “keypoints”), for each frame of video describing the pose of the subject at each time point.);
predicting, using a trained pose network and based on the synthetic image, a predicted pose heatmap, the trained pose network comprising a second trained convolutional neural network ([0117] performing pose estimation after object detection, full resolution pose keypoint heatmaps were used to infer the posture of a single mouse at every frame);
providing, as input to a random synthetic environment, the predicted viewpoint heatmap and the predicted pose heatmap ([0125] multiple repeated measurements have been taken at two different ages giving rise to a nested hierarchical data structure. The models (M1, M2 M3) follow the standard LMM notation with (Genotype, BodyLength, Speed, TestAge) denoting the fixed effects and (MouseID/TestAge) (test age nested within the animal) denoting the random effect.);
classifying the reconstructed three-dimensional pose as a particular type of pose of the human in the real image ([0152], , SVM is a supervised learning model with associated learning algorithms that analyze data and recognize patterns in the data, and which are commonly used for classification and regression analysis).
Fu discloses creating a reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment (0073] applies a backward loop to reconstruct a previous pose estimation from current frames to improve robustness and minimize inconsistent estimation)
Kumar and Fu are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify method for tracking body parts of subject of Kumar to include creating a reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment as described by Fu.
The motivation for doing so would have been to generate an interpretable low-dimension representation of bodies in images (Fu, [0003]).
Therefore, it would have been obvious to combine Kumar and Fu to obtain the invention as specified in claim 14.
Regarding claim 15, Kumar discloses determining that at least one of the predicted viewpoint heatmap or the predicted pose heatmap specify a fuzzy location. ([0050] output of the neural network of the point tracker component 110 may be compared with the keypoint-centered Gaussian distribution, and the loss may be calculated as the mean squared difference between the respective keypoint and the heatmap generated by the point tracker component)
Regarding claim 16, Kumar discloses determining, that at least one of the predicted viewpoint heatmap or the predicted pose heatmap include more than three dimensions ([0045] point tracker component 110 may generate a first heatmap, where each cell in the heatmap may correspond to a pixel within the video frame, and may represent a likelihood of a first subject body part (e.g., a right forepaw) being located at the respective pixel).
Regarding claim 17, Kumar discloses determining, that at least one of the predicted viewpoint heatmap or the predicted pose heatmap include a time dimension ([0045] point tracker component 110 may generate a first heatmap, where each cell in the heatmap may correspond to a pixel within the video frame, and may represent a likelihood of a first subject body part (e.g., a right forepaw) being located at the respective pixel).
Regarding claim 18, Kumar discloses wherein the trained pose network and the trained viewpoint network are created by:
randomly selecting a pose from a set of poses; randomly selecting a viewpoint from a set of viewpoints ([0050] point tracker component 110 may be trained using training video data of subjects having varying physical characteristics, such as, different coat color, different body lengths, different body sizes, etc.);
generating the synthetic environment based at least in part on the pose and the viewpoint ([0050] the point tracker component 110 may be trained using an optimization algorithm, for example, a stochastic gradient descent optimization algorithm); and
deriving, from the synthetic environment, an abstract representation, a viewpoint heatmap, and a pose heatmap, wherein the viewpoint heatmap and the pose heatmap are used as supervised training targets ([0049] , the point tracker component 110 may generate multiple heatmaps, each heatmap representing an inference of where one keypoint representing one subject body part is located within a frame of the video data).
Regarding claim 19, Kumar discloses the operations further comprising:
extracting, using a first feature extraction neural network to extract first features from the synthetic environment and the abstract representation ([0040] neural network (e.g., a deep convolutional neural network) that has been trained to perform pose estimation using top-down videos of an open field.);
training a viewpoint network using the first features to create the trained viewpoint network ([0040] neural network may provide multiple two-dimensional markers (in some embodiments, twelve such markers) of a subject's anatomical location (also referred to as “keypoints”), for each frame of video describing the pose of the subject at each time point.);
extracting, using a second feature extraction neural network to extract second features from the synthetic environment and the abstract representation ([0040] another modular component may be capable of extracting several posture metrics); and
training a pose network using the second features to create the trained pose network ([0040] is a neural network (e.g., a deep convolutional neural network) that has been trained to perform pose estimation using top-down videos).
Regarding claim 20, Kumar discloses wherein creating the reconstructed three-dimensional pose based on the predicted viewpoint heatmap, the predicted pose heatmap, and the random synthetic environment comprises:
transforming a camera’s position from subject-centered coordinates to world coordinates ([0046] The point tracker component 110 may be configured to locate two-dimensional coordinates of a set of subject body parts, identified as keypoints).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kumar et al (US 20240057892 A1) and Fu et al (US 20230126178 A1) as applied to claim 8 above, and further in view Bashkirov et al (US 20220143820 A1).
Regarding claim 13, Kumar is silent to adding perlin noise to the synthetic image to introduce granular missing patches.
Bashkirov discloses adding perlin noise to the synthetic image to introduce granular missing patches ([0086] Simplex or Perlin coherent noise may be used to pattern floor properties distribution while training the robot controller. The boundary shape may be governed by the initial coherent noise before applying a transformation. The overall shape of an area may be defined by the number of octaves, lacunarity and time persistence of the coherent noise frequency distribution of the coherent noise.)
Kumar, Fu and Bashkirov are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify method for tracking body parts of subject of Kumar to include adding perlin noise to the synthetic image to introduce granular missing patches as described by Bashkirov.
The motivation for doing so would have been for smooth life-like motions of a character may be obtained through training a NN to accept controlled mechanism/object sensor readings/observations as inputs and outputs (Bashkirov, [0022]).
Therefore, it would have been obvious to combine Kumar, Fu and Bashkirov to obtain the invention as specified in claim 13.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHIVANG I PATEL whose telephone number is (571)272-8964. The examiner can normally be reached on M-F 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached on (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHIVANG I PATEL/Primary Examiner, Art Unit 2615
Prosecution Insights