DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
All amendments to the claims as filed on 11/6/2025 have been entered and action follows:
Response to Arguments
Rejection under 35 USC 101
Per the applicant’s amendments and persuasive arguments the rejections under 35 USC 101 are withdrawn.
Rejection under 35 USC 102(a)(1)
Per the applicants amendments, Affidavit and persuasive arguments the rejections under 35 USC 102(a)(1) are withdrawn.
Please see the new rejections to the amended claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 8-11, 12, 13, 15 and 18-23 are rejected under 35 U.S.C. 103 as being unpatentable over Datta et al (US 10,909,691; this reference has a prior publication data of Mar. 21, 2019).
With respect to claim 1, Datta discloses A method for profiling a subject's physical behavior over a period of time (see figure 1), the method comprising:
obtaining information indicative of movement of one or more portions of the subject over the period of time, (see figure 1, 100 camera, 50 subject; col. 16, lines 25-26, wherein …image information within the contours can then be extracted to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time…);
[applying a transform to the information indicative of movement to obtain a wavelet representation of the subject's physical behavior over the period of time;
embedding the wavelet representation into two dimensions to obtain an embedded representation of the subject's physical behavior over the period of time;]
clustering instances of one or more physical behaviors based at least in part on [the embedded representation] information indicative of movement;
generating a profile of the subject's physical behavior over the period of time based on the clustered instances of the one or more physical behaviors, (see col. 16, lines 20-28, wherein … The contours of the light areas in the plurality of processed images can be found and parameters from both area and depth image information within the contours can then be extracted to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time. The posture data points can then be clustered so that point clusters represent animal behaviors); and
outputting the profile of the subject's physical behavior, (see figure 1, numerical 113 computer monitor for output display), as claimed.
However, Datta fails to explicitly disclose applying a transform to the information indicative of movement to obtain a wavelet representation of the subject's physical behavior over the period of time;
embedding the wavelet representation into two dimensions to obtain an embedded representation of the subject's physical behavior over the period of time;
clustering instances of one or more physical behaviors based at least in part on the embedded representation information indicative of movement, (emphasis added) as claimed.
But, in col. 19, lines 10-25, wherein …To …make modeling computationally tractable, various techniques may be employed to dimensionally reduce each image. For example, a five-level wavelet decomposition may be performed, thereby transforming the image into a representation “applying a transform to the information indicative of movement to obtain a wavelet representation of the subject's physical behavior over the period of time” in which each dimension captured and pooled information at a single spatial scale; in this transformation, some dimensions may code explicitly for fine edges “embedding the wavelet representation into two dimensions to obtain an embedded representation of the subject's physical behavior over the period of time” on the scale of a few millimeters, while others encoded broad changes over spatial scales of centimeters. And, col. 16, lines 25-28, wherein …
The posture data points can then be clustered so that point clusters represent animal behaviors “physical behaviors based at least in part on the embedded representation information indicative of movement”, as claimed.
Therefore, it would have been obvious to one ordinary skilled in the art at the effective date of invention to simply utilize the teaching of Datta from various embodiments to yield the predictable results of monitoring a physical behavior of a subject.
With respect to claim 4, Datta further discloses wherein obtaining the information indicative of movement comprises: obtaining a plurality of video frames, wherein video frames of the plurality of video frames recorded the subject's physical behavior over a period of time, the video frames having been synchronously acquired by two or more cameras at different positions; and extracting, from the plurality of video frames, the information indicative of movement, (see col. 16, lines 16-28, wherein …camera 100 is used to obtain a stream of video images of the animal 50 having both area and depth information. …The posture data points can then be clustered so that point clusters represent animal behaviors; also see col. 16 lines 55-58, wherein …stereo-vision cameras (which may include groups of two or more two-dimensional cameras calibrated to produce a depth image…), as claimed.
With respect to claim 8, Datta further discloses smoothing, using a filter, the information indicative of movement, (see col. 17, lines 41-48, wherein …tracking the evolution of an imaged mouse's pose over time requires identifying the mouse within a given video sequence, segmenting the mouse from the background “filter” (in this case the apparatus the mouse is exploring), orienting the isolated image of the mouse along the axis of its spine, correcting the image for perspective distortions, and then compressing the image for processing by the mode), as claimed.
With respect to claim 9, Datta further discloses wherein generating a profile of the subject's physical behavior comprises generating an ethogram, (see col. 15, lines 60-62, wherein …(ii) assigning the at least one set of sub-second modules to a category that represents a type of animal behavior “ethogram”), as claimed.
With respect to claim 10, Datta further discloses identifying repeated behavioral sequences of the subject by: obtaining a similarity matrix by computing pairwise correlations using the ethogram; determining off-diagonal elements of the similarity matrix having a value over a threshold value, the off-diagonal elements corresponding to related behaviors of the subject; and clustering the related behaviors of the subject to identify repeated behavioral sequences of the subject, (see col. 30 line 42 to col. 31 line 4, wherein … after model training each module is assigned a pairwise transition probability with every other module in the set; these probabilities summarize the sequences of modules that were expressed by the mouse during behavior. Plotting these transition probabilities as a matrix revealed that they were highly non-uniform, with each module preferentially connected in time to some modules and not others (FIG. 6B; average node degree without thresholding 16.82±0.95, after thresholding bigram probabilities lower than 5 percent, 4.08±0.10). This specific connectivity between pairs of modules restricted the module sequences that were observed in the dataset (8900/˜125,000 possible trigrams) demonstrating that certain module sequences were favored; this observation suggests that mouse behavior is predictable, as knowing what the mouse is doing at any given moment in time informs an observer about what the mouse is likely to do next. Information theoretic analysis of the transition matrix confirmed that mouse behavior is significantly predictable, as the average per-frame entropy rate was low relative to a uniform transition matrix (without self-transitions 3.78±0.03 bits, with self-transitions 0.72±0.01 bits, entropy rate in uniform matrix 6.022 bits), and the average mutual information between interconnected modules was significantly above zero (without self-transitions 1.92±0.02 bits, with self-transitions 4.84 bits±0.03 bits). This deterministic quality to behavior likely serves to ensure that the mouse emits coherent patterns of motion; consistent with this possibility, upon inspection frequently-observed module sequences were found to encode different aspects of exploratory behavior), as claimed.
Claim 11 is rejected for the same reasons as set forth in the rejections of claim 1, because claim 11 is claiming subject matter of similar scope as claimed in claim 1.
With respect to claim 12, Datta discloses all the limitation as in claim 1. Furthermore, Datta discloses generating three-dimensional spatially-aligned image volumes using the obtained images of the subject, (see col. 18, lines 45-50, wherein …first generating a tuple of (x,y,z) coordinates for each pixel in real-world coordinates…); generating, using a trained statistical model comprising a three-dimensional convolutional neural network and the three-dimensional spatially-aligned image volumes, landmark position data associated with the subject, (see col. 21, line 53-60 for training model and col. 4, lines 60-62 for use of neural network), as claimed.
With respect to claim 13, Datta further discloses generating the three-dimensional, spatially- aligned image volumes comprises: determining a three-dimensional position of the subject using triangulation and the obtained images; centering a three-dimensional grid comprising voxels around the determined three- dimensional position of the subject; projecting spatial coordinates of the voxels to two-dimensional space of the images based on known positions of the two or more cameras; and generating the three-dimensional, spatially-aligned image volumes by projecting RGB image content of the images at each two-dimensional voxel location to the voxel's three-dimensional position, (see col. 18, lines 41-51, wherein …additional information may be extracted 275 from the video data including the centroid, head and tail positions of the animal, orientation, length, width, height, and each of their first derivatives with respect to time. Characterization of the animal's pose dynamics required correction of perspective distortion in the X and Y axes. This distortion may be corrected by first generating a tuple of (x,y,z) coordinates for each pixel in real-world coordinates, and then resampling those coordinates to fall on an even grid in the (x,y) plane using Delaunay triangulation), as claimed.
With respect to claim 15, Datta further discloses wherein the convolutional neural network comprises one or more convolutional layers, (see col. 4, lines 60-62 for use of neural network), as claimed.
With respect to claims 18 and 19, Datta further discloses determining a three-dimensional pose of the subject using the output landmark position data; and wherein the images comprise video frames acquired synchronously by the two or more cameras over a period of time, and determining the three-dimensional pose of the subject comprises determining the three- dimensional pose of the subject over the period of time, (see 4, lines 5-10, wherein …sensors to capture three-dimensional (3D) pose dynamics of the mouse, and then quantified how the mouse's pose changed over time by centering and aligning the image of the mouse along the inferred axis of its spine [spine is read as landmark]; also col. 16 lines 55-58, wherein …stereo-vision cameras (which may include groups of two or more two-dimensional cameras calibrated to produce a depth image…), as claimed.
With respect to claim 20, Datta further discloses obtaining, using the three-dimensional pose of the subject over the period of time, information indicative of movement of the subject over the period of time; clustering instances of one or more physical behaviors based at least in part on the information indicative of movement; generating a profile of the subject's physical behavior over the period of time based on the clustered instances of the one or more physical behaviors, (see col. 16, lines 20-28, wherein … The contours of the light areas in the plurality of processed images can be found and parameters from both area and depth image information within the contours can then be extracted to form a plurality of multi-dimensional data points, each data point representing the posture of the animal at a specific time. The posture data points can then be clustered so that point clusters represent animal behaviors); and outputting the profile of the subject's physical behavior, (see figure 1, numerical 113 computer monitor for output display), as claimed.
With respect to claim 21, Datta further discloses identifying repeated behavioral sequences of the subject by: obtaining a similarity matrix by computing pairwise correlations using the profile of the subject's physical behavior; determining off-diagonal elements of the similarity matrix having a value over a threshold value, the off-diagonal elements corresponding to related behaviors of the subject; and clustering the related behaviors of the subject to identify repeated behavioral sequences of the subject, (see col. 30 line 42 to col. 31 line 4, wherein … after model training each module is assigned a pairwise transition probability with every other module in the set; these probabilities summarize the sequences of modules that were expressed by the mouse during behavior. Plotting these transition probabilities as a matrix revealed that they were highly non-uniform, with each module preferentially connected in time to some modules and not others (FIG. 6B; average node degree without thresholding 16.82±0.95, after thresholding bigram probabilities lower than 5 percent, 4.08±0.10). This specific connectivity between pairs of modules restricted the module sequences that were observed in the dataset (8900/˜125,000 possible trigrams) demonstrating that certain module sequences were favored; this observation suggests that mouse behavior is predictable, as knowing what the mouse is doing at any given moment in time informs an observer about what the mouse is likely to do next. Information theoretic analysis of the transition matrix confirmed that mouse behavior is significantly predictable, as the average per-frame entropy rate was low relative to a uniform transition matrix (without self-transitions 3.78±0.03 bits, with self-transitions 0.72±0.01 bits, entropy rate in uniform matrix 6.022 bits), and the average mutual information between interconnected modules was significantly above zero (without self-transitions 1.92±0.02 bits, with self-transitions 4.84 bits±0.03 bits). This deterministic quality to behavior likely serves to ensure that the mouse emits coherent patterns of motion; consistent with this possibility, upon inspection frequently-observed module sequences were found to encode different aspects of exploratory behavior), as claimed.
With respect to claim 22, Datta further discloses wherein obtaining images of the subject comprises obtaining images of an animal, (see figure 1, numerical 50 mouse), as claimed.
Claim 23 is rejected for the same reasons as set forth in the rejections of claim 12, because claim 23 is claiming subject matter of similar scope as claimed in claim 12.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Datta et al (US 10,909,691; this reference has a prior publication data of Mar. 21, 2019) in view of Aonuma (US Pub. 2015/0227658).
With respect to claim 3, Datta discloses all the limitations as claimed and rejected above in claim 1. However, Datta fails to explicitly disclose wherein obtaining information indicative of movement of one or more portions of the subject comprises obtaining information indicative of movement of one or more limbs, joints, and/or a torso of the subject, as claimed.
Aonuma teaches wherein obtaining information indicative of movement of one or more portions of the subject comprises obtaining information indicative of movement of one or more limbs, joints, and/or a torso of the subject, (see paragraph 0039, wherein … he part information acquisition section 42 acquires information such as body height, length of leg, length from hip joint to knee joint of leg, length from knee joint to heel of leg, length of arm, length from shoulder to elbow of arm, length from elbow to fingers, etc…), as claimed.
It would have been obvious to one ordinary skilled in the art at the effective date of invention to combine the two references as they are analogous because they are solving similar problem of finding body movements of an object using image analysis. The teaching of Aonuma to attain the information of the joints, legs etc. can incorporated into Datta’s system as suggested (see Datta col. 16 line25-26, data points representing the posture of animal), for suggestion, and modifying yields a system for evaluating the movements of a body (see Aonuma paragraph 0003), for motivation.
Claims 5 and 7 are rejected under 35 U.S.C. 103 as being unpatentable over Datta et al (US 10,909,691; this reference has a prior publication data of Mar. 21, 2019) in view of Nakajima (US Pub. 2015/0002518).
With respect to claim 5, Datta discloses all the limitations as claimed and rejected above in claim 1. However, Datta fails to explicitly disclose affixing markers to the subject's body, as claimed.
Nakajima teaches affixing markers to the subject's body, (see paragraph 0042, wherein …a plurality of items of motion capture data including human motions by detecting positions of markers placed on a person serving as a model and acquires, based on these, a skeleton motion (a motion of the model expressed by a bone structure formed of "bones" and "joints") as a standard…), as claimed.
It would have been obvious to one ordinary skilled in the art at the effective date of invention to combine the two references as they are analogous because they are solving similar problem of finding body movements of an object using image analysis. The teaching of Nakajima to imaging subjects body with markers can be incorporated in to Datta’s system as suggested (see Datta col. 7, lines 13-16, system can be applied for humans as well), for suggestion, and modifying the system yields predictable results to affix markers on the subject body for getting movements of the subject.
With respect to claim 7, combination of Datta and Nakajima further discloses wherein extracting, from the plurality of video frames, the information indicative of movement comprises extracting, from the plurality of video frames, information indicative of movement of the markers affixed to the subject's body, (see Nakajima paragraph 0090, wherein …the standard skeleton acquisition unit 152 extracts temporal axis parameters (position, rotation, and the like) of joints in respective normalized skeletons…), as claimed.
Claims 6 is rejected under 35 U.S.C. 103 as being unpatentable over Datta et al (US 10,909,691; this reference has a prior publication data of Mar. 21, 2019) in view of Nakajima (US Pub. 2015/0002518) as applied to claim 5 above, and further in view of Heath et al (US Pub. 2019/0188425).
With respect to claim 6, Datta and Nakajima discloses all the limitations as claimed and rejected above in claim 1. However, they fail to explicitly disclose wherein affixing markers to the subject's body comprises piercing the subject's body with a marker, as claimed.
Heath teaches affixing markers to the subject's body comprises piercing the subject's body with a marker, (see paragraph 0003 wherein …Behaviors of animals may be determined by various sensors, … methods of such identification are used in the art, such as RFID tags, which may be implanted or attached to the animal; tattoos [is read as piercing the subject's body with a marker], such as tail tattoos; and ear notches), as claimed.
It would have been obvious to one ordinary skilled in the art at the effective date of invention to combine the references as they are analogous because they are solving similar problem of finding body movements of an object using image analysis. The teaching of Heath to use tattoo for ID can be incorporated in to Datta and Nakajima’s system as suggested (see Nakajima paragraph 0042, markers placed on a person serving as a model), for suggestion, and modifying the system yields predictable results for getting movements of the subject, for motivation.
Claims 16 is rejected under 35 U.S.C. 103 as being unpatentable over Datta et al (US 10,909,691; this reference has a prior publication data of Mar. 21, 2019) in view of A variational U-net for motion retargeting, by Kim.
With respect to claim 16, Datta discloses all the limitations as claimed and rejected above in claim 15. However, Datta fails to explicitly disclose wherein the one or more convolutional layers are arranged as a U-net, as claimed.
Kim teaches convolutional layers are arranged as a U-net, (see figure 1) as claimed.
It would have been obvious to one ordinary skilled in the art at the effective date of invention to combine the two references as they are analogous because they are solving similar problem of finding body movements of an object using image analysis. The teaching of Kim use a u-net conventional neural network to target the motion of an subject can be incorporated in to Datta’s system as suggested (see Datta col. 4, lines 61-62, neural network), for suggestion, and modifying the system yields appropriate motion of a subject (see Summary of Kim), for motivation.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Datta et al (US 10,909,691; this reference has a prior publication data of Mar. 21, 2019) in view of Shaevitz et al (WO 2020/072918 A1).
With respect to claim 17, Datta discloses all the limitations as claimed and rejected above in claim 15. However, Datta fails to explicitly disclose wherein generating the landmark position data comprises averaging three-dimensional confidence maps generated by the trained statistical model, as claimed.
Shaevitz teaches wherein generating the landmark position data comprises averaging three-dimensional confidence maps generated by the trained statistical model, (see paragraph 0017 and 0021), as claimed.
It would have been obvious to one ordinary skilled in the art at the effective date of invention to combine the two references as they are analogous because they are solving similar problem of finding body movements of an object using image analysis. The teaching of Shaevitz to get the landmark positions can be incorporated into Datta’s system as suggested (see Datta col. 16, lines 24-25, data points representing posture of animal), for suggestion, and modifying the system yields animal behavior determination (see Shaevitz paragraph 0003), for motivation.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIKKRAM BALI whose telephone number is (571)272-7415. The examiner can normally be reached Monday-Friday 7:00AM-3:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VIKKRAM BALI/Primary Examiner, Art Unit 2663