Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, filed 8/20/2026, with respect to the rejection(s) of claim(s) 1-3, 5-21 under 35 U.S have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Akbas et. al. (United States Patent US 12,094,159 B1)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, 5-7, 9-21 are rejected under 35 U.S.C. 103 as being unpatentable over Choutas et. al. (European Patent EP 3547211 A1) in view of Narapureddy et. al. (United States Patent US 11625838 B1) and Akbas et. al. (United States Patent US 12,094,159 B1).
Regarding claims 1 and 9, Choutas et. al. discloses an apparatus comprising: a communications interface at which to receive (Choutas et. al. para. 0059) from an image source, raw image data that includes a representation of a first person and a second person (Choutas et. al. figure 2; paras. 0014, 0043); a memory storage unit in which to store the raw image data; and a neural network engine (Choutas et. al. para. 0002) to apply a first convolution to the raw image data to extract first features from a first output, to downsample the first output to extract a first set of subfeatures from a first suboutput, to apply by the first branch to the first output, a second convolution to extract a second set of features that is representative of a second output, the second convolution to the first suboutput to extract a second set of subfeatures from a second suboutput, (Choutas et. al. para. 0014-0019), to generate joint heatmaps (Choutas et. al. para. 0043-0045: the proposed representation can be extended to any kind of heatmap, human joints heatmaps is given as an example, which can be applied to bone. Furthermore, this solution can handle the presence of multiple people and is robust to occlusion and truncation. It extracts joint heatmaps as well as fields that represent the affinities between pairs of joints corresponding to bones, in order to associate the different joint candidates into instances of human poses.) of the first object and the second object, and bone heatmaps (Choutas et. al. para. 0043) of the first object and the second object (Choutas et. al. para. 0014-0019, figure 5). Choutas et. al. also teaches a neural network engine that includes a higher-resolution branch and a lower-resolution branch (Choutas et. al. para 0044, the spatial resolution of a heatmap could be lower than the input frame, due to the stride of the network).
However, Choutas et. al. fails to disclose as claimed, a neural network engine that includes (i) a first branch designed to handle inputs at a first resolution and (ii) a second branch designed to handle inputs at a second resolution lower than the first resolution, merge the second output generated by the first branch and the second suboutput generated by the second branch to produce a merged data, apply a third convolution to the merged data to extract a third set of features that is representative of a merged output; and generate, based on the merged output,(i) one or more first joint heatmaps of the first person, (ii) one or more joint heatmaps of the second person, (iii) one or more first bone heatmaps of the first person, and(iv) one or more second bone heatmaps of the second person.
Akbas et. al. teaches a neural network engine that includes (i) a first branch designed to handle inputs at a first resolution and (ii) a second branch designed to handle inputs at a second resolution lower than the first resolution (Akbas et. al. Abstract, Fig. 1, Backbone neural network splits into two subbranches shown as Keypoint Subnet and Person Detection Subnet, which is merged into Pose Residual Net, col 2, lines 60-67: The authors calculated the loss in each level of the network to learn coarser to finer representations of parts. Multi-person pose estimation solutions branched out as bottom-up and top-down methods. Thus, this coarser to finer representation is where the second resolution is higher than the first resolution or vice versa. Col 16, lines 1-20: all of the features are arranged in a single row, effectively turning the high-dimensional feature vector into a one-dimensional vector. Col 19, lines 5-10: 480x480 images were used as inputs and 120x120x(K+1) heatmaps were outputted per input. The lower resolutions harmed the mAP results, while higher resolutions yielded longer training and inference complexity. Col 9 lines 50-67, Col 10 lines 1-24: upsampling and downsampling of features in a pyramid network.).
PNG
media_image1.png
520
672
media_image1.png
Greyscale
PNG
media_image2.png
719
662
media_image2.png
Greyscale
PNG
media_image3.png
598
876
media_image3.png
Greyscale
Narapureddy et. al. teaches and merge the second output generated by the first branch and the second suboutput generated by the second branch to produce a merged data, apply a third convolution to the merged data to extract a third set of features that is representative of a merged output; and generate, based on the merged output,(i) one or more first joint heatmaps of the first person, (ii) one or more joint heatmaps of the second person, (iii) one or more first bone heatmaps of the first person, and(iv) one or more second bone heatmaps of the second person (Narapureddy et. al., Fig. 4, 422, Generating merged feature data based at least in part on merging the second feature data over time, col 4, lines 20-50, the techniques described herein operate in a common spatiotemporal volume, are end-to-end learnable, and are not restricted to the multi-view setting only. Joints and/or joint data, as described herein, may correspond to anatomical human joints. Col 6, lines 15-20, For each person, we train the network to detect its “center”, which is defined as the midpoint between neck and center of the hips of the person. The loss at each time t is expressed directly as a distance between the expected heatmap and the output heatmap. Fig. 2, col 5, lines 38-62, All matched descriptors which overlap are then merged (merging 216) into a single descriptor which is finally deconvolved (deconvolutions 218) into a 3D pose for the person tracked at central frame. Col 10, lines 1-14, a softmax-based merging strategy is used, Col 8., lines 20-21).
PNG
media_image4.png
318
524
media_image4.png
Greyscale
PNG
media_image5.png
872
416
media_image5.png
Greyscale
PNG
media_image6.png
602
822
media_image6.png
Greyscale
This is important to the claimed invention because the merging of two different suboutputs from the same convolution applied to two different neural network branches with two different inputs are combined back together to produce the final output for the joint heatmaps. Furthermore, the distinction between the two branches are important because one is higher resolution and one is lower resolution. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have substituted the neural network backbone and the generating merged feature data steps of Narapureddy et. al. from 3D to 2D, combined with the teachings of Choutas et. al. and neural network branches of Akbas et. al. so that these features are used in the 2D human pose estimation solution (KSR Rationale B, Rationale A). This method also produces fewer errors when estimating 2D poses in each frame, as 2D images are inherent in the 3D space.
Dependent claims 2-3, 5-6, and 8 rely on the apparatus of claim 1, and dependent claims 10-14 rely on the method of claim 9.
Claims 2 and 10 pertain to the apparatus of claim 1 and method of claim 9, wherein the second suboutput is upsampled and merged with the second output to generate the merged data. Narapureddy et. al. discloses second suboutput is upsampled and merged with the second output to generate the merged data (Narapureddy et. al., Fig. 4 422, Generating merged feature data based at least in part on merging the second feature data over time).
Claims 3 and 11 recite the apparatus of claim 2 and method of claim 10, wherein the second output is downsampled and merged with the second suboutput to generate the merged data. Narapureddy et. al. discloses wherein the second output is downsampled and merged with the second suboutput to generate the merged data (Narapureddy et. al. Fig. 2, Merging step).
Regarding claim 12, Narapureddy et. al. discloses the method of claim 11, further comprising applying a third convolution to the merged output prior to generating the joint heatmaps and the bone heatmaps (Narapureddy et. al. Fig. 2, col. 6, lines 52-60, col. 8, lines 23-27, Tesseract Convolutions. The tesseract is then passed through multiple 4D convolutions and max pooling layers of 4D CNN to produce a reduced size tesseract feature. The merged tesseract is finally passed through multiple 4D deconvolution layers during deconvolution to produce a vector of 3D heatmaps of the person’s joints at time t.).
Claims 5-6 and claims 13-14 specify the apparatus of claim 1 and method of claim 9, wherein the first set of features are low level features and more specifically as edges. Narapureddy et. al. discloses wherein the first set of features are low level features and more specifically as edges. (Narapureddy et. al. col 7, lines 5-15).
PNG
media_image7.png
312
548
media_image7.png
Greyscale
Claims 7 and 15-16 recite the apparatus of claim 1 and the method of claim 9, wherein downsampling comprises maximum pooling and upsampling comprises a deconvolution operation. Narapureddy et. al. discloses wherein downsampling comprises maximum pooling and upsampling comprises a deconvolution operation (Narapureddy et. al. col. 6 lines 52-60, col. 8, lines 23-27, Tesseract Deconvolutions. The input to this sub-network is the output of the HRNet (or other backbone network) pre-final layer which is cast in 3D at each time stamp. The same procedure is followed as for the person detection network to generate the features for each time instance of the tesseract. The tesseract is then passed through multiple 4D convolutions and max pooling layers of 4D CNN to produce a reduced size tesseract feature. The merged tesseract is finally passed through multiple 4D deconvolution layers during deconvolution 218 to produce a vector of 3D heatmaps of the person’s joints at time t). However, using maximum pooling and deconvolution for down-and upsampling, respectively, are regarded as an obvious alternative in the context of CNNs.
Regarding claim 17, which is a non-transitory computer readable medium encoded with codes, wherein the codes are to direct a processor to carry out the method of claim 9, and associated with the apparatus of claim 1, which the rejection analysis is incorporated herein.
Claims 18-20 recite the non-transitory computer readable medium of claim 17, wherein the codes are to direct the processor to carry out the tasks carried out by the apparatus as outlined by claims 2-4.
Claim 18 recites wherein the codes are to direct the processor to upsample the second suboutput and to merge the second suboutput with the second output to generate a first merged output. Choutas et. al. discloses this limitation in paragraph [0014]-[0019].
Claim 19 recites wherein the codes are to direct the processor to downsample the second output and to merge the second output with the second suboutput to generate a first merged suboutput. Choutas et. al. discloses this limitation in paragraph [0014]-[0019].
Claim 20 recites wherein the codes are to direct the processor to apply a third convolution to the second merged output to generate a third output, and to apply the third convolution the first merged suboutput to generate a third suboutput. Choutas et. al. discloses this limitation in paragraphs [0061]-[0062].
Claim 21 recites the non-transitory computer readable medium of claim 19, and Narapureddy et. al. further discloses wherein the first set of features are edges (Narapureddy et. al. col 7, lines 5-15).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Choutas et. al. (European Patent EP 3547211 A1) in view of Narapureddy et. al. (United States Patent US 11625838 B1) and Akbas et. al. (United States Patent US 12,094,159 B1) as applied to claim 1 above, and further in view of Jun et. al. (Jun, J., Lee, J., & Kim, C. (2020). Human Pose Estimation Using Skeletal Heatmaps. 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 1287-1292, and Wang et .al. (Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., Liu, W., & Xiao, B. (2019). Deep High-Resolution Representation Learning for Visual Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43, 3349-3364.).
Regarding claim 8, Choutas et. al. and Narapureddy et. al. disclose the apparatus of claim 1, wherein the merged output is used to generate (i) a first plurality of joint heatmaps of the first person, wherein each of the first plurality of joint heatmaps corresponds to a different joint of the first person. However, Choutas et. al. and Narapureddy et. al. fail to disclose (ii) a second plurality of joint heatmaps of the second person, wherein each of the second plurality of joint heatmaps corresponds to a different joint of the second person,(iii) the one or more first bone heatmaps of the first person, wherein each of the one or more first bone heatmaps corresponds to a different pair of the first plurality of joint heatmaps, and(iv) the one or more second bone heatmaps of the second person, wherein each of the one or more second bone heatmaps corresponds to a different pair of the second plurality of joint heatmaps.
Jun et. al. and Wang et. al. teaches the second output and the second suboutput are merged (Jun et. al.: first fusion step in figure 2; Wang et. al.: first fusion step in figure 2, sections 3.1-3.2, figure 3) to generate joint heatmaps of the first person and the second person (Jun et. al. figure 1f, output heatmaps in figure 2, first paragraph in section II, multiple people in figure 5), and bone heatmaps of the first person and the second person (Jun et. al. paragraph 6 in section 1: "Given an image in Figure. 1(a), skeletal attention module produces skeletal heatmaps as shown in Fig. 1(b)", HS in figure 3, section II.C). The merging of the second output and the second suboutput to generate joint heatmaps of the first person and the second person, and bone heatmaps of the first person and the second person, is important to the claimed invention because this is the step where two different branches with the same convolution applied are combined back together. Thus, it would have been obvious to one skilled in the art prior to the effective filing date of the claimed invention to have combined the teachings of Choutas et. al., Narapureddy et. al., Akbas et. al., Jun et. al., and Wang et. al., so that these features are shown in the final solution of the neural network.
Conclusion
Response to Amendment
Examiner has carefully considered the amended claims and performed an updated search. New prior art was found to reject the amended claims as outlined above.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JESSICA YIFANG LIN whose telephone number is (571)272-6435. The examiner can normally be reached M-F 7:00am-6:15pm, with optional day off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at 571-272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JESSICA YIFANG LIN/Examiner, Art Unit 2668 September 14, 2026
/VU LE/Supervisory Patent Examiner, Art Unit 2668