DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
This action is in response to the Application filed on 03/26/2024.
Claims 1-14 are pending.
Priority
The Application claims priority to Korean Application KR10-2023-0040549 with filing date 03/28/2023, which is acknowledged.
Information Disclosure Statement
The Information Disclosure Statement (IDS) filed on 03/26/2024 has been fully considered by the examiner.
Drawings
Figure 4 of the Drawings is objected to by the examiner.
Figure 4 is objected to because it is believed that the bottom-left-most box is supposed to read “Part Affinity Field Map” instead of “Max pool Gaussian Heatmap” (see page 19, lines 7-9 of the originally filed specification, and present claims 4 and 11).
The examiner observes that the corresponding box in Figure 4 of the Korean Application does read “Part Affinity Field Map”, and thus this is believed to be a mere error during translation across the Applications.
PNG
media_image1.png
371
655
media_image1.png
Greyscale
Annotated Figure 4 of Present U.S. Application
PNG
media_image2.png
340
624
media_image2.png
Greyscale
Annotated Figure 4 of Korean Application
Claim Rejections – 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract ideas without significantly more.
Analysis for claim 1 is provided in the following. Claim 1 is reproduced in the following (annotation added):
A method of operating a neural network device for extracting result information using machine learning, the method comprising:
extracting feature data from an image frame;
storing a pre-trained first training model generated by performing machine learning on the feature data and including first training data;
storing a pre-trained second training model that is generated by performing machine learning on the first training data and includes second training data generated according to four arithmetic operations based on the first training data;
and generating two-dimensional vector information on result information by performing a probability-based operation on the second training data in the image frame based on the second training model.
Step 1: Does the claim belong to one of the statutory categories? Claim 1 is directed to a process, which is a statutory category of invention (YES).
Step 2A Prong One: Does the claim recite a judicial exception? Part d recites that the second training data is generated according to four arithmetic operations based on the first training data, which is directed to an abstract idea of mathematical calculations. Part e recites generating any kind of two-dimensional vector information by performing any kind of probability-based operation on the second training data, which is directed to an abstract idea of mental processes that can be practically performed in the human mind, or by a human using pen and paper. Any kind of mental or pen-and-paper determination of vector data based on a probability analysis of the training data would read on the limitations of part e (YES).
Step 2A Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application? Part b recites extracting feature data from an image frame, which is regarded as mere data gathering. Part c recites generating a first training model by performing machine learning on the feature data, and part d recites generating a second training model by performing machine learning on both the first and second training data. Parts c and d are both applied in the claim to ultimately obtain the data analyzed in part e, thus these limitations are regarded as merely using a computer as a tool to perform an abstract idea (NO).
Step 2B: Does the claim as a whole amount to significantly more than the recited exception? The claim as a whole recites data gathering and high-level machine learning to ultimately perform mathematical calculations and mental processes that can be practically performed in the human mind (NO). Claim 1 is not eligible.
Similar analysis is applicable to independent claim 8. Claim 8 is not eligible.
Claims 2, 6, 9, and 13 narrow the image to include a plurality of areas of a human body, and narrow the first training data to include joint and relation information relating to the plurality of areas of the body. This does not integrate the judicial exceptions into a practical application, as these limitations are still performed ultimately to enable the mental processes and mathematical calculations in the claims. Claims 2, 6, 9, and 13 are not eligible.
Claims 3 and 10 further limit the joint information to include heat map information, which does not integrate the judicial exceptions into a practical application, for similar reasons to claims 2, 6, 9, and 13 above. Claims 3 and 10 are not eligible.
Claims 4 and 11 narrow the process of generating the second training data to summing four different types of information, which is directed to mathematical calculations. Claims 4 and 11 are not eligible.
Claims 7 and 14 narrow the result information that the two-dimensional vector information is based on to include skeletal information corresponding to each body. This does not integrate the judicial exceptions into a practical application, as this is regarded as merely assigning a particular species to the two-dimensional vector information – the vector information can still be practically obtained in the human mind, regardless of what the information specifically represents. Claims 7 and 14 are not eligible.
Claims 5 and 12 recites that the second training model applies the second training data as an additional layer to the first training model, which does not integrate the judicial exceptions into a practical application. Claims 5 and 12 are not eligible.
To overcome the above 101 rejections, one suggestion from the examiner is to add limitations indicating that a graphics processing unit (or “GPU”) performs the actions of generating the second training data instead of a CPU. Such limitations would apply the judicial exceptions with a particular machine, which is indicative of a practical application. Furthermore, such limitations would reflect the improvements discussed in the originally filed specification, such as on page 2, lines 15-20 and from page 13, line 18 to page 14, line 10, where moving the post-processing operations traditionally performed by the CPU (such as generating the second training data) to the GPU improves the functioning of the computer by lightening the processing load on the CPU.
Claim Rejections – 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 5-10, and 12-14 are rejected under 35 U.S.C. 102(a)(1) and 102(a)(2) as being anticipated by Wang et al. (U.S. Publ. US-2020/0272888-A1).
Regarding claim 1, Wang discloses a method of operating a neural network device for extracting result information using machine learning (see figure 2), the method comprising:
extracting feature data from an image frame (see figure 2, convolutional neural network 22, backbone network 42, and paragraphs 0024-0026, where the backbone network extracts feature maps from input images);
storing a pre-trained first training model generated by performing machine learning on the feature data and including first training data (see figure 2, first head neural networks 44 for keypoints, second head neural networks 48 for PAFs and paragraphs 0027-0029, where the first networks 44 are trained to generate keypoint heatmaps / joint information from the feature maps, and the second networks 48 are trained to generate part affinity field heatmaps / PAF heatmaps / relation information; note figure 2, training data set 60);
storing a pre-trained second training model (see figure 2, rightmost box reciting "skeleton linking", which is performed by figure 6, skeleton grouping network 66) that is generated by performing machine learning on the first training data and includes second training data generated according to four arithmetic operations based on the first training data (see paragraph 0033, where the skeleton grouping network 66 is trained to link the keypoint heatmaps with the PAF heatmaps to generate instances of virtual skeletons; the examiner notes the breadth of the term "four arithmetic operations" as claimed, and argues that such a complex process executed by a machine learning system would easily involve performing at least four arithmetic operations on the keypoint and PAF heatmaps);
and generating two-dimensional vector information on result information by performing a probability-based operation on the second training data in the image frame based on the second training model (see paragraph 0048, where the skeleton grouping network 66 can obtain final vector information by further distinguishing between multiple bodies/skeletons in an image according to confidence scores indicating probabilities that each skeleton belongs to a same body in the image).
Regarding claim 2, Wang discloses wherein the image frame includes an image area that corresponds to a human body and is divided into a plurality of areas (see figures 4-5 and paragraphs 0030-0031, where the image depicts various body parts that are analyzed by the models as separate areas),
and the first training data is generated based on feature information corresponding to the plurality of areas (see figures 7A-7D and paragraph 0034),
and includes relation information between the plurality of areas and joint information of the body generated by the machine learning based on the feature information (see figure 2, first head neural networks 44 for keypoints, second head neural networks 48 for PAFs and paragraphs 0027-0029, where the first networks 44 are trained to generate keypoint heatmaps / joint information from the feature maps, and the second networks 48 are trained to generate part affinity field heatmaps / PAF heatmaps / relation information).
Regarding claim 3, Wang discloses wherein the joint information includes heat map information that corresponds to one area randomly selected from among the plurality of areas and indicates a probability that the one area is the area corresponding to the joint of the body (see paragraph 0027, where the keypoint heatmaps represent probabilities that each respective pixel belongs to one of the keypoints/joints of the body).
Regarding claim 5, Wang discloses wherein the second training model applies the second training data as an additional layer to the first training model (see figures 2 and 6, where the skeleton grouping network 66 comes after the head networks 44, 48).
Regarding claim 6, Wang discloses wherein the body included in the image frame is composed of a plurality of entities (see figures 4-5 and paragraphs 0030-0031, where the image depicts various body parts that are analyzed by the models as separate areas; each body part that has a keypoint is treated as a separate entity),
and the joint information and the relation information correspond to the plurality of entities (see figure 5B, where the keypoints 46 and PAFs 50 correspond to their respective body parts/entities, such as an elbow, forearm, and wrist).
Regarding claim 7, Wang discloses wherein the result information is information on each body corresponding to the plurality of entities, and includes skeletal information corresponding to each body (see figure 7G and paragraph 0035, where the final skeletal linking output indicates skeletal information on all identified body parts of each body).
Regarding claim 8, Wang discloses an apparatus for operating a neural network device for extracting result information using machine learning (see figure 1), the apparatus comprising:
a memory in which at least one program is stored (see figure 1, volatile memory 16 and non-volatile memory 18);
and a processor that performs a calculation by executing the at least one program, wherein the processor is configured to (see figure 1, processor 14, which can be a CPU, GPU, FPGA, ASIC, etc.).
The remainder of claim 8 recites steps identical to those of claim 1. Therefore, Wang anticipates claim 8 as applied to claim 1 above.
Regarding claim 9, Wang discloses claim 9 as applied to claim 2 above.
Regarding claim 10, Wang discloses claim 10 as applied to claim 3 above.
Regarding claim 12, Wang discloses claim 12 as applied to claim 5 above.
Regarding claim 13, Wang discloses claim 13 as applied to claim 6 above.
Regarding claim 14, Wang discloses claim 14 as applied to claim 7 above.
Examiner Note
The examiner notes that the cited prior art of record fails to disclose or reasonably suggest claims 4 and 11 under 35 U.S.C. 102 or 103. Thus, once all of the above 35 U.S.C. 101 rejections are resolved, claims 4 and 11 would be allowable if rewritten in independent form, including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS JOHN HELCO whose telephone number is (703)756-5539. The examiner can normally be reached on Monday-Friday from 9:00 AM to 5:00 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella, can be reached at telephone number 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/NICHOLAS JOHN HELCO/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667