DETAILED ACTION
Remarks
This office action is issued in response to communication filed on 12/14/2023 Claims 1-20 are pending in this Office Action.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
2. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1 and 11:
Step 1: Statutory Category ?: Yes. claim 1 recites a method (i.e., a “process”) and claim 11 recites an apparatus (i.e., a “machine”) which are statutory categories.
Claim 1:
Step 2A-Prong 1: Judicial Exception Recited ?: Yes.
Claim 1 recites one or more limitations that can be performed in the human mind using observation, evaluation, judgment and opinion including with the help of a pen and paper:
setting a block location in which an occlusion will reside in each image of the set of images when the neural network is trained; adding a block to the block location in the set of training images;
Step 2A-Prong 2: Integrated into a practical application? No.
Claim 1 recites additional elements :
“initializing a neural network for a training procedure, the neural network structured to determine a pose of a manufacturing component in a testing image, each pose defined by a six dimensional pose which includes three rotations about separate axes and three translations along the separate axes” (pre/post solution activities which is insignificant extra solution activities (See MPEP 2106.05(g));
providing a set of training images to be used in training the neural network, each image in the set of training images including an associated pose; ( data gathering which is insignificant extra solution activities(See MPEP 2106.05(g)).
training the neural network using an error between a pose of a training image and the estimated pose of the training image provided by the neural network in light of the block added to each image in the set of training images. ( mere instructions to apply an abstract idea on a computer or merely using a computer as a tool to perform the abstract idea.(See MPEP 2106.05(f))
Step 2B: Recites additional elements that amount to significantly more than the judicial exception? No.
Claim 1 does not include additional elements that are sufficient to amount to significantly more than judicial exception. As indicates above, data gathering and pre/post solution activities are well-understood, routine conventional activities previously known to the industry and therefore do not amount to significantly more than the judicial exception. (See MPEP 2106.05(d)) and 2106.07(a)III). The neural network is at best equivalent of adding the words “apply it” to the exception. Even when considered in combination, the additional elements do not provide an inventive concept, claim 1 therefore is ineligible.
Claim 2 recites additional element of : “ wherein the training the neural network includes converging a loss function based on the error” which is mathematical calculations that falls within the mathematical concepts grouping of abstract ideas. Claim 2 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 2 is not patent eligible.
Claim 3 recites additional element of “ which further includes obtaining a test image and updating the training of the neural network through evaluation of a heat map of the test image” which is data gathering and therefore is insignificant extra solution activities(See MPEP 2106.05(g)) and is well-understood, routine conventional activities previously known to the industry and therefore do not amount to significantly more than the judicial exception. (See MPEP 2106.05(d)) and 2106.07(a)III). Even when considered in combination, the additional elements do not provide an inventive concept, claim 3 therefore is ineligible.
Claim 4 recites additional element of “ wherein the test image is separate from the set of training images, and which wherein the step of updating the training includes setting a test block location in which an occlusion will reside in the test image, adding a block to the test block location in the test image to form an occluded test image” which is a mental process . The additional element of “ calculating a heat map of the occluded test image” which is mathematical calculations that falls within the mathematical concepts grouping of abstract ideas. Claim 4 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 4 is not patent eligible.
Claim 5 recites additional element of “ which further includes evaluating the heat map against a resolution threshold, wherein if the heatmap fails to satisfy the resolution threshold then the step of setting a test block location is repeated with the test block at a new position” which is mathematical calculations that falls within the mathematical concepts grouping of abstract ideas. Claim 5 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 5 is not patent eligible.
Claim 6 recites additional element of “ wherein the repeated step of setting a test block location is accomplished by randomly setting a test block location” which is a mental process . Claim 6 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 6 is not patent eligible.
Claim 7 recites additional element of “ wherein the repeated step of setting a test block location is accomplished by defining a block location based upon the heat map of the occluded test image” which is a mental process . Claim 7 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 7 is not patent eligible.
Claim 8 recites additional element of “which further includes, prior to the step of adding a block to the block location in the set of training images, evaluating a comparison of the heat map of the occluded test image to a prior determined heat map against a threshold and if the threshold is satisfied then proceeding to the step of adding a block” which is a mental process . Claim 8 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 8 is not patent eligible.
Claim 9 recites additional element of “wherein the step of setting a test block location includes randomly setting the test block location, and which further includes evaluating the heat map against a resolution threshold, wherein if the heatmap fails to satisfy the resolution threshold then the step of setting a block location is repeated with a block at a new position” which is a mental process . Claim 9 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 9 is not patent eligible.
Claim 10 recites additional element of “wherein after the step of adding a block to the test block location to form an occluded test image then initializing a translation counting matrix and a rotation counting matrix corresponding to the pixels in the occluded test image, adding the value of one to the locations of each of the counting matrices that correspond to pixels covered by the block used to form the occluded test image, calculating a translation and rotation error based on a comparison between the translation pose and rotation pose of the test image and a pose result of driving the trained neural network with the occluded test image, cumulating a total translation error and rotation error if the step of setting a block location is repeated, and dividing the translation and rotation error by the respective counting matrices” which is mathematical calculations that falls within the mathematical concepts grouping of abstract ideas. Claim 10 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 10 is not patent eligible.
Claim 11:
Step 2A-Prong 1: Judicial Exception Recited ?: Yes.
Claim 11 recites one or more limitations that can be performed in the human mind using observation, evaluation, judgment and opinion including with the help of a pen and paper:
“receive a command to set a block location in which an occlusion will reside in each image of the collection of training images when the neural network is trained; add a block to the block location in the collection of training images”
Step 2A-Prong 2: Integrated into a practical application? No.
Claim 11 recites additional elements :
a collection of training images, each of image of the images paired with an associated pose of a manufacturing component, each pose defined by a six dimensional pose which includes three rotations about separate axes and three translations along the separate axes; (data gathering which is insignificant extra solution activities)
a controller structured to train the neural network and configured to perform the following: initialize the neural network for a training procedure to be conducted with the collection of training images; train the neural network using an error between a pose of a training image and the estimated pose of the training image provided by the neural network in light of the block added to each image in the collection of training images. ( mere instructions to apply an abstract idea on a computer or merely using a computer as a tool to perform the abstract idea.(See MPEP 2106.05(f))
Step 2B: Recites additional elements that amount to significantly more than the judicial exception? No.
Claim 11 does not include additional elements that are sufficient to amount to significantly more than judicial exception. As indicates above, data gathering is well-understood, routine conventional activities previously known to the industry and therefore do not amount to significantly more than the judicial exception. (See MPEP 2106.05(d)) and 2106.07(a)III). The controller and neural network are at best equivalent of adding the words “apply it” to the exception. Even when considered in combination, the additional elements do not provide an inventive concept, claim 11 therefore is ineligible.
Claim 12 recites additional element of “which further includes a loss function to assess the error between the pose of the training image and the estimated pose of the training image, wherein the controller is further structured to receive a command to update a block location and add a block to the updated block location if a loss from the loss function has not converged” which is mathematical calculations that falls within the mathematical concepts grouping of abstract ideas. Claim 12 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 12 is not patent eligible.
Claim 13 recites additional element of “wherein the controller is structured to restart training of a trained neural network based upon an evaluation of a heatmap of the test image, wherein the heatmap is determined after a heatmap step block location has been determined and a heatmap step block added at the heatmap step block location to the test image” which is mere instructions to apply an abstract idea on a computer or merely using a computer as a tool to perform the abstract idea.(See MPEP 2106.05(f)) and at best equivalent of adding the words “apply it” to the exception. Even when considered in combination, the additional elements do not provide an inventive concept, claim 13 therefore is ineligible.
Claim 14 recites additional element of “wherein the operation to restart training includes re-initializing the neural network so that it is ready for training, wherein the test image is separate from the set of training images, and wherein the controller is structured to set the block location and add the block to the block location to form an occluded test image after the controller restarts training of the trained neural network” which is mere instructions to apply an abstract idea on a computer or merely using a computer as a tool to perform the abstract idea.(See MPEP 2106.05(f)) and at best equivalent of adding the words “apply it” to the exception. Even when considered in combination, the additional elements do not provide an inventive concept, claim 14 therefore is ineligible.
Claim 15 recites additional element of “wherein the controller is further structured to evaluate the heat map against a resolution threshold, wherein if the heatmap fails to satisfy the resolution threshold then the controller is structured to repeat the operation to determine a heatmap step block location and add the heatmap step block to the heatmap step block location” which is a mental process . Claim 15 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 15 is not patent eligible.
Claim 16 recites additional element of “wherein when the controller is operated to repeat the determination of a heatmap step block location is accomplished by an operation to randomly set a heatmap step block location” ” which is a mental process . Claim 16 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 16 is not patent eligible.
Claim 17 recites additional element of “wherein when the controller is operated to repeat the determination of a heatmap step block location is accomplished an operation define a block location based upon the heat map of the occluded test image” which is a mental process . Claim 17 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 17 is not patent eligible.
Claim 18 recites additional element of “wherein the controller is further structured such that prior to the operation to add a block to the block location in the set of training images the controller is operated to evaluate a comparison of the heat map of the occluded test image to a prior determined heat map against a threshold and if the threshold is satisfied then proceeding to the operation to add a block” which is a mental process . Claim 18 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 18 is not patent eligible.
Claim 19 recites additional element of “wherein the operation to set a test block location includes an operation to randomly set the test block location, and wherein the controller is further structured to evaluate the heat map against a resolution threshold, wherein if the heatmap fails to satisfy the resolution threshold then the operation to set a block location is repeated with a block at a new position”
Claim 20 recites additional element of “wherein after the operation to add a block to the test block location to form an occluded test image, the controller is structured to initialize a translation counting matrix and a rotation counting matrix corresponding to the pixels in the occluded test image, add the value of one to the locations of each of the counting matrices that correspond to pixels covered by the block used to form the occluded test image, calculate a translation and rotation error based on a comparison between the translation pose and rotation pose of the test image and a pose result of driving the trained neural network with the occluded test image, cumulate a total translation error and rotation error if the step of setting a block location is repeated, and divide the translation and rotation error by the respective counting matrices” which is mathematical calculations that falls within the mathematical concepts grouping of abstract ideas. Claim 20 does not include any additional element that integrates the abstract idea into practical application in step 2A-Prong 2 and amounts to significantly more than the judicial exception in step 2B. Claim 20 is not patent eligible.
3. Claims 11- 20 are rejected under 35 U.S.C. 101 because claimed invention directed toward non-statutory subject matter.
Claim 11 recites an apparatus comprising a collection of training images and a controller . The specification of present application at page 6, indicates that the controller can be software, firmware and hardware or any combination “Alternatively, one or more of the controllers 112 and the program instructions executed thereby can be in the form of any combination of software, firmware and hardware, including state machines”. Accordingly, the recited "apparatus" fails to recite a required hardware element. During examination, the claims must be interpreted as broadly as their terms reasonably allow. The broadest reasonable interpretation of a claim drawn to an apparatus that fails to recite a required hardware element covers software per se. Software is not a “process”, a “machine”, a manufacture”, or a “composition of matter” as defined in 35 U.S.C. 101.
Accordingly, the recited “apparatus” is not a “process”, a “machine”, a “manufacture”, or a “composition of matter" as defined in 35 U.S.C 101 and claim 11 fails to recite statutory subject matter as defined in 35 U.S.C 101. Claims 12-20 are also rejected under 35 USC 101 for failing to recite a hardware element. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1 and 11 recite the limitation of “training the neural network using an error between a pose of a training image and the estimated pose of the training image provided by the neural network in light of the block added to each image in the set of training images”. It is not clear the phrase “in light of” is referring to in the claim language. Appropriate correction is required. Due to the dependency on claims 1 or 11, claims 2-10 and 12-20 are also indefinite.
Allowable Subject Matter
Claims 5-10 and 15-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Although these claims are allowable over prior art, all other rejections and/or objections (if any) such as 101/112/claim objection must be overcome before the claims are allowed.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 and 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over SU et al., "SynPo-Net-Accurate and Fast CNN-Based 6DoF Object Pose Estimation Using Synthetic Training," Sensors, 21(1): 16 pp. (Jan. 5, 2021) (Cited on applicant’s IDS filed 1/31/25), hereinafter “Su” and further in view of Mestha et al. WO/2020/077198 (Cited on applicant IDS filed 12/14/2023) , hereinafter “Mestha”)
As to claim 1, Su teaches a method to train a neural network using heat map derived feedback, the method comprising: initializing a neural network for a training procedure, the neural network structured to determine a pose of a manufacturing component in a testing image, each pose defined by a six dimensional pose which includes three rotations about separate axes and three translations along the separate axes (Su’s abstract teaches this paper presents a novel approach using a Convolution Neural Network trained exclusively on single channel synthetic images of objects to regress 6DoF object poses directly. Su section 4.2 teaches training dataset with proposed domain adaptation technique) ;
providing a set of training images to be used in training the neural network, each image in the set of training images including an associated pose; (Su section 4.2 teaches training data set)
setting a block location in which an occlusion will reside in each image of the set of images when the neural network is trained; adding a block to the block location in the set of training images (Su section 4.2 teaches we augmented our rendered training data by randomly adding various effects that, Gaussian noise, random contrast and brightness adjustment, motion blur , speckle noise.) ; and
[training the neural network using an error between a pose of a training image and the estimated pose of the training image provided by the neural network in light of the block added to each image in the set of training images]
Su fails to expressly teach training the neural network using an error between a pose of a training image and the estimated pose of the training image provided by the neural network in light of the block added to each image in the set of training images.
However, Mestha teaches training the neural network using an error between a pose of a training image and the estimated pose of the training image provided by the neural network in light of the block added to each image in the set of training images.(Mestha par [0243] teaches the optimal set of parameters may minimize the error between the actual pose data in the training data and the calculate pose data from the AI-based camera pose model (e.g., at block 4607); the parameters are adjusted to minimize the error using learning/optimization techniques applicable to deep networks depending on the network size and depth. The final output (e.g., at block 4609) is a trained AI-based camera pose model that computes the 6DOF coordinates from the image pixel intensities captured by a camera)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teaching of Su and Mestha to achieve the claimed invention. One would have been motivated to make such combination to improve the accuracy of the neural network.
As to claim 2, Su and Mestha teach the method of claim 1, wherein the training the neural network includes converging a loss function based on the error. (Mestha par [0243] teaches the optimal set of parameters may minimize the error between the actual pose data in the training data and the calculate pose data from the AI-based camera pose model)
Claims 11-12 merely recite an apparatus to perform the method of claims 1-2 respectively. Accordingly, Su and Mestha teach every limitation of claims 11-12 as indicates in the above rejection of claims 1-2 respectively.
Claims 3-4 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Su, Mestha and further in view of Gernorth et al.(US Patent Application Publication 2019/0080149 A1, hereinafter “Gernorth”)
As to claim 3, Su and Mestha teach the method of claim 1 but fail to teach which further includes obtaining a test image and updating the training of the neural network through evaluation of a heat map of the test image.
However, Gernorth teaches obtaining a test image and updating the training of the neural network through evaluation of a heat map of the test image.(Gernorth par [0042] teaches input image may be provided to neural network to generate one or more landmark heat maps and occlusion heat map).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teaching of Su , Mestha and Gernorth to achieve the claimed invention. One would have been motivated to make such combination to improve the effectiveness of the pose detection.
As to claim 4, Su, Mestha and Gernorth teach method of claim 3, wherein the test image is separate from the set of training images, and which wherein the step of updating the training includes setting a test block location in which an occlusion will reside in the test image, adding a block to the test block location in the test image to form an occluded test image (Su section 4.2 teaches we augmented our rendered training data by randomly adding various effects that, Gaussian noise, random contrast and brightness adjustment, motion blur , speckle noise. It is well known in the art that the test image is separate from training images) , and calculating a heat map of the occluded test image. ( Gernorth par [0042] teaches input image may be provided to neural network to generate one or more landmark heat maps and occlusion heat map)
As to claims 13-14, see the above rejection of claims 3-4 .
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Yabuuchi ., US 20230377315 A1 par [0006] discloses preparing a mask image in which a mask region covering a specific portion set from the original image. Katayama et al., US 20230298327 A1, par [0190] discloses training data from heat maps generated respectively from test images of various types of defects. Hiasa., US 20220254139 A1, par [0006] discloses adding blur to the image and a machine learning model using the blurred image and ground truth map.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HIEN DUONG whose telephone number is (571)270-7335. The examiner can normally be reached Monday-Friday 8:00AM-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HIEN L DUONG/Primary Examiner, Art Unit 2147
Prosecution Insights