DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a mental process of overlaying a generated status image on a generated simulated image without significantly more. The claim(s) 1 recite(s)
“An image processing device for performing processing operations to visualize a status of recognition of a detection target”, which is directed to the preamble and is not given patentable weight
“when performing an image recognition operation that uses a machine learning model for detecting a predetermined event from a captured image,” , which is directed to the preamble is not given patentable weight. If this were not part of the preamble, the neural network would be an additional element recited at a high level of generality and amount to “apply it” see MPEP 2106.05(f). the performing image processing is directed to a mental process, for example a person could look at an image ad detect a predetermined event.
“wherein the processing operations are performed by a processor,”, if this were not part of the preamble, the processor would be an additional element recited at a high level of generality and amounts to “apply it”. See MPEP 2106.05(f)
and include:
“in response to a user's operation to designate an image processing condition, performing an image processing operation on an original image based on the designated image processing condition to thereby generate a simulated image that reproduces an image captured in a specific situation; and”, which is directed to an mental process. For example a person could create a simulated image, for example cropping.
“generating a status image that represents a status of recognition of a detection target,”, which his directed to a mental process, for example a person can create an arrow on sticky paper.
“when performing the image recognition operation on the simulated image, and overlaying the generated status image on the simulated image to produce a result image as a result of visualization, which is output as display information.” Which is directed to a mental process for example a person could take the sticky paper and overlay it on the
This judicial exception is not integrated into a practical application. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claim 8 is rejected under similar grounds as claim 1.
Claims 2-7 are rejected as dependent upon claim 1 and not providing additional elements that amount to significantly more.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “ An image processing device for performing processing operations to visualize a status of recognition of a detection target when performing an image recognition operation that uses a machine learning model for detecting a predetermined event from a captured image, wherein the
processing operations are performed by a processor, and include:
in response to a user's operation to designate an image processing condition, performing an image processing operation on an original image based on the designated image processing condition to thereby generate a simulated image that reproduces an image captured in a specific situation; and
generating a status image that represents a status of recognition of a detection target,
when performing the image recognition operation on the simulated image, and overlaying the generated status image on the simulated image to produce a result image as a result of visualization, which is output as display information.”
It is not clear where the preamble ends and the body of the claim starts. For purpose of Examination, is assuming the preamble ends with the “:”.
The overlaying limitation is not clear if it is part of the “when clause” or separate. The Examiner believes it should be on a separate line separated by a “;”
The claim recites a device without any structure (since everything after the word “device” is written as intended use language and not given patentable weight). (Mixed statutory class) The Examiner recommends An image processing device comprising :
A machine learning model configured to detect a predetermined event from a captured image;
A processor configured to in response to ….
Claims 8 are rejected under similar grounds as claim 1.
Claims 2-7 are rejected as dependent on a rejected claim.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-8 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Jha (20222/0051020)
Jha discloses Claim 1. An image processing device for performing processing operations to visualize a status of recognition of a detection target when performing an image recognition operation that uses a machine learning model (Jha, Fig. 3A #106, “DNN Model”) for detecting a predetermined event from a captured image(Jha, Fig. 3A #302, paragraph 53 “acquire image data; #106 , paragraph 61” perform a cell-wise search for the detection of the object of interest”), wherein the
processing operations are performed by a processor, and include:
in response to a user's operation to designate an image processing condition(Jha, paragraph 48, “For example, the electronic apparatus 102 may receive a user input via the I/O device 206 to select an object of interest from multiple objects in an image frame and to initiate the XAI workflow (which includes object detection workflow and heatmap visualization workflow).”), performing an image processing operation on an original image based on the designated image processing condition to thereby generate a simulated image that reproduces an image captured in a specific situation; and (Jha, “[0054] After the acquisition, the circuitry 202 may input the first image frame 302A to the DNN model 106. In some embodiments, the circuitry 202 may resize the input first image frame 302A to match a size of an input layer of the DNN model 106 and then pass the resized first image frame 302A to the input layer of DNN model 106”)
generating a status image that represents a status of recognition of a detection target,
when performing the image recognition operation on the simulated image, (Jha, fig. 3B #322, paragraph 76, “[0076] At 320, an explanation generation operation may be performed. In such an operation, a heatmap generation may be performed at 322 which may be followed by followed by a metric calculation at 324.” )and overlaying the generated status image on the simulated image to produce a result image as a result of visualization, which is output as display information. , (Jha, fig. 3B #322, paragraph 77, “[[0077] At 322, a heatmap may be generated. In an embodiment, the circuitry 202 may generate a visualization frame 322A based on the determined weight value for each image portion of the first plurality of image portions 306A . . . 306N. The visualization frame 322A may include a heatmap visualization 322B of a region which is included in the first image frame 302A and is bounded by the bounding box coordinates 304B.”
PNG
media_image1.png
420
524
media_image1.png
Greyscale
)
Jha discloses Claim 2. The image processing device as claimed in claim 1, wherein the processing operations performed by the processor include:
presenting a detection target setting screen to the user; and in response to the user's operation on the detection target setting screen, setting the detection target designated by the user. (Jha, paragraph 48, “For example, the electronic apparatus 102 may receive a user input via the I/O device 206 to select an object of interest from multiple objects in an image frame and to initiate the XAI workflow (which includes object detection workflow and heatmap visualization workflow).”, see Fig. 5 and Fig. 8 ),
Jha discloses Claim 3. The image processing device as claimed in claim 1, wherein the processing operations performed by the processor include:
presenting a processing condition setting screen to the user; and in response to the user's operation on the processing condition setting screen, setting the image processing condition designated by the user. (Jha, paragraph 48, “For example, the electronic apparatus 102 may receive a user input via the I/O device 206 to select an object of interest from multiple objects in an image frame and to initiate the XAI workflow (which includes object detection workflow and heatmap visualization workflow).”, see Fig. 5 and Fig. 8),
Jha discloses Claim 4. The image processing device as claimed in claim 1, wherein the image processing operation includes at least one of a blurring operation, an illuminance adjusting operation, and a virtual object overlaying operation. (see claim 1, placement of heat map on top of simulated image; Note: The Examiner is treating this claim limitation at one or more of … or .., since the specification appears to treat these as alternative embodiments)
Jha discloses Claim 5. The image processing device as claimed in claim 1, wherein the processing operations performed by the processor include:
generating a tone image, in which a color tone at each part of the simulated image represents a contribution degree which is a degree to which the each part accounts for a recognition result of the image recognition operation; and overlaying the tone image as the status image on the simulated image. (Jha, paragraph 48, “For example, the electronic apparatus 102 may receive a user input via the I/O device 206 to select an object of interest from multiple objects in an image frame and to initiate the XAI workflow (which includes object detection workflow and heatmap visualization workflow).”, see Fig. 5 and Fig. 8’ where the heatmap paragraph 44 states color intensity based on color scheme of heatmap visualization),
Jha discloses Claim 6. The image processing device as claimed in claim 1, wherein the processing operations performed by the processor include:
generating a score image indicating a score that numerically expresses an accuracy of the status of recognition of the detection target in the simulated image; and (Jha, paragraph 40, “[0040] The DNN model 106 may receive the input first image frame through an input layer of the DNN model 106. In some embodiments, the input first image frame may be resized to match a size of the input layer of the DNN model 106. For example, if the first image frame is a high definition image (i.e. 1280×720 pixels) but the size of the input layer is 416×416, then the electronic apparatus 102 may resize the first image frame to match the size of the input layer. Through an output layer of the DNN model 106, the DNN model 106 may output an object detection result associated with an object of interest detected in the input first image frame. For example, if the soccer player 114 in the image 112 is detected by the DNN model 106, then the object detection result may be a vector which includes an object score for the soccer player 114, bounding box coordinates of a bounding box predicted to include the soccer player 114, and class scores (or class probabilities) for the soccer player 114. The object score may specify a confidence (in terms of a probability value between 0 and 1) of the DNN model 106 in detection of the soccer player 114 in the image 112. A higher object score may denote a higher confidence level of the DNN model 106. In some embodiments, a threshold object score (e.g., 0.95) may be set to filter out object detection results in which the confidence of the detection is below the threshold object score. The class scores may include soft labels or hard labels to represent object classes which can be detected by the DNN model 106. The object of interest may belong to one of such classes. For example, if the DNN model 106 supports three classes such as a human object class, a car object class, and a cat object class, then the class score may be 1 (or a number close to 1) for the human object class and 0 (or close to 0) for each of the car object class and the cat object class.”)
overlaying the score image as the status image on the simulated image.(Jah, paragraph 41, “then the DNN model 106 may output an object detection result for each of the different grid cells associated with the multiple objects of interest. The electronic apparatus 102 may extract the object detection result associated with the object of interest as an output of the DNN model 106 for the first image frame.”)
Jha discloses Claim 7. The image processing device as claimed in claim 1, wherein, when the user designates a plurality of detection targets, the processing operations performed by the processor include:generating the status image for each of the plurality of detection targets such that the respective status images for the plurality of detection targets are shown in a visually distinguishable manner; and overlaying the status images on the simulated image. (Jha, Fig. 9 shows detecting a plurality of objects)
Claim 8 is rejected under the same reasoning as clam 1.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GANDHI THIRUGNANAM whose telephone number is (571)270-3261. The examiner can normally be reached M-F 8:30-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GANDHI THIRUGNANAM/Primary Examiner, Art Unit 2672
Prosecution Insights