Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/31/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Preliminary Amendment
Preliminary amendment dated 04/01/2024 amends claims 1-18, cancels claims 19-20 and adds claims 21-22. As such, claims 1-18 and 21-22 are pending.
Claim Objections
Claim 10 is objected to because of the following informalities:
Claims 10 recites “extracting” in line 18. It should recite “extract” to be grammatically correct. Appropriate correction is required.
Claims 10 recites “obtaining” in line 20. It should recite “obtain” to be grammatically correct. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 10, 12-13, and 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Us patent application publication no. 2021/0097277 to Hirai et al. (hereinafter Hirai) in view of us patent application publication no. 2020/0005051 to Chen et al. (hereinafter Chen).
For claim 1, Hirai as applied teaches a method comprising:
obtaining first frame data of one frame in raw data of an image sensor (see, e.g., pars. 13, 17, 21, 31-32, and 47 and FIGS. 1, 3A and 4, which teach obtaining a first image from an image capturing device and executing a bounding box operation);
obtaining, from the first frame data, first data corresponding to a compact box, wherein the compact box comprises a target object in the first frame data (see, e.g., pars. 42 and 51 and FIGS. 3B and 4, which teach obtaining a second set of sub-image by cropping the first set of sub-images);
obtaining, from the first frame data, second data corresponding to a loose box, wherein the loose box comprises and is larger than the compact box (see, e.g., pars. 40-41 and 51 and FIGS. 3B and 4, which teach obtaining a first set of sub-images, wherein the first set of sub-images are larger than the second set of sub-images);
setting the first data and the second data as first input data of a target network (see, e.g., pars. 42-44 and FIG. 3B, which teach forming batches of the second sets of the sub-images as input data of the neural network model);
obtaining, using the target network and based on the first information, an output image (see, e.g., par. 52 and FIGS. 3B and 4, which teach obtaining images of the objects using the neural network model and the input sub-images).
Hirai as applied does not explicitly teach extracting, using the target network, information about channels in the image and using the extracted information to obtain output images. Chen in the analogous art teaches extracting multi-channel feature information from an image using a neural network model (see, e.g., abstract, pars. 61-62 and FIG. 1 of Chen) and using the extracted information to obtain the target recognition object image (see, e.g., abstract and pars. 63-72 and FIG. 1 of Chen).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hirai to extract and use the information about the image in obtaining in object recognition/detection image as taught by Chen because doing so would eliminate redundant image information from consideration, increasing speed of target recognition and improving sensitivity of an autonomous vehicle (see, e.g., abstract and pars. 47 and 72 of Chen).
For claim 10, Hirai as applied teaches an apparatus (see, e.g., FIG. 1) comprising:
a memory configured to store instructions (see, e.g., pars. 20 and 23-27 and FIG. 2); and
a processor coupled to the memory (see, e.g., pars. 20 and 26-27 and FIG. 2) and configured to execute the instructions to cause the apparatus to:
obtain first frame data of one frame in raw data of an image sensor (see, e.g., pars. 13, 17, 21, 31-32, and 47 and FIGS. 1, 3A and 4, which teach obtaining a first image from an image capturing device and executing a bounding box operation);
obtain, from the first frame data, first data corresponding to a compact boin the first frame data (see, e.g., pars. 42 and 51 and FIGS. 3B and 4, which teach obtaining a second set of sub-image by cropping the first set of sub-images);
obtain, from the first frame data, second data corresponding to a loose bo(see, e.g., pars. 40-41 and 51 and FIGS. 3B and 4, which teach obtaining a first set of sub-images, wherein the first set of sub-images are larger than the second set of sub-images); and
set the first data and the second data as first input data of a target networ(see, e.g., pars. 42-44 and FIG. 3B, which teach forming batches of the second sets of the sub-images as input data of the neural network model)
obtaining, using the target network based on the first information, an output image (see, e.g., par. 52 and FIGS. 3B and 4, which teach obtaining images of the objects using the neural network model and the input sub-images).
Hirai as applied does not explicitly teach extracting, using the target network, information about channels in the image and using the extracted information to obtain output images. Chen in the analogous art teaches extracting multi-channel feature information from an image using a neural network model (see, e.g., abstract, pars. 61-62 and FIG. 1 of Chen) and using the extracted information to obtain the target recognition object image (see, e.g., abstract and pars. 63-72 and FIG. 1 of Chen).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hirai to extract and use the information about the image in obtaining in object recognition/detection image as taught by Chen because doing so would eliminate redundant image information from consideration, increasing speed of target recognition and improving sensitivity of an autonomous vehicle (see, e.g., abstract and pars. 47 and 72 of Chen).
For claims 3 and 12, Hirai in view of Chen teaches:
performing a target detection on the first frame data to obtain position information of the target object in the first frame data (see, e.g., pars. 39 and 50 and FIGS. 3B and 4 of Hirai, which teach performing a region detection operation to obtain the detected region corresponding to the plurality of objects); and
generating, based on the position information, the compact box and the loose box (see, e.g., pars. 40-42 and 51 and FIGS. 3B and 4 of Hirai, which teach obtaining a first and second sets of sub-image).
For claims 4 and 13, Hirai in view of Chen teaches that obtaining the first frame data comprises:
receiving user input data (see, e.g., pars. 13, 17 and 21 and FIG. 1 of Hirai, which teach controlling the image capturing device to capture the first image; the examiner interprets the controlling to suggest receiving a user input); and
extracting, from the raw data based on the user input data, the first frame data (see, e.g., pars. 13, 17 and 21 and FIG. 1 of Hirai, which teach obtaining a first image from an image capturing device).
For claims 21 and 22, Hirai in view of Chen teaches obtaining the first frame data comprises:
performing a target detection on each frame in the raw data to obtain a detection result (see, e.g., pars. 32 and FIGS. 3A and 4 of Hirai, which teach detecting bounding boxes from the captured first image); and
extracting, from the raw data based on the detection result, the first frame data (see, e.g., pars.
33-38 and 49 and FIGS. 3A and 4 of Hirai, which teach performing a probability map determination operation on the detected bounding boxes of the first image).
Claim(s) 5 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirai in view of Chen and further in view of us patent application publication no. 2018/0174046 to Xiao et al. (hereinafter Xiao).
For claims 5 and 14, Hirai as applied teaches:
training, based on a recognition network and a training set, the target network (see, e.g., pars. 42-44 and FIGS. 3B and 4, which teach training one part of the neural network model to perform the object detection operation based on another part of the neural network model that performs the bounding box detection operation); and
setting, while training the target network, an output result of the recognition network as a constraint to update the target network, wherein the output result comprises semantic information in an input image (see, e.g., pars. 32 and 48 and FIGS. 3A and 4, which teach detecting the bounding box which is used by the neural network model to detect object; the examiner interprets the information about the detected bounding box to teach the claimed semantic information of the input image because it relates to the position of the object and par. 124 of the specification includes the position of the object as the semantic information).
Hirai as applied does not explicitly teach setting, while training the target network, an output result of the recognition network as a constraint to update the target network, wherein the output result comprises semantic information in an input image. Xiao in the analogous art teaches training, based on the first neural network and the training image, the second neural network (see, e.g., pars. 75 and 79 and FIG. 3 of Xiao) and setting, while training the second neural network, the output of the first neural network as a constraint to train and update the second neural network (see, e.g., pars. 75 and 79 and FIG. 3 of Xiao), wherein the output includes the heatmap information corresponding to the position information of the object (see, e.g., pars. 75 and 79 and FIG. 3 of Xiao; the position information is semantic information according to par. 124 of the specification).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hirai in view of Chen to train its neural network as taught by Xiao because doing so would allowing the target detection with a high accuracy (see pars. 91 and 99 of Xiao).
Allowable Subject Matter
Claims 2 and 11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regard to claims 2 and 11, when considering each claim as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“setting the first data as second input data of a first network of the target network to obtain first enhancement information comprising second information about a luminance channel in the first input data; and
setting the second data as third input data of a second network of the target network to obtain second enhancement information comprising the first information and
wherein obtaining the output image comprises fusing the first enhancement information and the second enhancement information.”
Claims 6-9 and 15-18 are allowed.
In regard to claims 6 and 15, when considering each claim as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“…
obtaining a training set comprising raw data of an image sensor and comprising a corresponding truth value tag;
obtaining a training set comprising raw data of an image sensor and comprising a corresponding truth value tag;
setting the training set as first input data of a target network;
extracting, using the target network and from the first input data, first information about a luminance channel corresponding to a compact box;
extracting, using the target network and from the first input data, second information about first channels corresponding to a loose box, wherein the loose box comprises and is larger than the compact box;
fusing the first information and the second information to obtain an enhancement result;
setting the training set as second input data of a recognition network to obtain a first recognition result;
setting the enhancement result as third input data of the recognition network to obtain a second recognition result; and
updating, based on a first difference between the enhancement result and the corresponding truth value tag and a second difference between the first recognition result and the second recognition result, the target network to obtain an updated target network.”
In regard to claims 7-9 and 16-18, they are allowed for their dependencies to claims 6 and 15.
Additional Citations
The following table lists several references that are relevant to the subject matter claimed and disclosed in this Application. The references are not relied on by the Examiner, but are provided to assist the Applicant in responding to this Office action.
Citation
Relevance
Yumbe (us pat. app. pub. No. 2021/0204794)
Describes a medical image processing apparatus, an endoscope system, and a method for operating the medical image processing apparatus that are for detecting a region of interest such as a lesion portion. In one embodiment, a processor device includes an image signal acquiring unit, an image processing unit, and a display control unit. The image signal acquiring unit acquires a digital image signal corresponding to an observation mode from an endoscope. The image processing unit includes a region-of-interest-detection-mode image processing unit. The display control unit sets an emphasized region having a larger area than a region of interest and including the region of interest, displays the emphasized region in a manner of emphasized display, and determines whether or not to change setting of the emphasized region in accordance with an amount of variation of the region of interest.
Park et al. (us pat. app. pub. No. 2021/0350145)
Describes a method of recognizing a neighboring object during autonomous driving and an autonomous driving device using the method. One embodiment discloses an object recognition method including: obtaining a first RGB image by using a camera; predicting at least one first region, in which an object is unrecognizable, in the first RGB image based on brightness information of the first RGB image; determining at least one second region, in which an object exists, from among the at least one first region, based on object information obtained through a dynamic vision sensor; obtaining an enhanced second RGB image by controlling photographic configuration information of the camera in relation to the at least one second region; and recognizing the object in the second RGB image.
Ng et al. (us pat. app. pub. No. 2023/0334637)
Describes fusion of two images (e.g., a near infrared (NIR) image and an RGB image) that are captured simultaneously in a scene. A computer system extracts a first luminance component and a first color component from the first image, and extracts a second luminance component from the second image. An infrared emission strength is determined based on the first and second luminance components. The computer system combines the first and second luminance components based on the infrared emission strength to obtain a combined luminance component. The combined luminance component is combined with the first color component to obtain a fused image.
Table 1
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See Table 1 and form 892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WOO RHIM whose telephone number is (571)272-6560. The examiner can normally be reached Mon - Fri 9:30 am - 6:00 pm et.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at 571-272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WOO C RHIM/Examiner, Art Unit 2676