Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Submission dated 05/26/2026 amends claims 1, 2, 10 and 11. Claims 1-18 and 21-22 are pending.
In view of the amendment, the previously set forth claim objections have been withdrawn.
Response to Arguments
On pages 11-16, the applicant argues that Hirai in view of Hu does not teach or suggest “setting the first data and the second data as first input data of a target network” because Hirai’s neural network receives only the second set of sub-images. The Applicant disagrees because the second set of sub-images correspond to both the first and second data. As described in pars. 42 and 51 of Hirai, the second set of sub-images is generated by cropping sub-images of the first set and hence, includes portions of the sub-images in the first set. As such, when the neural network receives the second set, it can be stated under the broadest reasonable interpretation that the received data corresponds to both the first and second sets. The examiner notes that the claim language does not require the first and second set to separately received by the neural network. Instead, the claim language requires the first and second data to be input as one input data set, i.e., the first input data. For the aforementioned reasons, the examiner finds the argument unpersuasive.
On pages 16-17, the applicant argues about the amended claim language in view of Hirai and Hu. As shown below, the examiner does not apply Hirai or Hu for the amended claim language. As such, the examiner finds the argument moot.
Claim Objections
Claims 1 and 10 are objected to because of the following informalities:
Claim 1 recites the limitation “the first enhancement” in line 9. The examiner suggests adding “information” to the end of the limitation to give the limitation sufficient antecedent basis. Appropriate correction is required.
Claim 10 recites the limitation “the first enhancement” in line 12. The examiner suggests adding “information” to the end of the limitation to give the limitation sufficient antecedent basis. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 10, 12-13, and 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Us patent application publication no. 2021/0097277 to Hirai et al. (hereinafter Hirai) in view of us patent application publication no. 2023/0080693 to Hu et al. (hereinafter Hu).
For claim 1, Hirai as applied teaches a method comprising:
obtaining first frame data of one frame in raw data of an image sensor (see, e.g., pars. 13, 17, 21, 31-32, and 47 and FIGS. 1, 3A and 4, which teach obtaining a first image from an image capturing device and executing a bounding box operation);
obtaining, from the first frame data, first data corresponding to a compact box, wherein the compact box comprises a target object in the first frame data (see, e.g., pars. 42 and 51 and FIGS. 3B and 4, which teach obtaining a second set of sub-image by cropping the first set of sub-images);
obtaining, from the first frame data, second data corresponding to a loose box, wherein the loose box comprises and is larger than the compact box (see, e.g., pars. 40-41 and 51 and FIGS. 3B and 4, which teach obtaining a first set of sub-images, wherein the first set of sub-images are larger than the second set of sub-images);
setting the first data and the second data as first input data of a target network (see, e.g., pars. 42-44 and FIG. 3B, which teach forming batches of the second sets of the sub-images as input data of the neural network model);
obtaining, using the target network, an output image (see, e.g., par. 52 and FIGS. 3B and 4, which teach obtaining images of the objects using the neural network model and the input sub-images).
While Hirai as applied does not explicitly teach, Hu in the analogous art teaches extracting, using the target network, first enhancement information and second enhancement information, wherein the first enhancement is about a luminance channel in the first input data, and wherein the second enhancement information is about a plurality of channels in the first input data (see, e.g., pars. 66-72, 123-124, 157-161, 191-195, and 203-207 and FIGS. 2, 3B, 3C, 3G, and 4A-B of Hu, which teach extracting, using an image denoising/tone adjustment model, brightness/Y channel information and multiple color/UV channel information of an input image); and
obtaining, using the target network, an output image by fusing the first enhancement information and the second enhancement information (see, e.g., pars. 66-72, 126 and 208 and FIGS. 2, 3B, 3C, 3G and 4A-B of Hu, which teach obtaining, using the image denoising model, an output image by fusing/concatenating the brightness channel information and multiple color channel information).”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hirai to extract and use the information about the image in obtaining in object recognition/detection image as taught by Hu because doing so would enhance the image quality and improve the user’ experience by improving the denoising performance and the tone of the image (see, e.g., pars. 88, 153 and 162 of Hu).
For claim 10, Hirai as applied teaches an apparatus (see, e.g., FIG. 1) comprising:
a memory configured to store instructions (see, e.g., pars. 20 and 23-27 and FIG. 2); and
a processor coupled to the memory (see, e.g., pars. 20 and 26-27 and FIG. 2) and configured to execute the instructions to cause the apparatus to:
obtain first frame data of one frame in raw data of an image sensor (see, e.g., pars. 13, 17, 21, 31-32, and 47 and FIGS. 1, 3A and 4, which teach obtaining a first image from an image capturing device and executing a bounding box operation);
obtain, from the first frame data, first data corresponding to a compact bo(see, e.g., pars. 42 and 51 and FIGS. 3B and 4, which teach obtaining a second set of sub-image by cropping the first set of sub-images);
obtain, from the first frame data, second data corresponding to a loose bo(see, e.g., pars. 40-41 and 51 and FIGS. 3B and 4, which teach obtaining a first set of sub-images, wherein the first set of sub-images are larger than the second set of sub-images); and
set the first data and the second data as first input data of a target networ(see, e.g., pars. 42-44 and FIG. 3B, which teach forming batches of the second sets of the sub-images as input data of the neural network model)
obtaining, using the target network , an output image (see, e.g., par. 52 and FIGS. 3B and 4, which teach obtaining images of the objects using the neural network model and the input sub-images).
While Hirai as applied does not explicitly teach, Hu in the analogous art teaches extracting, using the target network, first enhancement information and second enhancement information, wherein the first enhancement is about a luminance channel in the first input data, and wherein the second enhancement information is about a plurality of channels in the first input data (see, e.g., pars. 66-72, 123-124, 157-161, 191-195, and 203-207 and FIGS. 2, 3B, 3C, 3G, and 4A-B of Hu, which teach extracting, using an image denoising/tone adjustment model, brightness/Y channel information and multiple color/UV channel information of an input image); and
obtaining, using the target network, an output image by fusing the first enhancement information and the second enhancement information (see, e.g., pars. 66-72, 126 and 208 and FIGS. 2, 3B, 3C, 3G and 4A-B of Hu, which teach obtaining, using the image denoising model, an output image by fusing/concatenating the brightness channel information and multiple color channel information).”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hirai to extract and use the information about the image in obtaining in object recognition/detection image as taught by Hu because doing so would enhance the image quality and improve the user’ experience by improving the denoising performance and the tone of the image (see, e.g., pars. 88, 153 and 162 of Hu).
For claims 3 and 12, Hirai in view of Hu teaches:
performing a target detection on the first frame data to obtain position information of the target object in the first frame data (see, e.g., pars. 39 and 50 and FIGS. 3B and 4 of Hirai, which teach performing a region detection operation to obtain the detected region corresponding to the plurality of objects); and
generating, based on the position information, the compact box and the loose box (see, e.g., pars. 40-42 and 51 and FIGS. 3B and 4 of Hirai, which teach obtaining a first and second sets of sub-image).
For claims 4 and 13, Hirai in view of Hu teaches that obtaining the first frame data comprises:
receiving user input data (see, e.g., pars. 13, 17 and 21 and FIG. 1 of Hirai, which teach controlling the image capturing device to capture the first image; the examiner interprets the controlling to suggest receiving a user input); and
extracting, from the raw data based on the user input data, the first frame data (see, e.g., pars. 13, 17 and 21 and FIG. 1 of Hirai, which teach obtaining a first image from an image capturing device).
For claims 21 and 22, Hirai in view of Hu teaches obtaining the first frame data comprises:
performing a target detection on each frame in the raw data to obtain a detection result (see, e.g., pars. 32 and FIGS. 3A and 4 of Hirai, which teach detecting bounding boxes from the captured first image); and
extracting, from the raw data based on the detection result, the first frame data (see, e.g., pars.
33-38 and 49 and FIGS. 3A and 4 of Hirai, which teach performing a probability map determination operation on the detected bounding boxes of the first image).
Claim(s) 5 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirai in view of Hu and further in view of us patent application publication no. 2018/0174046 to Xiao et al. (hereinafter Xiao).
For claims 5 and 14, Hirai in view of Hu as applied teaches:
training, based on a recognition network and a training set, the target network (see, e.g., pars. 42-44 and FIGS. 3B and 4 of Hirai, which teach training one part of the neural network model to perform the object detection operation based on another part of the neural network model that performs the bounding box detection operation); and
setting, while training the target network, an output result of the recognition network as a constraint to update the target network, wherein the output result comprises semantic information in an input image (see, e.g., pars. 32 and 48 and FIGS. 3A and 4 of Hirai, which teach detecting the bounding box which is used by the neural network model to detect object; the examiner interprets the information about the detected bounding box to teach the claimed semantic information of the input image because it relates to the position of the object and par. 124 of the specification includes the position of the object as the semantic information).
Hirai as applied does not explicitly teach setting, while training the target network, an output result of the recognition network as a constraint to update the target network, wherein the output result comprises semantic information in an input image. Xiao in the analogous art teaches training, based on the first neural network and the training image, the second neural network (see, e.g., pars. 75 and 79 and FIG. 3 of Xiao) and setting, while training the second neural network, the output of the first neural network as a constraint to train and update the second neural network (see, e.g., pars. 75 and 79 and FIG. 3 of Xiao), wherein the output includes the heatmap information corresponding to the position information of the object (see, e.g., pars. 75 and 79 and FIG. 3 of Xiao; the position information is semantic information according to par. 124 of the specification).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Hirai in view of Hu to train its neural network as taught by Xiao because doing so would allowing the target detection with a high accuracy (see pars. 91 and 99 of Xiao).
Allowable Subject Matter
Claims 2 and 11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regard to claims 2 and 11, when considering each claim as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“setting the first data as second input data of a first network of the target network to obtain first enhancement information comprising second information about a luminance channel in the first input data; and
setting the second data as third input data of a second network of the target network to obtain second enhancement information comprising the first information and
wherein obtaining the output image comprises fusing the first enhancement information and the second enhancement information.”
Claims 6-9 and 15-18 are allowed.
In regard to claims 6 and 15, when considering each claim as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“…
obtaining a training set comprising raw data of an image sensor and comprising a corresponding truth value tag;
obtaining a training set comprising raw data of an image sensor and comprising a corresponding truth value tag;
setting the training set as first input data of a target network;
extracting, using the target network and from the first input data, first information about a luminance channel corresponding to a compact box;
extracting, using the target network and from the first input data, second information about first channels corresponding to a loose box, wherein the loose box comprises and is larger than the compact box;
fusing the first information and the second information to obtain an enhancement result;
setting the training set as second input data of a recognition network to obtain a first recognition result;
setting the enhancement result as third input data of the recognition network to obtain a second recognition result; and
updating, based on a first difference between the enhancement result and the corresponding truth value tag and a second difference between the first recognition result and the second recognition result, the target network to obtain an updated target network.”
In regard to claims 7-9 and 16-18, they are allowed for their dependencies to claims 6 and 15.
Additional Citations
The following table lists several references that are relevant to the subject matter claimed and disclosed in this Application. The references are not relied on by the Examiner, but are provided to assist the Applicant in responding to this Office action.
Citation
Relevance
Egilmez (us pat. app. pub. No. 2022/0191523)
Techniques are described herein for processing video data using a neural network system. For instance, a process can include generating, by a first convolutional layer of an encoder sub-network of the neural network system, output values associated with a luminance channel of a frame. The process can include generating, by a second convolutional layer of the encoder sub-network, output values associated with at least one chrominance channel of the frame. The process can include generating, by a third convolutional layer based on the output values associated with the luminance channel of the frame and the output values associated with the at least one chrominance channel of the frame, a combined representation of the frame. The process can further include generating encoded video data based on the combined representation of the frame.
Table 1
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See form 892 and Table 1.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WOO RHIM whose telephone number is (571)272-6560. The examiner can normally be reached Mon - Fri 9:30 am - 6:00 pm et.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at 571-272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WOO C RHIM/Examiner, Art Unit 2676
Prosecution Insights