Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Preliminary Amendment
Claims 1-12, 14, 16, 18-23 are pending;
Claims 13, 15, 17 are canceled;
Claim 3 is amended.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3, 6, 12, 16, 22-23 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by US20240375613A1, hereinafter “Ryoji”.
Ryoji discloses the following as claimed (Figs. 3-5 and associated disclosure)
PNG
media_image1.png
620
866
media_image1.png
Greyscale
PNG
media_image2.png
594
824
media_image2.png
Greyscale
PNG
media_image3.png
534
808
media_image3.png
Greyscale
Regarding Claim 1, A fixed object detector (201 “Information Processing Unit”) comprising:
a capturing device (Fig. 3, 25; Fig. 4, “Image Acquisition” 241 of images from “camera” 51; pars. 0114-0117; 0131);
“[0114] The information processing unit 201 includes an AI dirt detection unit 211, an image-change dirt detection unit 212, a dirt region identification unit 213, a communication control unit 214, and a wiping control unit 215.
[0115] The AI dirt detection unit 211 inputs a captured image captured by the camera 51 of the external recognition sensor 25 to an AI dirt discriminator using a neural network, and detects dirt from the captured image in real time. In the case of detecting dirt, the AI dirt detection unit 211 acquires a dirt region using a visualization method, and supplies the dirt detection result to the dirt region identification unit 213.
[0116] The image-change dirt detection unit 212 inputs a captured image captured by the camera 51 of the external recognition sensor 25 to an image-change dirt discriminator using an optical flow, and detects dirt from the captured image. In the case of detecting dirt, the image-change dirt detection unit 212 acquires a dirt region and supplies the dirt detection result to the dirt region identification unit 213.
[0117] The dirt region identification unit 213 identifies a dirt region in the captured image on the basis of the dirt detection result by the AI dirt detection unit 211 and the dirt detection result by the image-change dirt detection unit 212, and supplies the captured image and information indicating the dirt region to the action planning unit 62, the recognition unit 73, and the wiping control unit 215. The dirt detection result by the AI dirt detection unit 211 and the image-change dirt detection unit 212 is also supplied from the dirt region identification unit 213 to the wiping control unit 215.”
“[0131] The image acquisition unit 241 acquires a captured image captured by the camera 51 and inputs the captured image to the AI dirt discriminator 242.”
a fixed-object area information obtainer that searches for a fixed object candidate that is a candidate for a fixed object appearing in a video captured by the capturing device, and obtains a fixed-object area information item indicating coordinates of the fixed object candidate in the video, the fixed object having a fixed relative position with respect to the capturing device (211-212, “AI” and “Image-Change” Dirt Detection Units; pars. 0134-0138; Note: dirt reads on “fixed-object”; dirt region reads on “fixed-object area”; optical flow method for dirt region change(s) reads on “relative position” of fixed-object relative to the camera lens; see also “Matching Unit” 261 for dirt region of Fig. 5, 213 reads on fixed-object candidate);
“[0135] The image-change dirt detection unit 212 includes an image acquisition unit 251, an image-change dirt discriminator 252, and a dirt region acquisition unit 253.
[0136] The image acquisition unit 251 acquires a captured image captured by the camera 51 and inputs the captured image to the image-change dirt discriminator 252.
[0137] The image-change dirt discriminator 252 determines whether or not there is dirt in the input captured image using an optical flow method. Specifically, the image-change dirt discriminator 252 calculates an image change amount of the captured image for a predetermined time, and determines that there is dirt in a case where a region having a small image change amount occupies a predetermined percentage or more of the captured image.
[0138] In a case where the image-change dirt discriminator 252 determines that there is dirt in the captured image, the dirt region acquisition unit 253 acquires a region having a small amount of image change in the captured image as a dirt region. The dirt region acquisition unit 253 supplies information indicating whether or not there is dirt in the captured image and information indicating a dirt region to the dirt region identification unit 213 as a dirt detection result.
[0139] FIG. 5 is a block diagram illustrating a detailed configuration example of the dirt region identification unit 213.
[0140] The dirt region identification unit 213 includes a matching unit 261, a sensor linkage unit 262, and a determination unit 263.
[0141] The matching unit 261 matches the dirt region detected by the AI dirt detection unit 211 with the dirt region detected by the image-change dirt detection unit 212, and supplies the result of matching to the determination unit 263.
[0142] The sensor linkage unit 262 links position information of the vehicle 1 acquired by the position information acquisition unit 24 and sensor data of the external recognition sensor 25 with identification of the dirt region.
[0143] For example, the sensor linkage unit 262 identifies a location such as a specific building or wall included in the angle of view of the camera 51 at the self-position of the vehicle 1 on the basis of the position information of the vehicle 1 and the sensor data of the external recognition sensor 25. The sensor linkage unit 262 acquires, from the server 203 via the communication control unit 214, a history in which the AI dirt discriminator 242 or the image-change dirt discriminator 252 erroneously detects a region showing the location as dirt. The history in which the AI dirt discriminator 242 or the image-change dirt discriminator 252 detects dirt erroneously is supplied to the determination unit 263.
[0144] The determination unit 263 identifies the dirt region in the captured image on the basis of the result of matching by the matching unit 261.”
a storage that stores therein the fixed-object area information item obtained (203; par. 0120-0121 “server”);
“[0120] The server 203 performs learning using a neural network and manages a discriminator obtained by the learning. This discriminator is an AI dirt discriminator used by the AI dirt detection unit 211 to detect dirt. The server 203 updates the AI dirt discriminator by performing relearning using a captured image transmitted from the vehicle control system 11 as learning data. Furthermore, the server 203 also manages a history in which the AI dirt discriminator or the image-change dirt discriminator erroneously detects dirt.
[0121] The communication control unit 214 acquires a history in which the AI dirt discriminator or the image-change dirt discriminator erroneously detects a region such as a building appearing in the captured image as dirt from the server 203 via the communication unit 22. This history is used by the dirt region identification unit 213 to separate the erroneous detection region from the region in the captured image detected as dirt by the AI dirt detection unit 211 or the image-change dirt detection unit 212.”
and an observer that determines the coordinates indicated by the fixed-object area information item to be coordinates of the fixed object in the video and outputs a determination result, the coordinates indicated by the fixed-object area information item matching coordinates indicated by a past fixed-object area information item stored in the storage in past (Figs. 3-4, pars. 0121, 0129-0132 “AI Dirt Discriminator” 241 is an inference model).
“[0121] The communication control unit 214 acquires a history in which the AI dirt discriminator or the image-change dirt discriminator erroneously detects a region such as a building appearing in the captured image as dirt from the server 203 via the communication unit 22. This history is used by the dirt region identification unit 213 to separate the erroneous detection region from the region in the captured image detected as dirt by the AI dirt detection unit 211 or the image-change dirt detection unit 212.
[0129] FIG. 4 is a block diagram illustrating a detailed configuration example of the AI dirt detection unit 211 and the image-change dirt detection unit 212.
[0130] The AI dirt detection unit 211 includes an image acquisition unit 241, an AI dirt discriminator 242, and a dirt region acquisition unit 243.
[0131] The image acquisition unit 241 acquires a captured image captured by the camera 51 and inputs the captured image to the AI dirt discriminator 242.
[0132] The AI dirt discriminator 242 is an inference model that determines whether or not there is dirt in the captured image input to the neural network in real time. The AI dirt discriminator 242 is acquired from the server 203 at a predetermined timing and used in the AI dirt detection unit 211.”.
Regarding claim 3, The fixed object detector according to claim 1, further comprising: a fixed object notification device that notifies a user of the determination result output by the observer (Fig. 3, 213 reads on this aspect as the output is communicated to communication unit 22 to the server 203 wherein human machine interface is facilitated e.g., Fig. 1, 31, 61 the human machine interface (HMI) 31 will obtain “notification” from the result of the analysis unit 61 via communication BUS 41, detailed of which is reflected in Fig. 3 as explained).
Regarding claim 6, The fixed object detector according to claim 1, wherein the observer infers the coordinates of the fixed object in the video, based on the past fixed-object area information item stored in the storage, and outputs an inference result as the determination result (Rejection of claim 1 is incorporated herein. See also pars. 0120-0121 i.e., the learning from neural network to discriminate and detect dirt i.e., fixed-object based on a history of dirt discrimination from captured images reads on past fixed-object area information).
Regarding claim 12, The fixed object detector according to claim 1, further comprising: an image quality adjusting device that receives input of an original video captured by the capturing device, and outputs a video having image quality adjusted to increase a probability of finding one or more fixed object candidates that include the fixed object candidate (par. 0133, a visualization method such as Grad-CAM to enhance dirt detection during image acquisition reads on this aspect).
“[0133] In a case where the AI dirt discriminator 242 determines that there is dirt in the captured image, the dirt region acquisition unit 243 acquires the basis for determining that there is dirt using a visualization method. For example, by using a technology called Grad-CAM, a heat map indicating the basis for determining that there is dirt is acquired.”
Regarding claim 16, The fixed object detector according to claim 1, wherein the fixed object is a raindrop on a light-transmitting member, the raindrop being within a capturing area of the capturing device, and the fixed object candidate is a candidate for the raindrop (Rejection of claim 1 is incorporated herein. See also Fig. 5, 25 “External Recognition Sensor”, pars. 0002, 0058 wherein raindrop is also considered fixed-object besides dirt).
Regarding claim 22, A fixed object detection method executed with use of a computer, the fixed object detection method comprising: searching for a fixed object candidate that is a candidate for a fixed object appearing in a video captured by a capturing device, and obtaining a fixed-object area information item indicating coordinates of the fixed object candidate in the video, the fixed object having a fixed relative position with respect to the capturing device; storing the fixed-object area information item obtained; and determining that the coordinates indicated by the fixed-object area information item to be coordinates of the fixed object in the video and outputting a determination result, the coordinates indicated by the fixed-object area information item matching coordinates indicated by a past fixed-object area information item stored in past (Rejection of claim 1 is incorporated herein. This is a method claim corresponding to system claim 1).
Regarding claim 23, A non-transitory computer-readable recording medium having recorded thereon a program for causing the computer to execute the fixed object detection method according to claim 22 (Rejection of claim 1 is incorporated herein 1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 18 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over US20240375613A1, hereinafter “Ryoji”.
Regarding claim 18, The fixed object detector according to claim 1, wherein the fixed object is a defect that includes at least one of a crack or a scratch generated in a light-transmitting member, the at least one of the crack or the scratch being within a capturing area of the capturing device, and the fixed object candidate is a candidate for the defect. (The rejection of claim 1 is incorporated herein. Ryoji discloses “dirt, raindrops, and the like adhere to the lens of the camera”, par. 0002. A crack or a scratch as claimed is not expressly stated. However, this aspect is not inventive and are merely known “fixed-object(s)” that could appear on the camera lens. Ryoji teaches “and the like”, thus, would have encompassed these possibilities. Nothing in the claim recited additional aspects that are uniquely unexpected and/or deviated from the norm of detecting and discriminator a “fixed-object” adhering on the camera lens. Therefore, the recited crack and/or scratch are merely simple substitution not deviating from the manner of detecting “dirt” as fixed-object disclosed in Ryoji. The result would have been the same, obvious, and predictable. See MPEP 2143(I) on KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007).
Regarding claim 21, The fixed object detector according to claim 1, further comprising: a reducer that reduces the video (rejected in claim 1 as dirt region); a feature detector that detects a feature from a reduced video resulting from the reducer reducing the video (rejected in claim 1 as dirt being discriminated and detected within the dirt region); a feature storage that stores therein one or more features; a cumulative adder that adds the feature detected by the feature detector and a prestored feature that is stored in advance in the feature storage, and stores a result of adding the feature detected and the prestored feature into the feature storage as an added feature (rejected in claim 1 as the inference model using neural network; Note: inference model via neural network inherently covers some aspects recited here. As for the adding/cumulating detected features to prestored features, this is simply the learning phase of the inference model of the neural network, and thus, common knowledge and implied); a binarizer that binarizes the added feature stored (see Fig. 9A-B, pars. 0175-0177); and a straight line detector that detects, from data of the added feature binarized, coordinates of the fixed object candidate in a straight shape in the video, by using Hough transform that is a transform algorithm. (Ryoji does not expressly disclose using a Hough transform to detect straight line of the binarized added feature. However, Hough transform is well known in the art for such purposes as detecting straight lines and geometric shapes by transforming image objects into parametric objects. Clearly, Fig. 9 depicts binarized straight shapes of detected dirt features which would have implied some types of transformation. Thus, the utilization of Hough transform to detect dirt feature as fixed-object from acquired images is not unique nor exclusive to the applicant and would have been obvious to incorporate with predictable result. MPEP 2143(I) on KSR Int'l Co. v. Teleflex Inc., 550 U.S. 398, 415-421, 82 USPQ2d 1385, 1395-97 (2007)).
Allowable Subject Matter
Claims 2, 4-5, 7-11, 14, 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VU LE whose telephone number is (571)272-7332. The examiner can normally be reached M-F 8:00 - 17:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Vu Le can be reached at 2-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights