DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This communication is in response to the Application Filed on 03/12/2024
Claims 1–20 are pending in this application.
Drawings
The drawing(s) filed on 03/12/2024 are accepted by the Examiner.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/12/2024, and 01/23/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1–20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental process (concept performed in a human mind, including as observation, evaluation, judgment, opinion, organizing human activity and mathematical concepts and calculations). The independent claim(s) 1, 11 and 18 recite(s) a method, electronic device and a method, respectively. This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved .The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally and no additional features in the claims would preclude them from being performed as such except for the generic computer elements at high level of generality (i.e., processor, memory).
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that the independent claims 1, 11 and 18 are directed to an abstract idea as shown below:
STEP 1: Do the claims fall within one of the statutory categories? YES. Independent claims 1, 11 and 18 are directed to a method, electronic device and a method, respectively.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea? YES, the claims are directed toward a mental process (i.e. abstract idea).
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
Independent claims 1, 11 and 18 comprise a mental process that can be practicably performed in the human mind (or generic computers or components configured to perform the method) and, therefore, an abstract idea.
Regarding independent claim(s) 1: the limitations recite:
An operating method of an object recognition model configured to recognize traffic light objects, the operating method comprising:
obtaining an input image from a camera comprised in a vehicle, the input image among frames, including previous frames, captured by the camera (data gathering and data preparation);
estimating, based on prior information about traffic light objects, a first region of interest (RoI) for the input image (mental process including observation and evaluation);
determining a second RoI based on the first RoI and based on detection results of the previous frames, wherein the detection of results correspond to recognition results of recognizing traffic lights in the previous frames by the object recognition model (mental process including observation and evaluation); and
recognizing, by the object recognition model, a traffic light in the input image, wherein the recognizing is based on the input image and the second RoI (mental process including observation and evaluation).
Regarding independent claim(s) 11: the limitations recite:
An electronic device comprising:
a camera (generic computer component);
one or more processors (generic computer component); and
a memory storing instructions configured to cause the one or more processors (generic computer component) to:
obtain an input image from the camera (data gathering and data preparation);
estimate, based on prior information about traffic light objects, a first region of interest (RoI) for the input image (mental process including observation and evaluation);
determine a second RoI based on the first RoI and based on detection results of previous frames of the camera, wherein the detection of results of the previous frames correspond to recognitions results of recognizing traffic lights in the previous frames by an object recognition model configured to recognize traffic lights (mental process including observation and evaluation); and
recognize, by the object recognition model, a traffic light in the input image, wherein the recognizing is based on the input image and the second RoI (mental process including observation and evaluation).
Regarding independent claim(s) 18: the limitations recite:
A method performed by a computing device, the method comprising:
receiving input images captured by a camera of a moving vehicle (data gathering and data preparation);
for a first of the input images, determining a first region of interest (RoI), wherein the first RoI is determined based on historical traffic light observations (mental process including observation and evaluation);
inputting, to an object recognition model configured to recognize traffic lights, the first input image and an enlargement of a region in the first input image defined by the first RoI (data preparation and data gathering), based on which the object recognition model infers a location of a first traffic light in the first input image (mental process including observation and evaluation);
for a second of the input images captured after the first input image, determining a second RoI, wherein the second RoI is determined based on the location of the traffic light in the first input image (mental process including observation and evaluation); and
inputting, to the object recognition model, the second input image and an enlargement of a region in the second input image defined by the second RoI (data preparation and data gathering), based on which the object recognition model infers a location, in the second input image, of the first traffic light or a second traffic light (mental process including observation and evaluation).
These limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human. The Examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that “can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same).
As such, a person could mentally observe an image and determine a point of interest by observing the image and by using prior information like color, shape and size. The mere nominal recitation that the various steps are being executed by a processor, memory, and camera does not take the limitations out of the mental process grouping. Thus, the claims recite a mental process.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Independent claims 1, 11 and 18 do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application. Independent claims 1, 11 and 18 discloses an electronic device with a processor, memory and camera which are generic computer components and/or insignificant pre/post-solution extra activity that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a method.
These limitations are recited at a high level of generality (i.e. as a general action or change being taken based on the results of the acquiring step) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity. Further, the claims are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No, the claims do not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Independent claim(s) 1, 11 and 18 do not recite any additional elements that are not well-understood, routine or conventional. The use of a generic computer elements are routine, well-understood and conventional process that is performed by computers.
Thus, since independent claims 1, 11 and 18 are: (a) directed toward an abstract idea, (b) do not recite additional elements that integrate the judicial exception into a practical application, and (c) do not recite additional elements that amount to significantly more than the judicial exception, it is clear that independent claims 1, 11 and 18 are not eligible subject matter under 35 U.S.C 101.
Regarding claim 2–5 and 12: the additional limitations do not integrate the mental process into practical application or add significantly more to the mental process. They are mere recitations of data gathering and data preparation.
Regarding claim 6–11, 13–17 and 19–20: the additional limitations do not integrate the mental process into practical application or add significantly more to the mental process. They are mere recitation of a mental process.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claim(s) 1, 2, 4–7, 11, 12, 14 and 18–20 are rejected under 35 U.S.C. 103 as being unpatentable over Bush et al. (US 20230068046 A1, hereafter, "Bush") in view of Zhou et al. (US 20210383120 A1, hereafter, "Zhou").
Regarding claim 1, Bush teaches An operating method of an object recognition model configured to recognize traffic light objects (See Bush, [Abstract], The systems and methods receive perception data from a sensor system included in the vehicle, determine a focused Region Of Interest (ROI) in the perception data, scale the perception data of the at least one focused ROI, process the scaled perception data of the focused ROI using a neural network (NN)-based traffic object detection algorithm to provide traffic object detection data, and control at least one vehicle feature based, in part, on the traffic object detection data), the operating method comprising:
obtaining an input image from a camera comprised in a vehicle, the input image among frames, including previous frames, captured by the camera (See Bush, ¶ [0040], The optical cameras 140a-140n are mounted on the vehicle 10 and are arranged for capturing images (e.g. a sequence of images in the form of a video) of an environment surrounding the vehicle 10. Note: Sequence of images implies a current and previous frame);
estimating, based on prior information about traffic light objects, a first region of interest (RoI) for the input image (See Bush, ¶ [0054], The focus area (ROI) determination sub-module 312 serves as an attention pointer identifying ROIs in the perception data 304. In order to determine the ROIs, the focus area (ROI) determination sub-module 312 may receive localization data 316 from the vehicle localization module 306 defining a three-dimensional location of the vehicle 10. Further, focus area (ROI) determination sub-module 312 receives map data 318 from the maps 302 that defines, inter-alia, a road network reflecting roads in the real world and traffic objects, ..., In the exemplary embodiment, traffic object dimensions data 322 is provided as an input to the focus area (ROI) determination sub-module 312 to be used as the known dimensions. The traffic object dimensions data 322 can include dimensions of traffic lights, road signs, junction outlets, etc. as just some examples of traffic objects. The focus area (ROI) determination sub-module 312 outputs ROI data 320 defining, in image (or other perception data) space, the size and location of ROIs found by the focus area (ROI) determination sub-module 312. Note: Examiner is interpreting the prior information as the dimension data and the localization and map data);
[determining a second RoI based on the first RoI and based on detection results of the previous frames, wherein the detection of results correspond to recognition results of recognizing traffic lights in the previous frames by the object recognition model]; and
recognizing, by the object recognition model, a traffic light in the input image, wherein the recognizing is based on the input image and the second RoI (See Bush, ¶ [0034], The traffic object detection system 200 implements methods with automatically focused and scaled ROIs to sense distant traffic objects including road signs and Traffic Signal Devices (TSDs). Note: Examiner is interpreting the Traffic Signal Devices as Traffic Lights. ¶ [0054], The focus area (ROI) determination sub-module 312 outputs ROI data 320 defining, in image (or other perception data) space, the size and location of ROIs found by the focus area (ROI) determination sub-module 312. The ROI data 320 may include one or more bounding boxes defining a region in the perception data 304 that should be the focus of scaling and further processing by the traffic object detection module 308).
However, Bush fail(s) to teach determining a second RoI based on the first RoI and based on detection results of the previous frames, wherein the detection of results correspond to recognition results of recognizing traffic lights in the previous frames by the object recognition model.
Zhou, working in the same field of endeavor, teaches: determining a second RoI based on the first RoI and based on detection results of the previous frames, wherein the detection of results correspond to recognition results of recognizing traffic lights in the previous frames by the object recognition model (See Zhou, ¶ [0027], S102A, detecting a region of interest (ROI) in the current to-be-processed frame, in response to determining that the current to-be-processed frame is a detection picture frame, to determine at least one ROI in the current to-be-processed frame. ¶ [0028], S103A, updating a to-be-tracked ROI, based on the ROI in the current to-be-processed frame and a tracking result determined by a pre-order tracking picture frame. Note: Examiner is interpreting the pre-order tracking picture frame as the previous frame).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Bush’s reference to determining a second RoI based on the first RoI and based on detection results of the previous frames, wherein the detection of results correspond to recognition results of recognizing traffic lights in the previous frames by the object recognition model based on the method of Zhou’s reference. The suggestion/motivation would have been to increase the accuracy of the detection results (See Zhou, ¶ [0003–0011]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Zhou with Bush to obtain the invention as specified in claim 1.
Regarding claim 2, Bush teaches the operating method of claim 1, wherein the recognizing the traffic light (See Bush, ¶ [0034], The traffic object detection system 200 implements methods with automatically focused and scaled ROIs to sense distant traffic objects including road signs and Traffic Signal Devices (TSDs). Note: Examiner is interpreting the Traffic Signal Devices as Traffic Lights) comprises:
obtaining a target image by extracting a portion of the input image corresponding to the second RoI (See Bush, ¶ [0073], Step 606 is a step of scaling the perception data 304 of the ROIs included in the ROI data 320. In the exemplary embodiment of FIG. 6, the scaling step is a digital scaling step but other scaling processes are applicable such as optical zooming. Step 606 includes cropping and resampling (up-sampling or down-sampling) the perception data 602 to extract and resize the perception data 304 according to the 2D bounding boxes included in the ROI data 320); and
enlarging the target image to a size corresponding to a size of the input image (See Bush, ¶ [0073], Step 606 is a step of scaling the perception data 304 of the ROIs included in the ROI data 320. In the exemplary embodiment of FIG. 6, the scaling step is a digital scaling step but other scaling processes are applicable such as optical zooming. Step 606 includes cropping and resampling (up-sampling or down-sampling) the perception data 602 to extract and resize the perception data 304 according to the 2D bounding boxes included in the ROI data 320), wherein the recognizing by the object recognition model is further based on the enlarged target image (See Bush, ¶ [0074], In step 610, the scaled ROI perception data 332 is provided as an input to an NN based traffic object detection algorithm, specifically the traffic object detection module 308. In step 610, traffic object detection is performed, which results in the traffic object detection data 310).
Regarding claim 4, Bush teaches the operating method of claim 3, wherein the prior information about traffic light objects further comprises information on the first RoI generated based on the distribution data (See Bush, ¶ [0055], In a further embodiment, Lidar or Radar is used to estimate the distance away of the traffic object. These various techniques allow an estimation of likelihood of location in three-dimensional real world space, which can be converted to a ROI in perception data space using known projection transformation processing or other methods, or the location is provided directly in perception data space. In some embodiments, a weighted blend of these techniques is used to estimate a location of traffic object, thereby providing a distribution of locations (e.g. in the form of blended probability or heat maps) in, for example, real world space, which is converted to perception data space using a model of the perception data sensor (e.g. a camera) and known dimensions of the traffic object from the traffic object dimensions data 322).
Regarding claim 5, Bush teaches the operating method of claim 1, wherein the prior information about traffic light objects comprises driving environment information on an environment in which the vehicle is driving and a specification of the camera (See Bush, ¶ [0054], Further, focus area (ROI) determination sub-module 312 receives map data 318 from the maps 302 that defines, inter-alia, a road network reflecting roads in the real world and traffic objects. The map data 318 includes geospatial information for the traffic objects so that the location of different types of static traffic objects (e.g. road signs and TSDs) in the world can be known. Based on the 3D location of the vehicle 10 defined in the localization data 316 and the 3D location of traffic objects in the perception range of the vehicle 10, it is possible to estimate a depth (a distance away) of each traffic object relative to the vehicle 10. Based on a known model of the particular sensor device (e.g. a camera model when the perception data 304 is images), the relative location of the vehicle 10 and the traffic objects, known dimensions of the traffic objects (which can be a priori knowledge or data included in the maps 302), estimated location and size of the traffic objects in image space can be derived. Note: Examiner is interpreting the environment information as the map and camera specification as the known model).
Regarding claim 6, Bush teaches the operating method of claim 1, wherein the estimating the first RoI further comprises adjusting the first RoI based on information from a sensor of the vehicle (See Bush, ¶ [0055], In a further embodiment, Lidar or Radar is used to estimate the distance away of the traffic object. These various techniques allow an estimation of likelihood of location in three-dimensional real world space, which can be converted to a ROI in perception data space using known projection transformation processing or other methods, or the location is provided directly in perception data space. In some embodiments, a weighted blend of these techniques is used to estimate a location of traffic object, thereby providing a distribution of locations (e.g. in the form of blended probability or heat maps) in, for example, real world space, which is converted to perception data space using a model of the perception data sensor (e.g. a camera) and known dimensions of the traffic object from the traffic object dimensions data 322. Note: Examiner is interpreting the lidar or radar as the sensor of the vehicle).
Regarding claim 7, Bush teaches the operating method of claim 1, wherein the estimating the first RoI further comprises adjusting the first RoI based on a specification of the camera (See Bush, ¶ [0055], These various techniques allow an estimation of likelihood of location in three-dimensional real world space, which can be converted to a ROI in perception data space using known projection transformation processing or other methods, or the location is provided directly in perception data space. In some embodiments, a weighted blend of these techniques is used to estimate a location of traffic object, thereby providing a distribution of locations (e.g. in the form of blended probability or heat maps) in, for example, real world space, which is converted to perception data space using a model of the perception data sensor (e.g. a camera) and known dimensions of the traffic object from the traffic object dimensions data 322. Note: Examiner is interpreting the model of the perception data sensor as the specification of the camera).
Regarding claim 11, claim 11 is rejected the same as claim 1 and the arguments similar to that presented above for claim 1 are equally applicable to the claim 11, and all of the other limitations similar to claim 1 are not repeated herein, but incorporated by reference. Furthermore, Bush teaches An electronic device comprising: a camera; one or more processors; and a memory storing instructions configured to cause the one or more processors to (See Bush, ¶ [0042], The controller 34 includes at least one processor 44 and a computer readable storage device or media 46. The processor 44 can be any custom made or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller 34, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, any combination thereof, or generally any device for executing instructions. The computer readable storage device or media 46 may include volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processor 44 is powered down).
Regarding claim 12, claim 12 is rejected the same as claim 2 and the arguments similar to that presented above for claim 2 are equally applicable to the claim 12, and all of the other limitations similar to claim 2 are not repeated herein, but incorporated by reference.
Regarding claim 14, Bush teaches the electronic device of claim 11, wherein the electronic device is comprised in a vehicle, and wherein the instructions are further configured to cause the one or more processors to adjust the first RoI based on a sensor of the vehicle and according to a driving environment of the autonomous vehicle. (See Bush, ¶ [0055], In a further embodiment, Lidar or Radar is used to estimate the distance away of the traffic object. These various techniques allow an estimation of likelihood of location in three-dimensional real world space, which can be converted to a ROI in perception data space using known projection transformation processing or other methods, or the location is provided directly in perception data space. In some embodiments, a weighted blend of these techniques is used to estimate a location of traffic object, thereby providing a distribution of locations (e.g. in the form of blended probability or heat maps) in, for example, real world space, which is converted to perception data space using a model of the perception data sensor (e.g. a camera) and known dimensions of the traffic object from the traffic object dimensions data 322. Note: Examiner is interpreting the lidar or radar as the sensor of the vehicle)
Regarding claim 18, Bush teaches a method performed by a computing device (See Bush, [Abstract], The systems and methods receive perception data from a sensor system included in the vehicle, determine a focused Region Of Interest (ROI) in the perception data, scale the perception data of the at least one focused ROI, process the scaled perception data of the focused ROI using a neural network (NN)-based traffic object detection algorithm to provide traffic object detection data, and control at least one vehicle feature based, in part, on the traffic object detection data), the method comprising:
receiving input images captured by a camera of a moving vehicle (See Bush, ¶ [0040], The optical cameras 140a-140n are mounted on the vehicle 10 and are arranged for capturing images (e.g. a sequence of images in the form of a video) of an environment surrounding the vehicle 10);
for a first of the input images, determining a first region of interest (RoI), wherein the first RoI is determined based on historical traffic light observations (See Bush, ¶ [0055], In another embodiment, prior detection information (e.g. camera and/or Lidar) is used to develop a distribution over where the traffic objects usually are located in the perception data 304 (e.g. row, column, distance away) and this distribution can guide the ROI determination);
inputting, to an object recognition model configured to recognize traffic lights, the first input image and an enlargement of a region in the first input image defined by the first RoI, based on which the object recognition model infers a location of a first traffic light in the first input image (See Bush, ¶ [0073], Step 606 is a step of scaling the perception data 304 of the ROIs included in the ROI data 320. In the exemplary embodiment of FIG. 6, the scaling step is a digital scaling step but other scaling processes are applicable such as optical zooming. Step 606 includes cropping and resampling (up-sampling or down-sampling) the perception data 602 to extract and resize the perception data 304 according to the 2D bounding boxes included in the ROI data 320);
[for a second of the input images captured after the first input image, determining a second RoI], wherein the second RoI is determined based on the location of the traffic light in the first input image (See Bush, ¶ [0055], In another embodiment, prior detection information (e.g. camera and/or Lidar) is used to develop a distribution over where the traffic objects usually are located in the perception data 304 (e.g. row, column, distance away) and this distribution can guide the ROI determination); and
inputting, to the object recognition model, the second input image and an enlargement of a region in the second input image defined by the second RoI (See Bush, ¶ [0073], Step 606 is a step of scaling the perception data 304 of the ROIs included in the ROI data 320. In the exemplary embodiment of FIG. 6, the scaling step is a digital scaling step but other scaling processes are applicable such as optical zooming. Step 606 includes cropping and resampling (up-sampling or down-sampling) the perception data 602 to extract and resize the perception data 304 according to the 2D bounding boxes included in the ROI data 320), based on which the object recognition model infers a location, in the second input image, of the first traffic light or a second traffic light (See Bush, ¶ [0074], The traffic object detection data 310 includes traffic object location and dimensions (e.g. a refined bounding box around each detected traffic object), traffic object type, and confidence of the detection).
However, Bush fail(s) to teach for a second of the input images captured after the first input image, determining a second RoI.
Zhou, working in the same field of endeavor, teaches: for a second of the input images captured after the first input image, determining a second RoI (See Zhou, ¶ [0027], S102A, detecting a region of interest (ROI) in the current to-be-processed frame, in response to determining that the current to-be-processed frame is a detection picture frame, to determine at least one ROI in the current to-be-processed frame. ¶ [0028], S103A, updating a to-be-tracked ROI, based on the ROI in the current to-be-processed frame and a tracking result determined by a pre-order tracking picture frame. Note: Examiner is interpreting the pre-order tracking picture frame as the previous frame).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Bush’s reference for a second of the input images captured after the first input image, determining a second RoI based on the method of Zhou’s reference. The suggestion/motivation would have been to increase the accuracy of the detection results (See Zhou, ¶ [0003–0011]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Zhou with Bush to obtain the invention as specified in claim 18.
Regarding claim 19, Bush teaches the method of claim 18, further comprising: adjusting a position of the second RoI based on the first RoI or based on the historical traffic light observations (See Bush, ¶ [0056], In addition to, or in the alternative to, the above techniques, the focus area (ROI) determination sub-module 312 may use a tracking algorithm to track where traffic objects have previously been detected by the traffic object detection module 308, thereby informing the likely location of ROIs in future processing iterations).
Regarding claim 20, Bush teaches the method of claim 18, wherein the object recognition model comprises a neural network and wherein the inferred locations comprise respective bounding boxes (See Bush, ¶ [0054], The ROI data 320 may include one or more bounding boxes defining a region in the perception data 304 that should be the focus of scaling and further processing by the traffic object detection module 308).
Claim(s) 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Bush et al. (US 20230068046 A1, hereafter, "Bush") in view of Zhou et al. (US 20210383120 A1, hereafter, "Zhou") further in view of Hassan et al. (US 20210303886 A1, hereafter, "Hassan").
Regarding claim 3, Bush in view of Zhou teaches the operating method of claim 1, wherein the prior information about traffic light objects (See Bush, ¶ [0055], In another embodiment, prior map information (over time) is used to develop a distribution over where the traffic object are usually located in the perception data (e.g. row, column, distance away)) comprises:
[distribution data corresponding to a distribution of locations of previous observations of traffic lights selected based on having a size less than a threshold value].
However, Bush and Zhou fail(s) to teach distribution data corresponding to a distribution of locations of previous observations of traffic lights selected based on having a size less than a threshold value.
Hassan, working in the same field of endeavor, teaches: distribution data corresponding to a distribution of locations of previous observations of traffic lights selected based on having a size less than a threshold value (See Hassan, ¶ [0026], As a result of the training, the GAN model 200 determines a distribution of how likely any location in a real roadway scene image can contain a traffic light based on the semantics of the roadway scene. The determined distribution is used to generate augmented roadway scene images that are semantically-consistent with the roadway scene, as will be described in more detail below. In addition, the determined distribution includes anchor centers for sampling to detect small and occluded traffic lights).
Thus, it would have been obvious to one of ordinary skills in the art before the effective filing date of the claimed invention to modify Bush’s reference to distribution data corresponding to a distribution of locations of previous observations of traffic lights selected based on having a size less than a threshold value based on the method of Hassan’s reference. The suggestion/motivation would have been to provide diverse scene understanding of the environment (See Hassan, ¶ [0002–0004]).
Further, one skilled in the art could have combined the elements as described above by known method with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Therefore, it would have been obvious to combine Hassan with Bush and Zhou to obtain the invention as specified in claim 3.
Regarding claim 13, claim 13 is rejected the same as claim 3 and the arguments similar to that presented above for claim 3 are equally applicable to the claim 13, and all of the other limitations similar to claim 3 are not repeated herein, but incorporated by reference.
Allowable Subject Matter
Claim(s) 8–10 and 15–17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim(s) 8–10 and 15–17 contain subject matter that is not disclosed or made obvious in the cited art.
In regard to claim 8, when considering claim 8 as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“The operating method of claim 1, wherein the determining of the second RoI comprises:
aggregating the detection results of the previous frames;
obtaining centers of clusters of the detection results of the previous frames;
filtering the centers; and
obtaining the second RoI based on the filtered centers”.
In regard to claim 15, when considering claim 15 as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“The electronic device of claim 11, wherein the instructions are further configured to cause the one or more processors to
aggregate the detection results of the previous frames in chronological order thereof,
obtain centers of clusters of the detection results of the previous frames, and
remove some of the centers and determine the second RoI based thereon”.
In regard to claim 8, claim 9 and 10 depend on objected claim 8. Therefore, by virtue of their dependency, claim 9 and 10 is also indicated as objected subject matter.
In regard to claim 15, claim 16 and 17 depend on objected claim 15. Therefore, by virtue of their dependency, claim 16 and 17 is also indicated as objected subject matter.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Li et al. (See NPL attached, “An improved traffic lights recognition algorithm for autonomous driving in complex scenarios”) teaches We introduce a method to obtain the best size of the region of interest (ROI) dynamically, including four aspects. First, based on multi-sensor data (RTK BDS/GPS, IMU, camera, and LiDAR) acquired in a normal environment, we generated a prior map that contained sufficient traffic lights information. And then, by analyzing the relationship between the error of the sensors and the optimal size of ROI, the adaptively dynamic adjustment (ADA) model was built. Furthermore, according to the multi-sensor data fusion positioning and ADA model, the optimal ROI can be obtained to predict the location of traffic lights.
Alsallakh et al. (US 20210117730 A1) teaches weaknesses may be exposed in image object detectors. An image object is overlaid onto a background image at each of a plurality of locations, the background image including a scene in which the image objects can be present. A detector model is used to attempt detection of the image object as overlaid onto the background image, the detector model being trained to identify the image object in background images, the detection resulting in background scene detection scores indicative of likelihood of the image object being detected at each of the plurality of locations. A detectability map is displayed overlaid on the background image, the detectability map including, for each of the plurality of locations, a bounding box of the image object illustrated according to the respective detection score.
Uliyar et al. (US 20170337435 A1) teaches advanced driver assistance systems (ADAS) and methods for object detection such as traffic lights, speed signs, in an automotive environment, are disclosed. In an embodiment, ADAS includes camera system for capturing image frames of at least a part of surroundings of vehicle, memory comprising image processing instructions and processing system for detecting one or more objects in a coarse detection followed by a fine detection. Coarse detection includes detecting presence of the one or more objects in non-consecutive image frames of the image frames, where non-consecutive image frames are determined by skipping one or more frames of the image frames. Upon detection of presence of the one or more objects in coarse detection, fine detection of the one or more objects is performed in a predetermined number of neighboring image frames of a frame in which the presence of the objects is detected in coarse detection.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DION J SATCHER whose telephone number is (703)756-5849. The examiner can normally be reached Monday - Thursday 5:30 am - 2:30 pm, Friday 5:30 am - 9:30 am PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at (571) 272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DION J SATCHER/ Patent Examiner, Art Unit 2676
/Henok Shiferaw/ Supervisory Patent Examiner, Art Unit 2676