DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. KR10-2022-0147403, filed on 11/7/2022.
Response to Amendment
This action is in response to amendments and remarks filed on 03/18/2026. The examiner notes the following adjustments to the claims by the applicant:
Claims 1, 9 and 16 are amended;
No additional claims are cancelled;
No claims have been added.
Therefore, Claims 1-4, 6-7, 9-16, and 18-20 are pending examination, in which Claims 1, 9 and 16 are independent claims.
In light of the instant amendments and arguments:
The objection to the Specifications and Drawings for a minor informalities is withdrawn.
Further examination resulted in a new rejection of Claims 1-4, 6-7, 9-16, and 18-20 under 35 U.S.C. § 103, as detailed below.
THIS ACTION IS MADE FINAL. Necessitated by amendment.
Response to Arguments
Applicant presents the following arguments regarding the previous office action:
[A] To overcome the 35 U.S.C. § 103 rejection, the applicant has amended each independent claim, such as for Claim 1: "wherein the acquiring of the information about the target traffic light comprises: detecting the position of the vehicle using a position sensor; determining a vehicle-map-point corresponding to the detected position of the vehicle, wherein the vehicle-map-point is in a map comprised in the map data; selecting, based on the vehicle-map-point, a virtual traffic light among virtual traffic lights in the map as a target virtual traffic light that corresponds to the target traffic light; and acquiring the information about the target traffic light based on the selection of the target virtual traffic light, and wherein the selecting a virtual traffic light comprises: determining a map route of the vehicle in the map, identifying the virtual traffic light that corresponds to the target traffic light based on a distance on the map between the map route and a virtual traffic light among the virtual traffic lights in the map…to generate sequential position data representing a vertical order of the lamp regions…wherein the generating the intensity profile comprises generating, based on the average intensity of the pixels on the respective vertical line segments, the intensity profile representing a relationship between distance across the target traffic light and the average intensity, and wherein the determining the illuminated lamp among the lamp regions comprises: determining a maximum one of the average intensities; determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value; and selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio;
[B] “as noted above, Jeonq merely describes CNN-based object detection for detecting a traffic light and signal information output from the traffic light. However, as opposed to the Office's statement and interpretation, Jeonq is silent as to claimed features related to identifying boundaries of multiple lamp regions within a single traffic light and generating positional index data representing a vertical order of such lamp regions, as required by amended claim 1.”;
[C] “it is respectfully submitted that Kawanai and Jeong do not, and would/could not, disclose or suggest, taken individually or even in combination thereof, at least claimed features of "determining a sequential position of each lamp region of the target traffic light by performing, using a first neural network, an inference on the isolated image to identify boundaries of lamp regions and to generate sequential position data representing a vertical order of the lamp regions," as set forth in claim 1, for example”;
[D] “it is respectfully submitted that Ben-Shalom does not and would/could not, disclose or suggest, claimed features of "determining, using a second neural network, an illuminated lamp among the lamp regions by computing an average intensity of pixels on respective vertical line segments within each lamp region to generate an intensity profile, wherein the generating the intensity profile comprises generating, based on the average intensity of the pixels on the respective vertical line segments, the intensity profile representing a relationship between distance across the target traffic light and the average intensity, and wherein the determining the illuminated lamp among the lamp regions comprises: determining a maximum one of the average intensities; determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value; and selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio," as set forth in claim 1, for example.”.
Applicant's arguments [A], [B], [C] and [D] appear to be directed to the instantly amended subject matter. Accordingly, they have been addressed in the rejections below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-4, 6-7, 9-16, and 18-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The amended independent Claims 1, 9 and 16 recite the limitation:
"determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio”
It is unclear what constitutes the “threshold ratio”. What parameters go into determining the ratio? What are bounds of the this ratio?
Dependent Claims 2-4, 6-7, 10-16, 18 and 20 are also rejected under 35 U.S.C. 112(b) as depending from independent claims rejected under 35 U.S.C. 112(b).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-7, 9-16 and 18-20 are rejected under 35 U.S.C. §103 as being unpatentable over the combination of Kawanai (US 10,846,546 B2), Napanda et al. (US 2022/0114888 A1, henceforth Napanda) and Kamioka (US 8,242,934 B2).
Regarding Claim 1, Kawanai discloses the limitations: a method of determining a signal state of a target traffic light {“traffic signal recognition device”, Abstract}, the method comprising:
acquiring an isolated image of the target traffic light from an input image of the target traffic light {S12, Fig. 5: “Acquire image captured by camera”; object recognition unit 13, Fig. 1: “The object recognition unit 13 recognizes an object around the vehicle M based on the image captured by the camera 2 and the result of detection performed by the radar sensor 3.”, Col. 6, Lns. 48-51}, wherein the input image is captured when the vehicle is at a position {vehicle position recognition unit 11 and position determining unit 12, Fig. 1: “The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the information on the position from the GPS receiver 1 and the map information from the map database 4.”, Col. 5, Lns. 46-49; see also, Col. 5, Lns. 63 - Col. 6, Ln. 3}, wherein the isolated image is acquired using image processing on the input image {“If the object recognition unit 13 can recognize even one of the lighting portions of the traffic signal 110, since there is a possibility that the lighting portion is a currently lit point, the imaging possibility determination unit 14 may determine that the traffic signal can be imaged by the camera 2, and proceed with the processing.”, Col. 8, Lns. 62-67}, the input image having been captured by a camera module of the vehicle {camera 2, Fig. 1, and “FIG. 4A illustrates a captured image 3”, Col. 10, Lns. 14-15}; acquiring information about the target traffic light based on a map position in map data {“the imaging possibility determination unit 14 extracts the traffic signals and the surrounding structures existing around the vehicle from the traffic signal data and the surrounding structure data included in traffic signal related database 5 referring to the position of the vehicle on the map.”, Col. 7, Lns. 13-18; also, Col. 5, Lns. 33-36}, wherein the acquiring of the information about the target traffic light comprises: detecting the position of the vehicle using a position sensor {“The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the information on the position from the GPS receiver 1 and the map information from the map database 4.”, Col. 5, Lns. 46-49}; determining a vehicle-map-point corresponding to the detected position of the vehicle, wherein the vehicle-map-point is in a map comprised in the map data; selecting, based on the vehicle-map-point, a virtual traffic light among virtual traffic lights in the map as a target virtual traffic light that corresponds to the target traffic light {identification process to match the traffic light imaged by the vehicle to a database of traffic lights and their location: “determination unit 14 performs collation (matching) between the traffic signal and the surrounding structure extracted from the traffic signal related database 5 and the result of recognition performed by the object recognition unit 13.”, Col. 7, Lns. 23-27}; and acquiring the information about the target traffic light based on the selection of the target virtual traffic light {the combination of vehicle position recognition 11, position determination units 12, map database 4 and traffic signal related database 5, Fig. 1, enables matching of the traffic signal in front of the vehicle, Fig. 2, to the corresponding traffic signal associated with that specific location on the map: “traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map.”, Col. 4, Lns. 61-67}, and wherein the selecting a virtual traffic light comprises: determining a map route of the vehicle in the map {“The position determination unit 12 performs the above-described determination based on the position of the vehicle on the map recognized by the vehicle position recognition unit 11 and the position of the traffic signal on the map in the traffic signal data”, Col. 7, Lns. 13-18}, identifying the virtual traffic light that corresponds to the target traffic light based on a distance on the map between the map route and a virtual traffic light among the virtual traffic lights in the map {determining the distance to the traffic light, in order to be within the “traffic signal recognition range” is described in Col. 6, Lns. 4-14, after which identification of the specific traffic signal in front of the vehicle is achieved by consulting map database 4 and traffic signal related database 5 (Fig.1) as discussed in Col. 7, Lns. 23-27}; determining the signal state of the target traffic light based on the isolated image of the target traffic light and based on the information about the target traffic light {traffic sign recognition unit 17, Fig. 1: “The traffic signal recognition unit 17 recognizes the lighting state of the traffic signal by performing image recognition (pattern matching considering the color, for example) in the search area R”, Col. 10, Lns. 33-36, including determining a sequential position of each lamp region of the target traffic light {“A traffic signal recognition device 100 illustrated in FIG. 1 is mounted on a vehicle such as a passenger car and recognizes a lighting state of a traffic signal included in an image captured by a camera in the vehicle. The lighting state of the traffic signal includes at least a passing permission lighting state (for example, a green signal) and a passing prohibition lighting state (for example, a red signal). The lighting state of the traffic signal may include a transition lighting state (for example, a yellow signal) indicating a transition from the passing permission lighting state to the passing prohibition lighting state.”, Col. 3, Lns. 53-65}, including determining a sequential position of each lamp region of the target traffic light {traffic signal recognition device 200, Fig. 6, can determine whether the traffic signal is of a vertical or horizontal, Col. 14, Lns. 5-10, which one skilled in the art will appreciate determines whether the green right is below or to the right, respectively, of the red traffic light}; and mapping the determined sequential position of the illuminated lamp to a traffic-signal type stored in a signal record retrieved from the map data, the signal record indicating signal- types corresponding to respective lamp positions of the target traffic light {the combination of vehicle position recognition 11, position determination units 12, map database 4 and traffic signal related database 5, Fig. 1, enables matching of the traffic signal in front of the vehicle, Fig. 2, to the corresponding traffic signal associated with that specific location on the map: “ traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map. The information on the shape of the traffic signal means information on the three-dimensional shape of the traffic signal.”, Col. 4, Lns. 61-67 and Col. 5, Lns. 1-2}; and controlling autonomous driving of the vehicle by applying a result of the determining the signal state of the traffic light {“the vehicle 10 is an autonomous driving vehicle or operating an autonomous driving function, the apparatus 100 for extracting signal information may determine a control command for controlling the operation (stop, start, etc.) of the vehicle 10 based on the signal information.”, Col. 12, Lns. 54-58}.
Kawanai does not appear to explicitly recite the limitations: determining a sequential position of each lamp region of the target traffic light by performing, using a first neural network, an inference on the isolated image to identify boundaries of lamp regions and to generate sequential position data representing a vertical order of the lamp regions, determining, using a second neural network, an illuminated lamp among the lamp regions by computing an average intensity of pixels on respective vertical line segments within each lamp region to generate an intensity profile, wherein the generating the intensity profile comprises generating, based on the average intensity of the pixels on the respective vertical line segments, the intensity profile representing a relationship between distance across the target traffic light and the average intensity, and wherein the determining the illuminated lamp among the lamp regions comprises: determining a maximum one of the average intensities; determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value; and
selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio.
However, Napanda explicitly recites the limitations: determining a sequential position of each lamp region of the target traffic light by performing, using a first neural network {2123, Fig. 1}, an inference on the isolated image to identify boundaries of lamp regions and to generate sequential position data representing a vertical order of the lamp regions {first machine learning model 2123, Fig. 1, locates and positions a bounding box (1605, Fig. 1B) around the traffic signal (and thus identifies traffic signal orientation): “Position processor 2103 can provide filtered data 1603 to first machine learning model (MLM) 2123. First MLM 2123 can provide bounding boxes 1605 for objects located in sensor images 1602 (FIG. 1A)…First MLM 2123 (FIG. 1) can detect objects in images using, for example, but not limited to, a single deep neural network having a unified framework for training and inference….Bounding box processor 2105 (FIG. 1) can determine the traffic light of interest to the AV based on map point 1601.”, ¶[0033-0034]}; determining, using a second neural network {2115, Fig. 1, and “a second machine learning model to determine traffic light state probabilities, providing either the bounding boxes or the tracked candidate traffic lights to the second machine learning model, and receiving inferences from the model. Method 1950 can include updating 1963 tracking data, and publishing 1965 the probabilities of the traffic light states determined from the inferences”, ¶[0057]}, an illuminated lamp among the lamp regions by computing an average intensity of pixels on respective vertical line segments within each lamp region to generate an intensity profile {image processing to provide high quality intensity mapping of lamp areas: “if the feature is a traffic light, the colors of the bulbs are expected to be red, yellow, and green. The intensity of the colors can be increased if necessary, and noise can be removed, both by conventional methods. For example, noise can be removed by applying moving averages, moving averages with non-uniform weight…mean filter, median filter, and a bilateral filter. After noise is removed, a dilation (adding pixels to the boundary of the feature) followed by erosion (removing pixels from the feature boundary) (a morphological closing) can be applied to enlarge boundaries of foreground regions and shrink background color holes in the regions. The result is the filling in of small background color holes in image data.”, ¶[0033]}, wherein the generating the intensity profile comprises generating, based on the average intensity of the pixels on the respective vertical line segments, the intensity profile representing a relationship between distance across the target traffic light and the average intensity {having generated the quality intensity mapping of the traffic light bulb regions just described with respect to ¶[0033], one skilled in the art will appreciate that standard graphical interpretation techniques, such as taking averages or restricting attention to only part of the entire mapped area, can then be used}.
Kawanai and Napanda are analogous art because they both deal with traffic signal recognition.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Kawanai and Napanda before them, to modify the teachings of Kawanai to include the teachings of Napanda to accurately determine, with high probability, that a traffic light, encountered by an autonomous vehicle, is in a specific state to enhance navigation of an intersection {¶[0010]}.
The combination of Kawanai and Napanda does not appear to explicitly recite the limitations: wherein the determining the illuminated lamp among the lamp regions comprises:
determining a maximum one of the average intensities; determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value; and selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio.
However, Kamioka explicitly recites the limitations: wherein the determining the illuminated lamp among the lamp regions comprises: determining a maximum one of the average intensities {histogram (i.e., number of pixels vs. brightness) of a light source in Fig. 2C, reflective object in Fig. 3C, a luminous object in Fig. 4B, wherein each histogram interval is effectively an average of multiple brightness values, and the interval with the highest number of pixels or the highest brightness can be deemed a maximum}; determining a threshold intensity value based on the maximum one of the average intensities {“threshold α” (i.e., alpha) in Figs. 2C and 3C} and a threshold ratio {“FIGS. 9A and 10A show the domains which include pixels having brightness 90% of the brightest pixel extracted (hereafter called “the 90% domain”) while FIGS. 9B and 10B show the domains which include pixels having brightness 60% of the brightest pixel extracted (hereafter called “the 60% domain”).”, Col. 8, Lns. 22-27}; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value {with regard to Figs. 2C&3C, threshold-alpha: “In the histograms (FIGS. 2C and 3C), the threshold α for determining that it is the domain of the light source is shown. Regarding the group classified as having brightness less than this threshold α will be ignored when distribution of the number of pixels is considered.”, Col. 5, Lns. 40-44}; and selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio {Figs. 9A&10A represent the distinction between highly illuminated areas (white) and low illuminated areas (black) in which the size of the area is depends on the criteria or threshold employed to establish the demarcation between the two regions; ibid, Col. 8, Lns. 22-27}.
The combination of references Kawanai and Napanda along with Kamioka are analogous art because they deal with recognition of illuminated objects.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Kawanai, Napanda and Kamioka before them, to modify the teachings of the combination of Kawanai and Napanda to include the teachings of Kamioka to discriminate between a strong light source, such as a vehicle light, and weak light source, such as reflection off a road-sign {Abstract}.
Regarding Claim 2, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 1, as discussed supra. In addition, Kawanai explicitly recites the limitations: wherein the acquiring of the isolated image of the target traffic light comprises: identifying candidate traffic lights based on determining that the candidate traffic lights are in front of the vehicle along a driving route of the vehicle {“FIG. 3A is a diagram illustrating a traffic signal in front of the vehicle and a surrounding structure extracted from a traffic signal related database.”, Col. 3, Lns. 16-18 and Fig. 2}, and selecting, as the target traffic light, whichever of the candidate traffic lights is determined to be closest to the vehicle {as evident in Fig. 2}.
Regarding Claim 3, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 1, as discussed supra. In addition, Kawanai recites the limitations: wherein the acquiring of the information about the target traffic light comprises: detecting the position of the vehicle using a position sensor {“The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the information on the position from the GPS receiver 1 and the map information from the map database 4.”, Col. 5, Lns. 46-49}; determining a vehicle-map-point corresponding to the detected position of the vehicle, wherein the vehicle-map-point is in a map comprised in the map data; selecting, based on the vehicle-map-point, a virtual traffic light among virtual traffic lights in the map as a target virtual traffic light that corresponds to the target traffic light; and acquiring the information about the target traffic light based on the selection of the target virtual traffic light {the combination of vehicle position recognition 11, position determination units 12, map database 4 and traffic signal related database 5, Fig. 1, enables matching of the traffic signal in front of the vehicle, Fig. 2, to the corresponding traffic signal associated with that specific location on the map: “ traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map. The information on the shape of the traffic signal means information on the three-dimensional shape of the traffic signal.”, Col. 4, Ln. 61 - Col. 5, Ln. 2}.
Regarding Claim 4, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 1, as discussed supra. In addition, Kawanai recites the limitations: wherein the information about the target traffic light comprises indicating signal-type {“In traffic signal related database 21, information on the type of the traffic signal is stored in the traffic signal data in addition to the content in the first embodiment. The information on the type of the traffic signal includes at least the information on whether or not the traffic signal is a hanging type. The information on the type of the traffic signal may include information on whether the traffic signal is a horizontal arrangement type or a vertical arrangement type.”, Col. 14, Lns. 5-12}, wherein the information about the target traffic light comprises first information indicating a number of lamps of the traffic light and second information indicating signal-types of respective positions of the lamps of the target traffic light {“traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map.”, Col. 4, Ln. 61 - Col. 5, Ln. 2}.
Regarding Claim 6, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 4, as discussed supra. The combination of Kawanai and Napanda does not appear to explicitly recite the limitation: wherein the calculating the sequential position comprises dividing the isolated image into areas respectively corresponding to the lamps of the target traffic light and computing average intensity values of the respective areas.
However, Kamioka explicitly recites the limitation: wherein the calculating the sequential position comprises dividing the isolated image into areas respectively corresponding to the lamps of the target traffic light and computing average intensity values of the respective areas {averaging process involved in the generation of histograms in Fig. 2C (light source) and Fig. 3C (reflective object), of number of pixels vs. grouped-brightness, defined in ¶[0013] as “a term “grouped brightness” may be the average, maximum, mean, minimum etc. of the brightness for the group”}.
Regarding Claim 7, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 4, as discussed supra. In addition, Kawanai explicitly recites the limitation: wherein the illuminated lamp is selected as a basis for determining the signal state, and wherein the illuminated lamp is selected based on the average intensity value of its corresponding area {“the traffic signal recognition unit 17 performs the traffic signal recognition for recognizing the lighting state of the traffic signal in the search area R. The traffic signal recognition unit 17 recognizes the lighting state of the traffic signal by performing image recognition in the search area R.”, Col. 12, Lns. 39-44 and Fig. 8}.
Regarding Claim 9, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 4, as discussed supra. In addition, Kawanai explicitly recites the limitation: wherein the determining of the signal state of the target traffic light comprises extracting the signal information from among a set of signal information based on the signal information corresponding to the determined sequential position {“the traffic signal recognition unit 17 performs the traffic signal recognition for recognizing the lighting state of the traffic signal in the search area R. The traffic signal recognition unit 17 recognizes the lighting state of the traffic signal by performing image recognition in the search area R.”, Col. 12, Lns. 39-44 and Fig. 8}.
Regarding Claim 10, Kawanai discloses the limitations: an electronic device {Fig. 1} for determining a signal state of a target traffic light {“traffic signal recognition device”, Abstract}, the electronic device comprising: a camera module {2, Fig. 1} configured to capture an image of the surroundings of a vehicle connected with the electronic device {S12, Fig. 5: “Acquire image captured by camera”; object recognition unit 13, Fig. 1: “The object recognition unit 13 recognizes an object around the vehicle M based on the image captured by the camera 2 and the result of detection performed by the radar sensor 3.”, Col.6, Lns. 48-51}; one or more processors {ECU 10 of traffic signal recognition device 100 in Fig. 1}; memory storing instructions configured to cause the one or more processors {ECU 10 incudes CPU, memory and programming, Col. 4, Lns. 6-16} to: receive map data from a server {map database 4, Fig. 1, and “The traffic signal related database 5 may be configured as a database integrated with the map database 4, or may be formed in a server capable of communicating with the vehicle.”, Col. 5, Lns. 33-36}; acquire an isolated image of the target traffic light from an input image of the target traffic light captured by the camera module {S12, Fig. 5: “Acquire image captured by camera”; object recognition unit 13, Fig. 1: “The object recognition unit 13 recognizes an object around the vehicle M based on the image captured by the camera 2 and the result of detection performed by the radar sensor 3.”, Col. 6, Lns. 48-51}while the vehicle is at a sensed physical location {vehicle position recognition unit 11 and position determining unit 12, Fig. 1: “The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the information on the position from the GPS receiver 1 and the map information from the map database 4.”, Col. 5, Lns. 46-49; see also, Col. 5, Lns. 63 - Col. 6, Ln. 3}, acquire, from the map data, information about the target traffic light, using the sensed physical location of the vehicle{“the imaging possibility determination unit 14 extracts the traffic signals and the surrounding structures existing around the vehicle from the traffic signal data and the surrounding structure data included in traffic signal related database 5 referring to the position of the vehicle on the map.”, Col. 7, Lns. 13-18; also, Col. 5, Lns. 33-36}, wherein the acquiring of the information about the target traffic light comprises: detecting the position of the vehicle using a position sensor {“The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the information on the position from the GPS receiver 1 and the map information from the map database 4.”, Col. 5, Lns. 46-49}; determining a vehicle-map-point corresponding to the detected position of the vehicle, wherein the vehicle-map-point is in a map comprised in the map data; selecting, based on the vehicle-map-point, a virtual traffic light among virtual traffic lights in the map as a target virtual traffic light that corresponds to the target traffic light {identification process to match the traffic light imaged by the vehicle to a database of traffic lights and their location: “determination unit 14 performs collation (matching) between the traffic signal and the surrounding structure extracted from the traffic signal related database 5 and the result of recognition performed by the object recognition unit 13.”, Col. 7, Lns. 23-27}; and acquiring the information about the target traffic light based on the selection of the target virtual traffic light {the combination of vehicle position recognition 11, position determination units 12, map database 4 and traffic signal related database 5, Fig. 1, enables matching of the traffic signal in front of the vehicle, Fig. 2, to the corresponding traffic signal associated with that specific location on the map: “traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map.”, Col. 4, Lns. 61-67}, and wherein the selecting a virtual traffic light comprises: determining a map route of the vehicle in the map {“The position determination unit 12 performs the above-described determination based on the position of the vehicle on the map recognized by the vehicle position recognition unit 11 and the position of the traffic signal on the map in the traffic signal data”, Col. 7, Lns. 13-18}, identifying the virtual traffic light that corresponds to the target traffic light based on a distance on the map between the map route and a virtual traffic light among the virtual traffic lights in the map {determining the distance to the traffic light, in order to be within the “traffic signal recognition range” is described in Col. 6, Lns. 4-14, after which identification of the specific traffic signal in front of the vehicle is achieved by consulting map database 4 and traffic signal related database 5 (Fig.1) as discussed in Col. 7, Lns. 23-27}; determining the signal state of the target traffic light based on the isolated image of the target traffic light and based on the information about the target traffic light {traffic sign recognition unit 17, Fig. 1: “The traffic signal recognition unit 17 recognizes the lighting state of the traffic signal by performing image recognition (pattern matching considering the color, for example) in the search area R”, Col. 10, Lns. 33-36}, including determining a sequential position of each lamp region of the target traffic light {“A traffic signal recognition device 100 illustrated in FIG. 1 is mounted on a vehicle such as a passenger car and recognizes a lighting state of a traffic signal included in an image captured by a camera in the vehicle. The lighting state of the traffic signal includes at least a passing permission lighting state (for example, a green signal) and a passing prohibition lighting state (for example, a red signal). The lighting state of the traffic signal may include a transition lighting state (for example, a yellow signal) indicating a transition from the passing permission lighting state to the passing prohibition lighting state.”, Col. 3, Lns. 53-65}, including determining a sequential position of each lamp region of the target traffic light {traffic signal recognition device 200, Fig. 6, can determine whether the traffic signal is of a vertical or horizontal, Col. 14, Lns. 5-10, which one skilled in the art will appreciate determines whether the green right is below or to the right, respectively, of the red traffic light}; and mapping the determined sequential position of the illuminated lamp to a traffic-signal type stored in a signal record retrieved from the map data, the signal record indicating signal-types corresponding to respective lamp positions of the target traffic light {the combination of vehicle position recognition 11, position determination units 12, map database 4 and traffic signal related database 5, Fig. 1, enables matching of the traffic signal in front of the vehicle, Fig. 2, to the corresponding traffic signal associated with that specific location on the map: “ traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map. The information on the shape of the traffic signal means information on the three-dimensional shape of the traffic signal.”, Col. 4, Ln. 61 - Col. 5, Ln. 2}; and control autonomous driving of the vehicle by applying a result of the determining the signal state of the traffic light {“the vehicle 10 is an autonomous driving vehicle or operating an autonomous driving function, the apparatus 100 for extracting signal information may determine a control command for controlling the operation (stop, start, etc.) of the vehicle 10 based on the signal information.”, Col. 12, Lns. 54-58}.
Kawanai does not appear to explicitly recite the limitations: determining a sequential position of each lamp region of the target traffic light by performing, using a first neural network, an inference on the isolated image to identify boundaries of lamp regions and to generate sequential position data representing a vertical order of the lamp regions, determining, using a second neural network, an illuminated lamp among the lamp regions by computing an average intensity of pixels on respective vertical line segments within each lamp region to generate an intensity profile, wherein the generating the intensity profile comprises generating, based on the average intensity of the pixels on the respective vertical line segments, the intensity profile representing a relationship between distance across the target traffic light and the average intensity, and wherein the determining the illuminated lamp among the lamp regions comprises: determining a maximum one of the average intensities; determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value; and
selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio.
However, Napanda explicitly recites the limitations: determining a sequential position of each lamp region of the target traffic light by performing, using a first neural network {2123, Fig. 1}, an inference on the isolated image to identify boundaries of lamp regions and to generate sequential position data representing a vertical order of the lamp regions {first machine learning model 2123, Fig. 1, locates and positions a bounding box (1605, Fig. 1B) around the traffic signal (and thus identifies traffic signal orientation): “Position processor 2103 can provide filtered data 1603 to first machine learning model (MLM) 2123. First MLM 2123 can provide bounding boxes 1605 for objects located in sensor images 1602 (FIG. 1A)…First MLM 2123 (FIG. 1) can detect objects in images using, for example, but not limited to, a single deep neural network having a unified framework for training and inference….Bounding box processor 2105 (FIG. 1) can determine the traffic light of interest to the AV based on map point 1601.”, ¶[0033-0034]}; determining, using a second neural network {2115, Fig. 1, and “a second machine learning model to determine traffic light state probabilities, providing either the bounding boxes or the tracked candidate traffic lights to the second machine learning model, and receiving inferences from the model. Method 1950 can include updating 1963 tracking data, and publishing 1965 the probabilities of the traffic light states determined from the inferences”, ¶[0057]}, an illuminated lamp among the lamp regions by computing an average intensity of pixels on respective vertical line segments within each lamp region to generate an intensity profile {image processing to provide high quality intensity mapping of lamp areas: “if the feature is a traffic light, the colors of the bulbs are expected to be red, yellow, and green. The intensity of the colors can be increased if necessary, and noise can be removed, both by conventional methods. For example, noise can be removed by applying moving averages, moving averages with non-uniform weight…mean filter, median filter, and a bilateral filter. After noise is removed, a dilation (adding pixels to the boundary of the feature) followed by erosion (removing pixels from the feature boundary) (a morphological closing) can be applied to enlarge boundaries of foreground regions and shrink background color holes in the regions. The result is the filling in of small background color holes in image data.”, ¶[0033]}, wherein the generating the intensity profile comprises generating, based on the average intensity of the pixels on the respective vertical line segments, the intensity profile representing a relationship between distance across the target traffic light and the average intensity {having generated the quality intensity mapping of the traffic light bulb regions just described with respect to ¶[0033], one skilled in the art will appreciate that standard graphical interpretation techniques, such as taking averages or restricting attention to only part of the entire mapped area, can then be used.}.
The combination of Kawanai and Napanda does not appear to explicitly recite the limitations: wherein the determining the illuminated lamp among the lamp regions comprises:
determining a maximum one of the average intensities; determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value; and selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio.
However, Kamioka explicitly recites the limitations: wherein the determining the illuminated lamp among the lamp regions comprises: determining a maximum one of the average intensities {histogram (i.e., number of pixels vs. brightness) of a light source in Fig. 2C, reflective object in Fig. 3C, a luminous object in Fig. 4B, wherein each histogram interval is effectively an average of multiple brightness values, and the interval with the highest number of pixels or the highest brightness can be deemed a maximum}; determining a threshold intensity value based on the maximum one of the average intensities {“threshold α” (i.e., alpha) in Figs. 2C and 3C} and a threshold ratio {“FIGS. 9A and 10A show the domains which include pixels having brightness 90% of the brightest pixel extracted (hereafter called “the 90% domain”) while FIGS. 9B and 10B show the domains which include pixels having brightness 60% of the brightest pixel extracted (hereafter called “the 60% domain”).”, Col. 8, Lns. 22-27}; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value {with regard to Figs. 2C&3C, threshold-alpha: “In the histograms (FIGS. 2C and 3C), the threshold α for determining that it is the domain of the light source is shown. Regarding the group classified as having brightness less than this threshold α will be ignored when distribution of the number of pixels is considered.”, Col. 5, Lns. 40-44}; and selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio {Figs. 9A&10A represent the distinction between highly illuminated areas (white) and low illuminated areas (black) in which the size of the area is depends on the criteria or threshold employed to establish the demarcation between the two regions; ibid, Col. 8, Lns. 22-27}.
Regarding Claim 11, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 10, as discussed supra. In addition, Kawanai explicitly recites the limitations: wherein the instructions are further configured to cause the one or more processors to identify candidate traffic lights from images captured by respective cameras of the camera module, wherein the candidate traffic lights are determined based on being in front of the vehicle according to a driving route of the vehicle {“FIG. 3A is a diagram illustrating a traffic signal in front of the vehicle and a surrounding structure extracted from a traffic signal related database.”, Col. 3, Lns. 16-18 and Fig. 2}, wherein instructions are configured to select, as the target traffic light, whichever candidate traffic light is determined to be closest to the vehicle among the identified candidate traffic lights {as evident in Fig. 2}.
Regarding Claim 12, the combination of Kawanai and Napanda the limitations of Claim 10, as discussed supra. In addition, Kawanai recites the limitations wherein the instructions are further configured to cause the one or more the processors to: determine a vehicle-map-point corresponding to the detected position of the vehicle, wherein the vehicle-map-point is in a map comprised in the map data; select, based on the vehicle-map-point, a virtual traffic light among virtual traffic lights in the map as a target virtual traffic light that corresponds to the target traffic light; and acquire the information about the target traffic light based on the selection of the target virtual traffic light {the combination of vehicle position recognition 11, position determination units 12, map database 4 and traffic signal related database 5, Fig. 1, enables matching of the traffic signal in front of the vehicle, Fig. 2, to the corresponding traffic signal associated with that specific location on the map: “ traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map. The information on the shape of the traffic signal means information on the three-dimensional shape of the traffic signal.”, Col. 4, Ln. 61 - Col. 5, Ln. 2}.
Regarding Claim 13, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 10, as discussed supra. In addition, Kawanai recites the limitations: wherein the information about the target traffic light comprises traffic-signal types respectively associated with sequential positions of lamps of the target traffic light {“In traffic signal related database 21, information on the type of the traffic signal is stored in the traffic signal data in addition to the content in the first embodiment. The information on the type of the traffic signal includes at least the information on whether or not the traffic signal is a hanging type. The information on the type of the traffic signal may include information on whether the traffic signal is a horizontal arrangement type or a vertical arrangement type.”, Col. 14, Lns. 5-12}.
Regarding Claim 14, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 13, as discussed supra. In addition, Kawanai recites the limitations: wherein the instructions are further configured to cause the one or more processors to determine, from the isolated image of the target traffic light {Region R, represented by H2xW2, Fig. 4B}, a sequential position of a turned-on lamp of the target traffic light {“A traffic signal recognition device 100 illustrated in FIG. 1 is mounted on a vehicle such as a passenger car and recognizes a lighting state of a traffic signal included in an image captured by a camera in the vehicle. The lighting state of the traffic signal includes at least a passing permission lighting state (for example, a green signal) and a passing prohibition lighting state (for example, a red signal). The lighting state of the traffic signal may include a transition lighting state (for example, a yellow signal) indicating a transition from the passing permission lighting state to the passing prohibition lighting state.”, Col. 3, Lns. 53-65}, and determine the signal state of the target traffic light to be the traffic-signal type associated with the determined sequential position of the turned-on lamp {traffic sign recognition unit 17, Fig. 1: “The traffic signal recognition unit 17 recognizes the lighting state of the traffic signal by performing image recognition (pattern matching considering the color, for example) in the search area R”, Col. 10, Lns. 33-36}.
Regarding Claim 15, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 14, as discussed supra. The combination of Kawanai and Napanda does not appear to explicitly recite the limitation: wherein the instructions are further configured to cause the one or more processors to compute pixel-intensity measures of areas in the isolated image that contain respective lamps of the target traffic light.
However, Kamioka explicitly recites the limitation: wherein the instructions are further configured to cause the one or more processors to compute pixel-intensity measures of areas in the isolated image that contain respective lamps of the target traffic light {histograms in Fig. 2C (light source) and Fig. 3C (reflective object), of number of pixels vs. grouped-brightness, defined in ¶[0013] as “a term “grouped brightness” may be the average, maximum, mean, minimum etc. of the brightness for the group”}.
Regarding Claim 16, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 15, as discussed supra. In addition, Kawanai explicitly recites the limitation: wherein the instructions are further configured to cause the one or more processors to determine the sequential position of the turned-on lamp based on the pixel-intensities {“the traffic signal recognition unit 17 performs the traffic signal recognition for recognizing the lighting state of the traffic signal in the search area R. The traffic signal recognition unit 17 recognizes the lighting state of the traffic signal by performing image recognition in the search area R.”, Col. 12, Lns. 39-44 and Fig. 8}.
Regarding Claim 18, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 14, as discussed supra. In addition, Kawanai explicitly recites the limitation: wherein the determined sequential position is used to select the traffic-signal type {Region R, represented by H2xW2, Fig. 4B} from among the traffic-signal types associated with the target traffic lamp {“the traffic signal recognition unit 17 performs the traffic signal recognition for recognizing the lighting state of the traffic signal in the search area R. The traffic signal recognition unit 17 recognizes the lighting state of the traffic signal by performing image recognition in the search area R.”, Col. 12, Lns. 39-44 and Fig. 8}.
Regarding Claim 19, Kawanai discloses the limitations: a method comprising: sensing a location of a moving vehicle {vehicle position recognition unit 11 and position determining unit 12, Fig. 1: “The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the information on the position from the GPS receiver 1 and the map information from the map database 4.”, Col. 5, Lns. 46-49; also, Col. 5, Ln. 63 - Col. 6, Ln. 3} when a camera of the vehicle {S12, Fig. 5: “Acquire image captured by camera”; object recognition unit 13, Fig. 1: “The object recognition unit 13 recognizes an object around the vehicle M based on the image captured by the camera 2 and the result of detection performed by the radar sensor 3.”, Col.6, Lns. 48-51} captures an image {Fig. 4B} of a target traffic light {Fig. 2}; mapping the sensed location of the vehicle to a vehicle-map-location in a map {“The vehicle position recognition unit 11 recognizes the position of the vehicle on the map based on the information on the position from the GPS receiver 1 and the map information from the map database 4.”, Col. 5, Lns. 46-49}, wherein each traffic light element has a respectively associated signal record, wherein each signal record indicates traffic-signal-types of respective lamp positions of the corresponding traffic light element {“ traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map. The information on the shape of the traffic signal means information on the three-dimensional shape of the traffic signal.”, Col. 4, Ln. 61 - Col. 5, Ln. 2}; determining a route of the moving vehicle in the map according to the vehicle-map-location {vehicle moves through the intersection in Fig. 2 based on the traffic signal instructions: “the vehicle 10 is an autonomous driving vehicle or operating an autonomous driving function, the apparatus 100 for extracting signal information may determine a control command for controlling the operation (stop, start, etc.) of the vehicle 10 based on the signal information.”, Col. 12, Lns. 54-58}; selecting candidate traffic light elements from among the traffic light elements {as evident in Fig. 2, the route of vehicle M had led to an intersection with traffic signal 110, wherein analysis of signal identification is performed, as described in Col. 14, Lns. 5-12} based on a distance on the map between the route in the map and a traffic light element among the traffic light elements in the map {determining the distance to the traffic light, in order to be within the “traffic signal recognition range” is described in Col. 6, Lns. 4-14, after which identification of the specific traffic signal in front of the vehicle is achieved by consulting map database 4 and traffic signal related database 5 (Fig.1) as discussed in Col. 7, Lns. 23-27}, and selecting, from among the candidate traffic light elements, a target traffic light element as representative of the target traffic light {Region R, represented by H2xW2, Fig. 4B}; determining, based on the image of the target traffic light, a sequential lamp position of a turned-on lamp of the target traffic light {“A traffic signal recognition device 100 illustrated in FIG. 1 is mounted on a vehicle such as a passenger car and recognizes a lighting state of a traffic signal included in an image captured by a camera in the vehicle. The lighting state of the traffic signal includes at least a passing permission lighting state (for example, a green signal) and a passing prohibition lighting state (for example, a red signal). The lighting state of the traffic signal may include a transition lighting state (for example, a yellow signal) indicating a transition from the passing permission lighting state to the passing prohibition lighting state.”, Col. 3, Lns. 53-65}; determining the current traffic signal state of the target traffic light based on the isolated image of the target traffic light and based on the information about the target traffic light {traffic sign recognition unit 17, Fig. 1: “The traffic signal recognition unit 17 recognizes the lighting state of the traffic signal by performing image recognition (pattern matching considering the color, for example) in the search area R”, Col. 10, Lns. 33-36}, including determining a sequential position of each lamp region of the target traffic light {traffic signal recognition device 200, Fig. 6, can determine whether the traffic signal is of a vertical or horizontal, Col. 14, Lns. 5-10, which one skilled in the art will appreciate determines whether the green right is below or to the right, respectively, of the red traffic light}; and mapping the determined sequential position of the illuminated lamp to a traffic-signal-type stored in a signal record retrieved from the map data, the signal record indicating signal-types corresponding to respective lamp positions of the target traffic light {the combination of vehicle position recognition 11, position determination units 12, map database 4 and traffic signal related database 5, Fig. 1, enables matching of the traffic signal in front of the vehicle, Fig. 2, to the corresponding traffic signal associated with that specific location on the map: “ traffic signal related database 5 is a database that stores traffic signal data and surrounding structure data. The traffic signal data includes information on the position of the traffic signal on the map and information on the shape of the traffic signal. The information on the position of the traffic signal on the map means information on the position coordinates of the traffic signal on the map. The information on the shape of the traffic signal means information on the three-dimensional shape of the traffic signal.”, Col. 4, Ln. 61 - Col. 5, Ln. 2}; and controlling autonomous driving of the vehicle by applying a result of the determining the signal state of the traffic light {“the vehicle 10 is an autonomous driving vehicle or operating an autonomous driving function, the apparatus 100 for extracting signal information may determine a control command for controlling the operation (stop, start, etc.) of the vehicle 10 based on the signal information.”, Col. 12, Lns. 54-58}.
Kawanai does not appear to explicitly recite the limitations: determining a sequential position of each lamp region of the target traffic light by performing, using a first neural network, an inference on the image the target traffic light to identify boundaries of lamp regions and to generate sequential position data representing a vertical order of the lamp regions, determining, using a second neural network, an illuminated lamp among the lamp regions by computing an average intensity of pixels on respective vertical line segments within each lamp region to generate an intensity profile, wherein the generating the intensity profile comprises generating, based on the average intensity of the pixels on the respective vertical line segments, the intensity profile representing a relationship between distance across the target traffic light and the average intensity, and wherein the determining the illuminated lamp among the lamp regions comprises: determining a maximum one of the average intensities; determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value; and
selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio.
However, Napanda explicitly recites the limitations: determining a sequential position of each lamp region of the target traffic light by performing, using a first neural network {2123, Fig. 1}, an inference on the image the target traffic light to identify boundaries of lamp regions and to generate sequential position data representing a vertical order of the lamp regions {first machine learning model 2123, Fig. 1, locates and positions a bounding box (1605, Fig. 1B) around the traffic signal (and thus identifies traffic signal orientation): “Position processor 2103 can provide filtered data 1603 to first machine learning model (MLM) 2123. First MLM 2123 can provide bounding boxes 1605 for objects located in sensor images 1602 (FIG. 1A)…First MLM 2123 (FIG. 1) can detect objects in images using, for example, but not limited to, a single deep neural network having a unified framework for training and inference….Bounding box processor 2105 (FIG. 1) can determine the traffic light of interest to the AV based on map point 1601.”, ¶[0033-0034]}; determining, using a second neural network {2115, Fig. 1, and “a second machine learning model to determine traffic light state probabilities, providing either the bounding boxes or the tracked candidate traffic lights to the second machine learning model, and receiving inferences from the model. Method 1950 can include updating 1963 tracking data, and publishing 1965 the probabilities of the traffic light states determined from the inferences”, ¶[0057]}, an illuminated lamp among the lamp regions by computing an average intensity of pixels on respective vertical line segments within each lamp region to generate an intensity profile {image processing to provide high quality intensity mapping of lamp areas: “if the feature is a traffic light, the colors of the bulbs are expected to be red, yellow, and green. The intensity of the colors can be increased if necessary, and noise can be removed, both by conventional methods. For example, noise can be removed by applying moving averages, moving averages with non-uniform weight…mean filter, median filter, and a bilateral filter. After noise is removed, a dilation (adding pixels to the boundary of the feature) followed by erosion (removing pixels from the feature boundary) (a morphological closing) can be applied to enlarge boundaries of foreground regions and shrink background color holes in the regions. The result is the filling in of small background color holes in image data.”, ¶[0033]}, wherein the generating the intensity profile comprises generating, based on the average intensity of the pixels on the respective vertical line segments, the intensity profile representing a relationship between distance across the target traffic light and the average intensity {having generated the quality intensity mapping of the traffic light bulb regions just described with respect to ¶[0033], one skilled in the art will appreciate that standard graphical interpretation techniques, such as taking averages or restricting attention to only part of the entire mapped area, can then be used.}.
The combination of Kawanai and Napanda does not appear to explicitly recite the limitations: wherein the determining the illuminated lamp among the lamp regions comprises:
determining a maximum one of the average intensities; determining a threshold intensity value based on the maximum one of the average intensities and a threshold ratio; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value; and selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio.
However, Kamioka explicitly recites the limitations: wherein the determining the illuminated lamp among the lamp regions comprises: determining a maximum one of the average intensities {histogram (i.e., number of pixels vs. brightness) of a light source in Fig. 2C, reflective object in Fig. 3C, a luminous object in Fig. 4B, wherein each histogram interval is effectively an average of multiple brightness values, and the interval with the highest number of pixels or the highest brightness can be deemed a maximum}; determining a threshold intensity value based on the maximum one of the average intensities {“threshold α” (i.e., alpha) in Figs. 2C and 3C} and a threshold ratio {“FIGS. 9A and 10A show the domains which include pixels having brightness 90% of the brightest pixel extracted (hereafter called “the 90% domain”) while FIGS. 9B and 10B show the domains which include pixels having brightness 60% of the brightest pixel extracted (hereafter called “the 60% domain”).”, Col. 8, Lns. 22-27}; for each lamp region, determining a number of the respective vertical line segments having an average intensity greater than or equal to the threshold intensity value {with regard to Figs. 2C&3C, threshold-alpha: “In the histograms (FIGS. 2C and 3C), the threshold α for determining that it is the domain of the light source is shown. Regarding the group classified as having brightness less than this threshold α will be ignored when distribution of the number of pixels is considered.”, Col. 5, Lns. 40-44}; and selecting, as the illuminated lamp, a lamp region corresponding to a peak point of the intensity profile and for which the determined number satisfies the threshold ratio {Figs. 9A&10A represent the distinction between highly illuminated areas (white) and low illuminated areas (black) in which the size of the area is depends on the criteria or threshold employed to establish the demarcation between the two regions; ibid, Col. 8, Lns. 22-27}.
Regarding Claim 20, the combination of Kawanai, Napanda and Kamioka discloses all the limitations of Claim 19, as discussed supra. In addition, Kawanai recites the limitations: wherein the determined current traffic signal state is used to control an autonomous or assisted driving system that is at least partially controlling driving of the vehicle {“In S24, the ECU 10 determines whether or not the lighting state of the traffic signal can be recognized using the traffic signal recognition unit 17. If it is determined that the lighting state of the traffic signal can be recognized (YES in S24), the ECU 10 makes the process proceed to S26.”, Col. 12, Lns. 46-50, and “In S26, the ECU 10 outputs the result of recognition of the traffic signal. The ECU 10 outputs the result of recognition of the traffic signal to an autonomous driving system of a vehicle connected to the traffic signal recognition device 100.”, Col. 12, Lns. 54-58}.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RICHARD EDWIN GEIST whose telephone number is (703)756-5854. The examiner can normally be reached Monday-Friday, 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Christian Chace can be reached at (571) 272-4190. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.E.G./Examiner, Art Unit 3665
/CHRISTIAN CHACE/Supervisory Patent Examiner, Art Unit 3665
Prosecution Insights