DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
This action is in response to the amendments and remarks filed on 09/24/2025.
Claims 1-8, 10-11, and 15-20 are pending.
Corrective Actions by Applicant
Claims 1-5, 8, 10-11, and 15-18 have been amended.
Claims 9 and 12-14 have been canceled.
Response to Arguments
The examiner has fully considered Applicant’s presented arguments.
The examiner does not object to the amendment to paragraph 0002 of the originally filed specification, as it does not raise any issues of new matter.
On page 1 of the remarks, Applicant argues that the amendments to claims 8 and 17-18 overcome the objections to claims 8 and 17-18. This is persuasive. The objections to claims 8 and 17-18 have been withdrawn.
On pages 1-2 of the remarks, Applicant argues that the amendments to claims 1 and 11 overcome the nonstatutory double patenting rejections of claims 1 and 11. This is persuasive. The nonstatutory double patenting rejections of claims 1 and 11 have been withdrawn.
On pages 2-5 of the remarks, Applicant argues that the amendments to independent claims 1, 11, and 18 overcome the 35 U.S.C. 101 rejections of claims 1-8 and 11-20. This is persuasive. The previous 35 U.S.C. 101 rejections of claims 1-8 and 11-20 have been withdrawn.
On pages 5-7 of the remarks, Applicant argues that the amendments to claims 1 and 11 overcome the 35 U.S.C. 103 rejections of claims 1, 8-11, and 17 in view of Ben Yaacov and Creusot. This is persuasive. The previous 35 U.S.C. 103 rejections of claims 1, 8-11, and 17 in view of Ben Yaacov and Creusot have been withdrawn.
On pages 7-9 of the remarks, Applicant argues that Yan Lu fails to remedy the deficiencies of Ben Yaacov and Creusot regarding independent claims 1, 11, and 18. This is persuasive.
On pages 10-11 of the remarks, Applicant argues that Kwant fails to remedy the deficiencies of Ben Yaacov, Creusot, or Yan Lu regarding amended independent claims 1, 11, and 18. This is persuasive.
Although Kwant discloses generating a first segmentation mask based at least in part on the image (see figures 11A-11B and paragraphs 0067-0068, where a road sign 1103 is first photographed and segmented to obtain a polygonal representation 1111), Kwant fails to disclose generating a second segmentation mask based on four points which correspond to four corners of the first segmentation mask (see figure 11B and paragraphs 0067-0068, where the first mask is compared to various preexisting, known segmentations, such as 1113a or 1113b using the Jaccard index to obtain a normalized fitness score; although features of the mask, such as corners, are used in this comparison, Kwant’s second mask is not generated, but rather retrieved from a preexisting set of masks).
PNG
media_image1.png
470
687
media_image1.png
Greyscale
Figure 7A of Yang Lu et al.
On pages 11-13 of the remarks, Applicant argues that Yang Lu fails to remedy the deficiencies of Ben Yaacov, Creusot, Yan Lu, or Kwant regarding amended independent claim 18. The examiner respectfully disagrees.
Although the two vanishing points in figure 7A of Yang Lu are assigned the same reference number of 710, the examiner argues that the vanishing points are still distinct. The vanishing point on the right-hand side of figure 7A originates from dashed lines extending from two distinct horizontal edges of the object, whereas the vanishing point on the left-hand side of figure 7A originates from dashed lines extending from two distinct vertical edges of the object. Thus, Yang Lu discloses identify a first vanishing point of the traffic sign based at least in part on two horizontal edges of the plurality of edges and identify a second vanishing point of the traffic sign based at least in part on two vertical edges of the plurality of edges, as recited by amended claim 18.
Furthermore, Yang Lu discloses determine a pan angle and a tilt angle of the traffic sign based at least in part on the first vanishing point and the second vanishing point (see figure 7B and column 7, lines 18-31, where the vanishing points can be used to determine yaw/pan angles and roll/pitch/tilt angles).
The examiner notes that, although the object in Yang Lu’s disclosure is not a traffic sign, it still would have been obvious to one of ordinary skill in the art to apply Yang Lu’s method to Ben Yaacov’s, as the object is still rectangular and Yang Lu’s method allows for automatic determination of orientations of objects in images (see Yang Lu column 1, lines 47-57), such as traffic signs.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-8, 10-11, and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Ben Yaacov et al. (U.S. Patent US-11380110-B1) in view of Creusot (U.S. Publ. US-2017/0364759-A1), Mayer (U.S. Patent US-10325374-B1), and Yang Lu et al. (U.S. Patent US-10636137-B1).
Regarding claim 1, Ben Yaacov discloses a system for determining a relevance of a traffic sign for a vehicle (see figure 2), the system comprising:
at least one vehicle camera configured to provide a view of an environment surrounding the vehicle (see figure 1, vehicle 100 and cameras 210, as well as column 5, lines 16-24);
and a vehicle controller in electrical communication with the at least one vehicle (see figure 2, controller 204 and camera 210, as well as column 5, line 63 to column 6, line 12) camera
capture an image using the at least one vehicle camera (see figure 4, step 410 and column 12, lines 51-55);
determine a pan angle and a tilt angle of the traffic sign (see figure 4, steps 430-440 and column 12, line 60 to col 13, line 13, where azimuth and elevation data are used to obtain a global orientation, which includes a yaw angle, which is analogous to a pan angle, and pitch and roll angles, both of which can be considered tilt angles)
and determine a location of the traffic sign based at least in part on the location of the vehicle (see column 1, lines 54-67, where the 3D position of the traffic sign is determined; see column 2, lines 20-29, where this position includes Z, which is the forward distance from the camera attached to the vehicle to the traffic sign).
Ben Yaacov fails to disclose a global navigation satellite system (GNSS);
a vehicle controller in electrical communication with the at least one vehicle camera and the GNSS (emphasis added via underline);
determine the relevance of the traffic sign based at least in part on the pan angle and the tilt angle of the traffic sign; determine a location of the vehicle using the GNSS; and save the relevance of the traffic sign and the location of the traffic sign in a non-transitory memory of the vehicle controller in response to determining that the traffic sign is relevant.
Pertaining to the same field of endeavor, Creusot discloses a global navigation satellite system (GNSS) (see paragraph 0046, where sensing devices include global positioning systems that provide geographic locations of the vehicle);
a vehicle controller in electrical communication with the at least one vehicle camera and the GNSS (see paragraph 0046);
determine the relevance of the traffic sign based at least in part on the pan angle and the tilt angle of the traffic sign (see figure 7 and paragraph 0072, where the stop sign is determined to be not intended for the vehicle, or not relevant, based on the sign’s rotation; this includes tilt angles, with rotation of sign 702, and pan angles, with rotation of signs 703 and 704);
determine a location of the vehicle using the GNSS (see paragraph 0046, where sensing devices include global positioning systems that provide geographic locations of the vehicle);
and save the relevance of the traffic sign and the location of the traffic sign in a non-transitory memory of the vehicle controller in response to determining that the traffic sign is relevant (see figure 1, data storage device 32 and paragraph 0048; see figure 1, computer-readable storage device or media 46 and paragraph 0049; see paragraph 0078, where it is necessary for the presence of the traffic sign and traffic control person to be stored in one of these memory devices for it to be transmitted to a server).
Ben Yaacov and Creusot are considered analogous art, as they are both directed to analysis of traffic signs in images for autonomous vehicle control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Creusot into Ben Yaacov because doing so enables determining what traffic signs are intended to direct the movement of the present vehicle (see Creusot paragraph 0070), and also allows for storing data in conjunction with providing an autonomous driving system (see Creusot paragraph 0060).
Ben Yaacov in view of Creusot fails to further disclose generate a first segmentation mask based at least in part on the image; identify the traffic sign in the image based at least in part on the first segmentation mask and a second segmentation mask, wherein the second segmentation mask is generated based on four points which correspond to four corners of the first segmentation mask, and wherein the second segmentation mask includes a plurality of edges. In other words, Ben Yaacov in view of Creusot does disclose segmentation of traffic signs, just not via the above recited image processing method.
Pertaining to the same field of endeavor, Mayer discloses generate a first segmentation mask based at least in part on the image (see figure 3 & column 5, line 52 to column 6, line 35, where a canny edge map of a rectangular object can be generated and edges considered);
identify the traffic sign in the image based at least in part on the first segmentation mask and a second segmentation mask (see column 4, lines 37-53, where the region of the image within the two segmentation masks is then analyzed to identify/classify the object),
wherein the second segmentation mask is generated based on four points which correspond to four corners of the first segmentation mask, and wherein the second segmentation mask includes a plurality of edges (see figure 4, figure 5, step 5 and column 6, lines 36-50, where candidate boundary lines, such as 402A1-402D2, can be overlaid on the first mask; column 7, lines 23-27 specifies that the final second mask is chosen by the combination of four edges and the four corners that the edges make).
Ben Yaacov and Mayer are considered analogous art, as they are both directed to image analysis for segmentation of polygonal image regions. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Mayer into Ben Yaacov and Creusot because doing so enables leveraging the shape of rounded corners of rectangular objects for improved segmentation of rectangular objects (see Mayer column 3, lines 51-60).
Ben Yaacov in view of Creusot and Mayer fails to further disclose determine a pan angle and a tilt angle of the traffic sign based at least in part on the plurality of edges (emphasis added via underline).
Pertaining to the same field of endeavor, Yang Lu discloses determine a pan angle and a tilt angle of the traffic sign based at least in part on the plurality of edges (see figure 7B and column 7, lines 18-31, where the vanishing points determined from edges can be used to determine yaw/pan angles and roll/pitch/tilt angles).
Ben Yaacov and Yang Lu are considered analogous art, as they are both directed to determining orientations of objects from images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Yang Lu into Ben Yaacov, Creusot, and Mayer because doing so allows for automatic determination of orientations of objects in images (see Yang Lu column 1, lines 47-57).
Regarding claim 2, Ben Yaacov in view of Creusot and Yang Lu fails to disclose the limitations of claim 2.
Pertaining to the same field of endeavor, Mayer discloses wherein to identify the traffic sign in the image, the vehicle controller is further programmed to: identify the traffic sign based at least in part on the plurality of edges (see column 4, lines 37-53, where the region of the image within the two segmentation masks is then analyzed to identify/classify the object).
Ben Yaacov and Mayer are considered analogous art, as they are both directed to image analysis for segmentation of polygonal image regions. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Mayer into Ben Yaacov, Creusot, and Yang Lu because doing so enables leveraging the shape of rounded corners of rectangular objects for improved segmentation of rectangular objects (see Mayer column 3, lines 51-60).
Regarding claim 3, Ben Yaacov in view of Creusot and Yang Lu discloses wherein to generate the first segmentation mask, the vehicle controller is further programmed to: extract a region of interest of the image using a deep learning model, wherein the region of interest includes an object (see Ben Yaacov figure 4, step 420 and column 12, lines 55-59, where bounding box data representing the location and dimensions of traffic signs in the image data are generated).
Ben Yaacov in view of Creusot and Yang Lu fails to disclose and generate the first segmentation mask of the region of interest, wherein the first segmentation mask includes a portion of the region of interest having the object.
Pertaining to the same field of endeavor, Mayer discloses and generate the first segmentation mask of the region of interest, wherein the first segmentation mask includes a portion of the region of interest having the object (see figure 3 & column 5, line 52 to col 6, line 35, where a canny edge map of a rectangular object can be generated and edges considered).
Ben Yaacov and Mayer are considered analogous art, as they are both directed to image analysis for segmentation of polygonal image regions. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Mayer into Ben Yaacov, Creusot, and Yang Lu because doing so enables leveraging the shape of rounded corners of rectangular objects for improved segmentation of rectangular objects (see Mayer column 3, lines 51-60).
Regarding claim 4, Ben Yaacov in view of Creusot and Yang Lu fails to disclose the limitations of claim 4.
Pertaining to the same field of endeavor, Mayer discloses wherein to generate the second segmentation mask, the vehicle controller is further programmed to: determine the four points which correspond to the four corners of the first segmentation mask; identify the plurality of edges, wherein a first terminus and a second terminus of each of the plurality of edges is one of the four points, and wherein the plurality of edges form a closed polygon; and generate the second segmentation mask, wherein the second segmentation mask is an area enclosed by the plurality of edges (see figure 4, figure 5, step 5, and column 6, lines 36-50, where candidate boundary lines, such as 402A1-402D2, can be overlaid on the first mask; column 7, lines 23-27 specifies that the final second mask is chosen by the combination of four edges and the four corners that the edges make).
Ben Yaacov and Mayer are considered analogous art, as they are both directed to image analysis for segmentation of polygonal image regions. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Mayer into Ben Yaacov, Creusot, and Yang Lu because doing so enables leveraging the shape of rounded corners of rectangular objects for improved segmentation of rectangular objects (see Mayer column 3, lines 51-60).
Regarding claim 7, Ben Yaacov in view of Creusot and Mayer fails to disclose the limitations of claim 7.
Pertaining to the same field of endeavor, Yang Lu discloses wherein to determine the pan angle and the tilt angle of the traffic sign, the vehicle controller is further programmed to: identify a first vanishing point of the traffic sign based at least in part on the plurality of edges; identify a second vanishing point of the traffic sign based at least in part on the plurality of edges (see figure 7A, vanishing points 710, and column 15, lines 24-42, where edges of an object are projected to find intersection points that correspond to vanishing points);
and determine the pan angle and the tilt angle of the traffic sign based at least in part on the first vanishing point and the second vanishing point (see figure 7B and column 7, lines 18-31, where the vanishing points can be used to determine yaw/pan angles and roll/pitch/tilt angles).
Ben Yaacov and Yang Lu are considered analogous art, as they are both directed to determining orientations of objects from images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Yang Lu into Ben Yaacov, Creusot, and Mayer because doing so allows for automatic determination of orientations of objects in images (see Yang Lu column 1, lines 47-57).
Regarding claim 8, Ben Yaacov in view of Mayer and Yang Lu fails to disclose the limitations of claim 8.
Pertaining to the same field of endeavor, Creusot discloses wherein to determine the relevance of the traffic sign, the vehicle controller is further programmed to: compare the pan angle of the traffic sign to a predetermined pan angle threshold; compare the tilt angle of the traffic sign to a predetermined tilt angle threshold (see figure 7 and paragraph 0072, where the pan and tilt angles of a traffic sign are considered to determine if the sign is adequately facing the vehicle);
determine the relevance of the traffic sign to be irrelevant in response to determining that at least one of: the pan angle of the traffic sign is greater than or equal to the predetermined pan angle threshold (see figure 7, signs 703-704 and paragraph 0072, where if the pan angle of the sign is close to a 90-degree angle with sign 703, or higher, such as sign 704 facing the opposite direction, the sign is not relevant) or the tilt angle of the traffic sign is greater than or equal to the predetermined tilt angle threshold (see figure 7, sign 702 and paragraph 0072, where if the sign 702 is held an oblique upward angle, the sign is not relevant);
and determine the relevance of the traffic sign to be relevant in response to determining that: the pan angle of the traffic sign is less than the predetermined pan angle threshold and the tilt angle of the traffic sign is less than the predetermined tilt angle threshold (see figure 7, sign 701 and paragraph 0072, where sign 701 is considered relevant, as it has lower pan and tilt angles than non-relevant signs 702-704).
Ben Yaacov and Creusot are considered analogous art, as they are both directed to analysis of traffic signs in images for autonomous vehicle control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Creusot into Ben Yaacov, Mayer, and Yang Lu because doing so enables determining what traffic signs are intended to direct the movement of the present vehicle (see Creusot paragraph 0070).
Regarding claim 10, Ben Yaacov in view of Mayer and Yang Lu discloses a vehicle communication system in electrical communication with the vehicle controller (see column 6, lines 51-63, where the interface 234 allows the controller to communicate with outside systems or components).
Ben Yaacov in view of Mayer and Yang Lu fails to disclose wherein the vehicle controller is further programmed to: transmit the relevance of the traffic sign and the location of the traffic sign to a remote server system using the vehicle communication system.
Pertaining to the same field of endeavor, Creusot discloses wherein the vehicle controller is further programmed to: transmit the relevance of the traffic sign and the location of the traffic sign to a remote server system using the vehicle communication system (see paragraph 0078, where the validity of the traffic sign is determined, and then the presence of the traffic sign and a traffic control person can be transmitted to a server).
Ben Yaacov and Creusot are considered analogous art, as they are both directed to analysis of traffic signs in images for autonomous vehicle control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Creusot into Ben Yaacov, Mayer, and Yang Lu because doing so allows for the server system to communicate with users and dispatch autonomous vehicles (see Creusot paragraph 0057).
Regarding claim 11, Ben Yaacov discloses a method for determining a relevance of a traffic sign (see figure 4) for an autonomous vehicle (see column 2, lines 52-55), the method comprising:
capturing an image using at least one vehicle camera (see figure 4, step 410 and column 12, lines 51-55);
identifying an object in the image; extracting a region of interest of the image using a deep learning model, wherein the region of interest includes the object (see figure 4, step 420 and column 12, lines 55-59, where bounding box data representing the location and dimensions of traffic signs in the image data are generated);
and determining a pan angle and a tilt angle of the traffic sign in response to determining that the object is the traffic sign (see figure 4, steps 430-440 and column 12, line 60 to column 13, line 13, where azimuth and elevation data are used to obtain a global orientation, which includes a yaw angle, which is analogous to a pan angle, and pitch and roll angles, both of which can be considered tilt angles).
Ben Yaacov fails to disclose determining the relevance of the traffic sign based at least in part on the pan angle and the tilt angle of the traffic sign; and controlling a path of the autonomous vehicle based at least in part on the relevance of the traffic sign.
Pertaining to the same field of endeavor, Creusot discloses determining the relevance of the traffic sign based at least in part on the pan angle and the tilt angle of the traffic sign (see figure 7 and paragraph 0072, where the stop sign is determined to be not intended for the vehicle, or not relevant, based on the sign’s rotation; this includes tilt angles, with rotation of sign 702, and pan angles, with rotation of signs 703 and 704);
and controlling a path of the autonomous vehicle based at least in part on the relevance of the traffic sign (see paragraphs 0064 and 0066).
Ben Yaacov and Creusot are considered analogous art, as they are both directed to analysis of traffic signs in images for autonomous vehicle control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Creusot into Ben Yaacov because doing so enables the autonomous vehicle to follow appropriate requirements imposed by the traffic signs (see Creusot paragraph 0066).
Ben Yaacov in view of Creusot fails to further disclose and generating a first segmentation mask of the region of interest, wherein the first segmentation mask includes a portion of the region of interest having the object; determining four points which correspond to four corners of the first segmentation mask; identifying a plurality of edges of the object, wherein a first terminus and a second terminus of each of the plurality of edges is one of the four points, and wherein the plurality of edges form a closed polygon; generating a second segmentation mask based at least in part on the four points of the first segmentation mask, wherein the second segmentation mask is an area enclosed by the plurality of edges; determining the object to be the traffic sign based at least in part on the plurality of edges. In other words, Ben Yaacov in view of Creusot does disclose segmentation of traffic signs, just not via the above recited image processing method.
Pertaining to the same field of endeavor, Mayer discloses generating a first segmentation mask of the region of interest, wherein the first segmentation mask includes a portion of the region of interest having the object (see figure 3 and column 5, line 52 to column 6, line 35, where a canny edge map of a rectangular object can be generated and edges considered);
determining four points which correspond to four corners of the first segmentation mask; identifying a plurality of edges of the object, wherein a first terminus and a second terminus of each of the plurality of edges is one of the four points, and wherein the plurality of edges form a closed polygon; generating a second segmentation mask based at least in part on the four points of the first segmentation mask, wherein the second segmentation mask is an area enclosed by the plurality of edges (see figure 4, figure 5, step 5, and column 6, lines 36-50, where candidate boundary lines, such as 402A1-402D2, can be overlaid on the first mask; column 7, lines 23-27 specifies that the final second mask is chosen by the combination of four edges and the four corners that the edges make);
and determining the object to be the traffic sign based at least in part on the plurality of edges (see column 4, lines 37-53, where the region of the image within the two segmentation masks is then analyzed to identify/classify the object).
Ben Yaacov and Mayer are considered analogous art, as they are both directed to image analysis for segmentation of polygonal image regions. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Mayer into Ben Yaacov and Creusot because doing so enables leveraging the shape of rounded corners of rectangular objects for improved segmentation of rectangular objects (see Mayer column 3, lines 51-60).
Ben Yaacov in view of Creusot and Mayer fails to further disclose wherein the pan angle and the tilt angle are determined based at least in part on the plurality of edges.
Pertaining to the same field of endeavor, Yang Lu discloses wherein the pan angle and the tilt angle are determined based at least in part on the plurality of edges (see figure 7B and column 7, lines 18-31, where the vanishing points determined from edges can be used to determine yaw/pan angles and roll/pitch/tilt angles).
Ben Yaacov and Yang Lu are considered analogous art, as they are both directed to determining orientations of objects from images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Yang Lu into Ben Yaacov, Creusot, and Mayer because doing so allows for automatic determination of orientations of objects in images (see Yang Lu column 1, lines 47-57).
Regarding claim 16, Ben Yaacov in view of Creusot, Mayer, and Yang Lu discloses claim 16 as applied to claim 7 above.
Regarding claim 17, Ben Yaacov in view of Creusot, Mayer, and Yang Lu discloses claim 17 as applied to claim 8 above.
Regarding claim 18, Ben Yaacov discloses a system for determining a relevance of a traffic sign (see figure 4) for an autonomous vehicle (see column 2, lines 52-55), the system comprising:
at least one vehicle camera configured to provide a view of an environment surrounding the autonomous vehicle (see figure 1, vehicle 100 and cameras 210, and column 5, lines 16-24);
and a vehicle controller in electrical communication with the at least one vehicle camera, (see figure 2, controller 204 and camera 210 and column 5, line 63 to col 6, line 12) wherein the vehicle controller is programmed to:
capture an image using the at least one vehicle camera (see figure 4, step 410 and column 12, lines 51-55);
extract a region of interest of the image using a deep learning model, wherein the region of interest includes an object (see figure 4, step 420 and column 12, lines 55-59, where bounding box data representing the location and dimensions of traffic signs in the image data are generated);
and determine a pan angle and a tilt angle of the traffic sign (see figure 4, steps 430-440 and column 12, line 60 to column 13, line 13, where azimuth and elevation data are used to obtain a global orientation, which includes a yaw angle, which is analogous to a pan angle, and pitch and roll angles, both of which can be considered tilt angles)
Ben Yaacov fails to disclose compare the pan angle of the traffic sign to a predetermined pan angle threshold; compare the tilt angle of the traffic sign to a predetermined tilt angle threshold; determine the relevance of the traffic sign to be irrelevant in response to determining that at least one of: the pan angle of the traffic sign is greater than or equal to the predetermined pan angle threshold or the tilt angle of the traffic sign is greater than or equal to the predetermined tilt angle threshold; determine the relevance of the traffic sign to be relevant in response to determining that: the pan angle of the traffic sign is less than the predetermined pan angle threshold and the tilt angle of the traffic sign is less than the predetermined tilt angle threshold; and control a path of the autonomous vehicle based at least in part on the relevance of the traffic sign.
Pertaining to the same field of endeavor, Creusot discloses compare the pan angle of the traffic sign to a predetermined pan angle threshold; compare the tilt angle of the traffic sign to a predetermined tilt angle threshold (see figure 7 and paragraph 0072, where the pan and tilt angles of a traffic sign are considered to determine if the sign is adequately facing the vehicle);
determine the relevance of the traffic sign to be irrelevant in response to determining that at least one of: the pan angle of the traffic sign is greater than or equal to the predetermined pan angle threshold (see figure 7, signs 703-704 and paragraph 0072, where if the pan angle of the sign is close to a 90-degree angle with sign 703, or higher, such as sign 704 facing the opposite direction, the sign is not relevant) or the tilt angle of the traffic sign is greater than or equal to the predetermined tilt angle threshold (figure 7, sign 702 and paragraph 0072, where if the sign 702 is held an oblique upward angle, the sign is not relevant);
determine the relevance of the traffic sign to be relevant in response to determining that: the pan angle of the traffic sign is less than the predetermined pan angle threshold and the tilt angle of the traffic sign is less than the predetermined tilt angle threshold (see figure 7, sign 701 and paragraph 0072, where sign 701 is considered relevant, as it has lower pan and tilt angles than non-relevant signs 702-704);
and control a path of the autonomous vehicle based at least in part on the relevance of the traffic sign (see paragraphs 0064 and 0066).
Ben Yaacov and Creusot are considered analogous art, as they are both directed to analysis of traffic signs in images for autonomous vehicle control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Creusot into Ben Yaacov because doing so enables determining what traffic signs are intended to direct the movement of the present vehicle (see Creusot paragraph 0070), and also enables the autonomous vehicle to follow appropriate requirements imposed by the traffic signs (see Creusot paragraph 0066).
Ben Yaacov in view of Creusot fails to further disclose generate a first segmentation mask of the region of interest, wherein the first segmentation mask describes a portion of the region of interest including only the object, and wherein the first segmentation mask is an irregular polygon; determine four points which correspond to four corners of the first segmentation mask; identify a plurality of edges of the object, wherein a first terminus and a second terminus of each of the plurality of edges is one of the four points, and wherein the plurality of edges form a closed polygon; generate a second segmentation mask based at least in part on the four points, wherein the second segmentation mask is an area enclosed by the plurality of edges; determine the object to be the traffic sign based at least in part on the plurality of edges of the object.
Pertaining to the same field of endeavor, Mayer discloses generate a first segmentation mask of the region of interest, wherein the first segmentation mask describes a portion of the region of interest including only the object (see figure 3 and column 5, line 52 to column 6, line 35, where a canny edge map of a rectangular object can be generated and edges considered),
and wherein the first segmentation mask is an irregular polygon (see figure 3, where the canny edge map produces various irregularities across the edges);
determine four points which correspond to four corners of the first segmentation mask; identify a plurality of edges of the object, wherein a first terminus and a second terminus of each of the plurality of edges is one of the four points, and wherein the plurality of edges form a closed polygon; generate a second segmentation mask based at least in part on the four points, wherein the second segmentation mask is an area enclosed by the plurality of edges (see figure 4, figure 5, step 5 and column 6, lines 36-50, where candidate boundary lines, such as 402A1-402D2, can be overlaid on the first mask; column 7, lines 23-27 specifies that the final second mask is chosen by the combination of four edges and the four corners that the edges make);
and determine the object to be the traffic sign based at least in part on the plurality of edges of the object (see column 4, lines 37-53, where the region of the image within the two segmentation masks is then analyzed to identify/classify the object).
Ben Yaacov and Mayer are considered analogous art, as they are both directed to image analysis for segmentation of polygonal image regions. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Mayer into Ben Yaacov and Creusot because doing so enables leveraging the shape of rounded corners of rectangular objects for improved segmentation of rectangular objects (see Mayer column 3, lines 51-60).
Ben Yaacov in view of Creusot and Mayer fails to further disclose identify a first vanishing point of the traffic sign based at least in part on two horizontal edges of the plurality of edges; identify a second vanishing point of the traffic sign based at least in part on two vertical edges of the plurality of edges; determine a pan angle and a tilt angle of the traffic sign based at least in part on the first vanishing point and the second vanishing point.
Pertaining to the same field of endeavor, Yang Lu discloses identify a first vanishing point of the traffic sign based at least in part on two horizontal edges of the plurality of edges (see figure 7A, vanishing points 710 and column 15, lines 24-42, where the vanishing point 710 on the right-hand side of figure 7A originates from dashed lines extending from two horizontal edges of the object);
identify a second vanishing point of the traffic sign based at least in part on two vertical edges of the plurality of edges (see figure 7A, vanishing points 710 and column 15, lines 24-42, where the vanishing point 710 on the left-hand side of figure 7A originates from dashed lines extending form two vertical edges of the object);
determine a pan angle and a tilt angle of the traffic sign based at least in part on the first vanishing point and the second vanishing point (see figure 7B and column 7, lines 18-31, where the vanishing points can be used to determine yaw/pan angles and roll/pitch/tilt angles).
Ben Yaacov and Yang Lu are considered analogous art, as they are both directed to determining orientations of objects from images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Yang Lu into Ben Yaacov, Creusot, and Mayer because doing so allows for automatic determination of orientations of objects in images (see Yang Lu column 1, lines 47-57).
Claims 5-6, 15, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Ben Yaacov et al. (U.S. Patent US-11380110-B1) in view of Creusot (U.S. Publ. US-2017/0364759-A1), Mayer (U.S. Patent US-10325374-B1), and Yang Lu et al. (U.S. Patent US-10636137-B1), and further in view of Kwant et al. (U.S. Publ. US-2019/0073774-A1).
Regarding claim 5, Ben Yaacov in view of Creusot, Mayer, and Yang Lu fails to disclose the limitations of claim 5.
Pertaining to the same field of endeavor, Kwant discloses wherein to identify the traffic sign, the vehicle controller is further programmed to: determine a normalized fitness score of the second segmentation mask with respect to the first segmentation mask (see figures 11A-11B and paragraphs 0067-0068, where a road sign 1103 is first photographed and segmented to obtain a polygonal representation 1111; this is compared to various preset segmentations, such as 1113a or 1113b using the Jaccard index to obtain a normalized fitness score);
compare the normalized fitness score to a predetermined normalized fitness score threshold; and determine the object to be the traffic sign in response to determining that the normalized fitness score is greater than or equal to the predetermined normalized fitness score threshold (see paragraph 0068, where the Jaccard index value can be compared to a threshold to determine if the polygons are similar, and thus if the predicted road sign segmentation and pose are accurate).
Ben Yaacov and Kwant are considered analogous art, as they are both directed to analysis of traffic signs in images for autonomous vehicle control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Kwant into Ben Yaacov, Creusot, Mayer, and Yang Lu because confirming the road sign prediction allows for localizing the vehicle (see Kwant paragraph 0068).
Regarding claim 6, Ben Yaacov in view of Creusot, Mayer, and Yang Lu fails to disclose the limitations of claim 6.
Pertaining to the same field of endeavor, Kwant discloses wherein to determine the normalized fitness score, the vehicle controller is further programmed to: determine an intersection area between the first segmentation mask and the second segmentation mask; determine a union area between the first segmentation mask and the second segmentation mask; and determine the normalized fitness score, wherein the normalized fitness score is equal to the intersection area divided by the union area (see figures 11A-11B and paragraphs 0067-0068, where a road sign 1103 is first photographed and segmented to obtain a polygonal representation 1111; this is compared to various preset segmentations, such as 1113a or 1113b using the Jaccard index, where the area of intersection of the two polygons is divided by the area of their union).
Ben Yaacov and Kwant are considered analogous art, as they are both directed to analysis of traffic signs in images for autonomous vehicle control. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Kwant into Ben Yaacov, Creusot, Mayer, and Yang Lu because the Jaccard index value can be compared to a threshold to determine if the polygons are similar, and thus if the segmentation and predicted road sign pose is accurate (see Kwant paragraph 0068).
Regarding claim 15, Ben Yaacov in view of Creusot, Mayer, Yang Lu, and Kwant discloses claim 15 as applied to claims 5 and 6 above.
Regarding claim 19, Ben Yaacov in view of Creusot, Mayer, Yang Lu, and Kwant discloses claim 19 as applied to claims 5 and 6 above.
Regarding claim 20, Ben Yaacov in view of Creusot, Mayer, and Kwant fails to disclose the limitations of claim 20.
Pertaining to the same field of endeavor, Yang Lu discloses wherein to determine the pan angle and the tilt angle of the traffic sign, the vehicle controller is further programmed to: identify a first vanishing point of the traffic sign based at least in part on the plurality of edges; identify a second vanishing point of the traffic sign based at least in part on the plurality of edges (see figure 7A, vanishing points 710, and column 15, lines 24-42, where edges of an object are projected to find intersection points that correspond to vanishing points);
and determine the pan angle and the tilt angle of the traffic sign based at least in part on the first vanishing point and the second vanishing point (see figure 7B and column 7, lines 18-31, where the vanishing points can be used to determine yaw/pan angles and roll/pitch/tilt angles).
Ben Yaacov and Yang Lu are considered analogous art, as they are both directed to determining orientations of objects from images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have integrated the teachings of Yang Lu into Ben Yaacov, Creusot, Mayer, and Kwant because doing so allows for automatic determination of orientations of objects in images (see Yang Lu column 1, lines 47-57).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS JOHN HELCO whose telephone number is (703)756-5539. The examiner can normally be reached on Monday-Friday from 9:00 AM to 5:00 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella, can be reached at telephone number 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/NICHOLAS JOHN HELCO/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights