DETAILED ACTION
This Action is in response to Applicant's response filed on 08/26/2025. Claims 1-4, 6-13, 15-20 are still pending in the present application. This Action is made FINAL.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/16/2025 was filed and is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
Applicant’s amendments to claims 1, 10 and 19 have been considered but are moot in view of the new ground(s) of rejection in view Jang (US 10,650,529 B2) and Westmacott et. al (US 11,900,627 B2).
Applicant’s cancelation of claims 5 and 14 and consequently changes of dependency of claims 6 and 15 are acknowledged.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-13, and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (The World Is Changing: Finding Changes on the Street, hereinafter Chen) in view of Tatourian et al. (US 9,858,478 B2, hereinafter Tatourian), Jang (US 10,650,529 B2, hereinafter Jang) and Westmacott et. al (US 11,900,627 B2, hereinafter Westmacott).
Regarding claims 1, 10, and 19 Chen discloses
Claim 1: A method for detecting a road change, comprising:
Claim 10: An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute actions including:
Claim 19: A non-transitory computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to execute actions including:
obtaining a road image comprising a road to be detected (P. 424 Section 3.1 Data Harvesting: “In countries like Taiwan, Russian, etc., dashcam are commonly installed on cars. Through crawling YouTube, we can locate many dashcam videos with rough longitude and latitude positions annotated by the users who uploaded the videos”);
extracting a road region in the road image as a target region, wherein the road region corresponds to where the road to be detected is located (P. 425 Fig. 4: “The road information is illustrated by overlaid yellow paths”);
obtaining a target geographical location of the target region (P. 424 Section 3.1 Data Harvesting: “In countries like Taiwan, Russian, etc., dashcam are commonly installed on cars. Through crawling YouTube, we can locate many dashcam videos with rough longitude and latitude positions annotated by the users who uploaded the videos”);
determining a reference region for the target region from pre-stored road regions based on the target geographical location (P. 426 Section 3.3 Reference Images from Street View: “For each dashcam video, we query Street View images within a rectangle region centered at the user provided rough longitude and latitude position”, P. 427 Candidate Reference Images Along the Path: “For each frame, we first use its holistic feature to retrieve the top K similar reference images by considering cosine similarity”);
calculating a similarity between the target region and the reference region (P. 427 Geometric Verification: “The similarity between a pair of reference image and frame can be more reliably confirmed by applying geometric verification method consisting of (1) raw SIFT keypoint matches with ratio test (i.e., the ratio between the distance of the top match and the distance of the second top match), and (2) RANSAC matching with Epipolar geometric verification. However, geometric verification is more computational expensive than calculating similarity using holistic representation”); and
However, Chen does not disclose
At least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor
A non-transitory computer-readable storage medium storing computer instructions
determining, based on the similarity, whether a passability of the road to be detected is changed;
wherein the extracting the road region in the road image as the target region comprises:
performing instance segmentation for obtaining the road region in the road image and determining one or more first regions obtained through the instance segmentation, wherein the one or more first regions includes at least one of a forward road region, a left intersection road region. or a right intersection road region;
merging, based on determining a plurality of the first regions, the plurality of first regions to obtain a second region; and
determining the road region in the road image based on the second region.
Tatourian teaches
At least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor (Col. 9 Lns. 25-30: “Embodiments may be implemented in one or a combination of hardware, firmware, and software. Embodiments may also be implemented as instructions stored on a machine-readable storage device, which may be read and executed by at least one processor to perform the operations described herein”);
A non-transitory computer-readable storage medium storing computer instructions (Col. 9 Lns. 30-32: “A machine-readable storage device may include any non-transitory mechanism for storing information in a form readable by a machine (e.g., a computer)”);
determining, based on the similarity, whether a passability of the road to be detected is changed (Col 8 Lns. 6-18: “In an example, at operation 506, the change is analyzed and a category determined for the change. For example, computer vision and image analysis may be used to examine the changed portion of the current image--as identified by the change detection module 218--to match the changed portion of the image with existing categorized imagery. The change detection module 218 may also analyze the changed portion of the image to determine what is being changed…For example, if the location in previous image data indicates a road and the current image data indicates a change to the road, the change may be categorized as a road hazard”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen with the processor and memory necessary for the functions and determining passability of a road based on similarity of a road of Tatourian to effectively increase the efficiency of analyzing changes to an area over a period of time.
However, Chen in view of Tatourian does not explicitly teach
wherein the extracting the road region in the road image as the target region comprises:
performing instance segmentation for obtaining the road region in the road image and determining one or more first regions obtained through the instance segmentation, wherein the one or more first regions includes at least one of a forward road region, a left intersection road region. or a right intersection road region;
merging, based on determining a plurality of the first regions, the plurality of first regions to obtain a second region; and
determining the road region in the road image based on the second region.
Jang teaches
determining regions that contains lanes (Fig. 7, Col. 11 Lns. 30-53: “Referring to FIG. 7, in a forward-view image 610, additional ROIs 612, 614, and 618 for which accurate lane detection is needed are set along with a basic ROI 616… the first detector 726, the second detector 728, the third detector 724, and the fourth detector 722 are a unified single detector 720. Through the first detector 726, the second detector 728, the third detector 724, and the fourth detector 722, lane pixels may be estimated, in parallel, from the basic ROI 716 and the additional ROIs 718, 714, and 712”);
merging, based on determining a plurality of the first regions, the plurality of first regions to obtain a second region; and determining the road region in the road image based on the second region (Fig. 7, Col. 11 Lns. 26-27: “FIG. 7 is a diagram illustrating an example of a method of determining a lane region based on a plurality of ROIs”, Col. 11 Lns. 59-61: Respective results 746, 748, 744, and 742 obtained through the restoring are merged into a final lane detection result image 740”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen in view of Tatourian with detecting multiple regions of road in and image and merging the determined regions to determine the road region of Jang to effectively increase the safety of autonomous driving.
However Chen in view of Tatourian and Jang does not teach
performing instance segmentation for obtaining the road region in the road image and determining one or more first regions obtained through the instance segmentation, wherein the one or more first regions includes at least one of a forward road region, a left intersection road region. or a right intersection road region.
Westmacott teaches
performing instance segmentation for obtaining the road region in the road image and determining one or more first regions obtained through the instance segmentation, wherein the one or more first regions includes at least one of a forward road region, a left intersection road region. or a right intersection road region (Fig. 25, Col. 11 Lns. 24-29: “A semi-automated annotation method for lane instances in 3D, requiring only inexpensive dash-cam equipment Road surface annotations in dense traffic scenarios despite occlusion Experimental results for road, ego-lane and lane instance segmentation using a CNN”, Col. 13 Lns. 14-20: “FIG. 25 shows an example road image (top left), including annotations for road (top right), ego-lane (bottom left) and lane instance (bottom right). Road and lanes below vehicles are annotated despite being occluded. Non-coloured parts have not been annotated, i.e. the class is not known”, Col. 33 Lns, 60-64 5.2 Lane Instance Segmentation: “The annotation of multiple distinct lanes per image, the number of which is variable across images and potentially sequences, naturally suggests an instance segmentation task against the present dataset”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen in view of Tatourian and Jang with performing instance segmentation to determine lanes in in an image of Westmacott to effectively increase the speed of identifying lanes in an image.
Regarding claims 2, 11 and 20, dependent upon claims 1, 10 and 19 respectively, Chen in view of Tatourian, Jang and Westmacott teaches all of the element as stated regarding claims 1, 10 and 19
Chen further discloses
determining first candidate regions of the target region from the pre-stored road regions based on the target geographical location (P. 427 Candidate Reference Images Along the Path: “For each frame, we first use its holistic feature to retrieve the top K similar reference images by considering cosine similarity”); and
Tatourian further teaches
determining, from the first candidate regions based on a time at which each respective region of the first candidate regions is collected, a first candidate region collected at a latest time as the reference region for the target region (Col. 7 Lns. 50-55: “In an example, the current image data is infrared image data. The current image data may be a subset of a current picture of the geographic location. For example, it may be the changes from the previous time the UAV captured a picture of the geographic location”).
Regarding claims 3 and 12, dependent upon claims 2 and 11 respectively, Chen in view of Tatourian, Jang and Westmacott teaches all of the element as stated regarding claims 2 and 11.
Chen further discloses
calculating, for each pre-stored road region of the pre-stored road regions, a matching degree between a geographical location of the respective pre-stored road region and the target geographical location; and determining the first candidate regions of the target region from the pre-stored road regions based on the calculated matching degrees (P. 427 Candidate Reference Images Along the Path: “For each frame, we first use its holistic feature to retrieve the top K similar reference images by considering cosine similarity”).
Regarding claims 4 and 13, dependent upon claims 2 and 11 respectively, Chen in view of Tatourian, Jang and Westmacott teaches all of the element as stated regarding claims 2 and 11.
Chen further discloses
after the determining the first candidate regions of the target region from the pre-stored road regions based on the target geographical location, determining second candidate regions of the target region from the first candidate regions based on a target region location of the target region in the road image (P. 427 Candidate Reference Images Along the Path: “Then, we apply geometric verification only on the retrieved images to re-order them according to the number of inlier matches which is referred to as the “confidence score””);
Tatourian further teaches
wherein the determining, from the first candidate regions based on the time at which each respective region of the first candidate regions is collected, the first candidate region collected at the latest time comprises: determining, from the second candidate regions, based on a time that each respective region of the second candidate regions is collected, a second candidate region collected at the latest time (Col. 7 Lns. 50-55: “In an example, the current image data is infrared image data. The current image data may be a subset of a current picture of the geographic location. For example, it may be the changes from the previous time the UAV captured a picture of the geographic location”, Since the second region is derived from the first region, they are using the same image, which means they taken at the “previous time”. ).
Regarding claims 7 and 16, dependent upon claims 1 and 10 respectively, Chen in view of Tatourian, Jang and Westmacott teaches all of the element as stated regarding claims 1 and 10.
Chen further discloses
extracting a first image feature of the target region and obtaining a second image feature of the reference region (P. 426 Holistic Features Representation: “For both the reference images and frames, we detect sparse SIFT keypoints and represent each keypoint using SIFT descriptor [25]. The number of keypoints in each images can be very different depending on the texture in the scene and the image quality”); and
calculating a similarity between the first image feature and the second image feature as the similarity between the target region and the reference region (P. 427 Holistic Features Representation (extended from P.426): “It has been shown in [26] that the fisher vector representation can be used to efficiently and accurately retrieve visually similar pairs of reference images and frames”, P. 427 Geometric Verification: “The similarity between a pair of reference image and frame can be more reliably confirmed by applying geometric verification method consisting of (1) raw SIFT keypoint matches with ratio test (i.e., the ratio between the distance of the top match and the distance of the second top match), and (2) RANSAC matching with Epipolar geometric verification. However, geometric verification is more computational expensive than calculating similarity using holistic representation”).
Regarding claims 8 and 17, dependent upon claims 1 and 10 respectively, Chen in view of Tatourian, Jang and Westmacott teaches all of the element as stated regarding claims 1 and 10.
Chen further discloses
determining that the passability of the road to be detected is changed based on the similarity being smaller than a similarity threshold; and determining that the passability of the road to be detected is not changed based on the similarity being greater than or equal to the similarity threshold (P. 428 Quality Prediction: “In a few rare cases, our method slightly decrease the performance. We propose the following criteria to automatically decide whether to use our results or not. We use our results only if (i) the ratio between number of inlier over the total number of pairs is above a threshold λ; or (ii) the ratio between the average of the confidence scores of our method over the average of the confidence scores of the baseline method is higher than a threshold γ”).
Regarding claims 9 and 18, dependent upon claims 1 and 10 respectively, Chen in view of Tatourian, Jang and Westmacott teaches all of the element as stated regarding claims 1 and 10.
Chen further discloses
wherein the road to be detected is an intersection road (P. 425 Fig. 4, In the crop image, the road detection extended into an intersection road).
Claims 6 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (The World Is Changing: Finding Changes on the Street, hereinafter Chen) in view of Tatourian et al. (US 9,858,478 B2, hereinafter Tatourian), Jang (US 10,650,529 B2, hereinafter Jang), Westmacott et. al (US 11,900,627 B2, hereinafter Westmacott) and Sun et al. (CN 111540010 A, hereinafter Sun).
Regarding claims 6 and 15, dependent upon claims 1 and 10 respectively, Chen in view of Tatourian, Jang and Westmacott teaches all of the element as stated regarding claims 1 and 15.
However Chen in view of Tatourian, Jang and Westmacott does not teach
determining a region in a preset shape including the second region as the road region in the road image.
Sun teaches
determining a region in a preset shape including the second region as the road region in the road image (P. 9 Para. 7: “First, the determination method of the road may be determined by using a road recognition model trained in advance, or by using a vehicle driving track”, P. 9 Para. 13: “The information on the road may include the length of the road, the shape of the road, and the like”, P. 10 Para. 2: “The shape of the road can be a straight line type road, an arc type (left turn, right turn or waiting turn) road, a U-shaped (turning around) road and the like. The representation of the road shape may be a functional expression”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have Chen in view of Tatourian, Jang and Westmacott with determining a shape of the road based on preset shapes of Sun to effectively increase the efficiency of identifying changes to the road.
Relevant Prior Art Directed to State of Art
Reschka et al. (US 2020/0310450 A1, hereinafter Reschka) is prior art not applied in the rejection(s) above. Reschka discloses systems and processes for controlling an autonomous vehicle in an environment, including a new environment in which the autonomous vehicle has not previously traveled.
Mori et al. (US 11,861,841 B2, hereinafter Mori) is prior art not applied in the rejection(s) above. Mori discloses a device, a method, or a program for estimating a lane in which a moving object is moving on a road comprises: acquiring image data obtained by the moving object imaging a range including a road region; recognizing a shape indicating the road region from the acquired image data; calculating a feature value of the road region based on the recognized shape; and estimating the lane in which the moving object is moving based on the calculated feature value.
Ma et al. (US 2020/0372262 A1, hereinafter Ma) is prior art not applied in the rejection(s) above. Ma discloses techniques for detecting whether a lane of a roadway is open or closed. Detecting a lane as being closed may include detecting an object in or near the lane, which may comprise determining a size, location, and/or classification associated with the object, and dilating the size associated with the object. The lane may be indicated as being closed if a distance between a dilated object detection and another object detection, dilated object detection, or lane extent is less than a threshold distance. The techniques may additionally or alternatively comprise determining an alternative lane shape based at least in part on one or more object detections and/or determining that one or more lanes are closed and/or uploading a lane closure and/or alternative lane shape to a central database for retrieval by/dissemination to other computing devices.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA CHEN whose telephone number is (703)756-5394. The examiner can normally be reached M-Th: 9:30 am - 4:30pm ET F: 9:30 am - 2:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, STEPHEN R KOZIOL can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J. C./ Examiner, Art Unit 2665
/Stephen R Koziol/ Supervisory Patent Examiner, Art Unit 2665