Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 3, and 9 have been considered but are moot in view of the new grounds of rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 2, 7-10, 15, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Wang) (US 2021/0374547) in view of Sikka (US 2022/0391689).
Regarding claim 1, Wang discloses a method for image processing, comprising:
receiving a plurality of image frames, wherein a traffic sign is depicted in the plurality of image frames ([0067], a video (i.e. a plurality of frames) is input, [0197], a traffic sign in an image is detected over multiple frames);
extracting image features corresponding to the traffic sign based on the plurality of image frames ([0144], [0197], object and text recognition is performed using extracted features from a traffic sign);
determining, using at least one machine learning model, an embedding based on the image features and text included on the traffic sign ([0095], [0099], text-image embedding network); and
determining, using the at least one machine learning model, a plurality of descriptors of the traffic sign based on the embedding ([0055], natural language processing is used for classifying and annotating images; [0082], [0619], [0657], labels for images are generated using natural language processing), the plurality of descriptors including a first descriptor having text indicating a type of the traffic sign and a second descriptor having text indicating the text included in the traffic sign ([0197], a first neural network identifies the sign as a traffic sign (i.e. a first characteristic) while a second neural network interprets text written on a sign (i.e. a second characteristic) and a third neural network identifies flashing lights on a sign (i.e. a third characteristic).
Wang is silent about a plurality of descriptors of the traffic sign, the plurality of natural language descriptors including a first natural language descriptor having natural language text indication a type of the traffic sign and a second natural language descriptor having natural language text indicating the text included in the traffic sign.
Sikka from the same or similar field of endeavor discloses a plurality of natural language descriptors of the traffic sign, the plurality of natural language descriptors including a first natural language descriptor having natural language text indication a type of the traffic sign and a second natural language descriptor having natural language text indicating the text included in the traffic sign ([0024], [0031], [0032], [0034], natural language descriptors of shape, color, text, and type of a stop sign is output; FIG. 3, [0049], [0083], natural language expressions that describe results (i.e. classification type) and inputs (i.e. shape, color, and text of a sign) of the rule 322 is output).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sikka into the teachings of Wang for allowing a user to more easily understand features extracted from an image.
Regarding claims 2 and 10, Wang discloses wherein the plurality descriptors is determined using a large language machine learning model of the at least one machine learning model ([0094], large scale data set with natural language processing (NLP) generated image labels; [0103], a large data set is used).
Wang is silent about the plurality of natural language descriptors.
Sikka from the same or similar field of endeavor discloses the plurality of natural language descriptors ([0024], [0031], [0032], [0034], natural language descriptors of shape, color, text, and type of a stop sign is output; FIG. 3, [0049], [0083], natural language expressions that describe results (i.e. classification type) and inputs (i.e. shape, color, and text of a sign) of the rule 322 is output).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sikka into the teachings of Wang for allowing a user to more easily understand features extracted from an image.
Regarding claims 7 and 15, Wang discloses wherein the plurality of image frames are received from at least one camera (FIG. 9B, cameras 970-998) disposed on a vehicle (900).
Regarding claims 8 and 16, Wang discloses wherein the operations further include controlling a function of a vehicle based on at least one of the plurality of descriptors ([0142], [0196], descriptors generated from reading traffic signs are used for autonomous route planning, navigation, and braking).
Wang is silent about the plurality of natural language descriptors.
Sikka from the same or similar field of endeavor discloses the plurality of natural language descriptors ([0024], [0031], [0032], [0034], a natural language descriptor of shape, color, text, and type of a stop sign is output; FIG. 3, [0049], [0083], natural language expressions that describe results (i.e. classification type) and inputs (i.e. shape, color, and text of a sign) of the rule 322 is output).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sikka into the teachings of Wang for allowing a user to more easily understand features extracted from an image.
Regarding claim 9, Wang discloses an apparatus, comprising:
a memory storing processor-readable code ([0086], a stored program is executed by a processor); and
at least one processor coupled to the memory, the at least one processor configured to execute the processor-readable code to cause the at least one processor to perform operations ([0086], a stored program is executed by a processor) including:
receiving a plurality of image frames, wherein a traffic sign is depicted in the plurality of image frames ([0067], a video (i.e. a plurality of frames) is input, [0197], a traffic sign in an image is detected);
determining, using at least one machine learning model, image features corresponding to the traffic sign based on the plurality of image frames ([0066], image feature detection using machine learning; [0144], [0197], object and text recognition is performed using extracted features from a traffic sign);
determining, using the at least one machine learning model, text features associated with the traffic sign based on the plurality of image frames ([0144], [0197], object and text recognition is performed using extracted features from a traffic sign);
determining, using the at least one machine learning model, an embedding that combines the image features and the text features ([0067], [0082], [0196], text of the traffic sign is interpreted); and
determining, using the at least one machine learning model, a plurality of descriptors of the traffic sign based on the embedding ([0055], natural language processing is used for classifying and annotating images; [0082], [0619], [0657], labels for images are generated using natural language processing), the plurality of descriptors including a first descriptor having text indication a type of the traffic sign and a second having text indicating the text included in the traffic sign ([0197], a first neural network identifies the sign as a traffic sign (i.e. a first characteristic) while a second neural network interprets text written on a sign (i.e. a second characteristic) and a third neural network identifies flashing lights on a sign (i.e. a third characteristic).
Wang is silent about a plurality of natural language descriptors of the traffic sign, the plurality of natural language descriptors including a first natural language descriptor having natural language text indication a type of the traffic sign and a second natural language descriptor having natural language text indicating the text included in the traffic sign
Sikka from the same or similar field of endeavor discloses a plurality of natural language descriptors of the traffic sign, the plurality of natural language descriptors including a first natural language descriptor having natural language text indication a type of the traffic sign and a second natural language descriptor having natural language text indicating the text included in the traffic sign ([0024], [0031], [0032], [0034], natural language descriptors of shape, color, text, and type of a stop sign is output; FIG. 3, [0049], [0083], natural language expressions that describe results (i.e. classification type) and inputs (i.e. shape, color, and text of a sign) of the rule 322 is output).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sikka into the teachings of Wang for allowing a user to more easily understand features extracted from an image.
Claim(s) 3 and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Wang) (US 2021/0374547) in view of Sikka (US 2022/0391689), and further in view of Shetty et al. (Shetty) (US 2025/0136134).
Regarding claims 3 and 11, Sikka further discloses wherein the plurality of natural
language descriptors further include a third natural language descriptor having natural language text indicating a shape of the traffic sign ([0024], [0031], [0032], [0034], a natural language descriptor of shape, color, text, and type of a stop sign is output; FIG. 3, [0049], [0083], natural language expressions that describe results (i.e. classification type) and inputs (i.e. shape, color, and text of a sign) of the rule 322 is output), and a fifth natural language descriptor having natural language text indicating a color of the traffic sign ([0024], [0031], [0032], [0034], natural language descriptors of shape, color, text, and type of a stop sign is output; FIG. 3, [0049], [0083], natural language expressions that describe results (i.e. classification type) and inputs (i.e. shape, color, and text of a sign) of the rule 322 is output).
Wang in view of Sikka is silent about a fourth natural language descriptor having natural language text indicating an intended recipient of the traffic sign.
Shetty from the same or similar field of endeavor discloses a fourth natural language descriptor having natural language text indicating an intended recipient of the traffic sign ([0067], a natural language descriptor indicating if a sign is related to a parking spot is generated, [0069], [0143], a natural language output indicating the intended recipient (i.e. a vehicle operator) missed a stop sign is generated).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Shetty into the teachings of Wang in view of Sikka for allowing a user to more easily understand features extracted from an image.
Claim(s) 4, 6, 12, and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Wang) (US 2021/0374547) in view of Sikka (US 2022/0391689), and further in view of Chen et al. (Chen) (US 2023/0343109).
Regarding claims 4 and 12, Wang in view of Sikka discloses the apparatus of claim 9 (see claim 9 above).
Wang in view of Sikka is silent about wherein the plurality of image frames also depict a plurality of traffic lights, the operations further including determining a relevant traffic light of the plurality of traffic lights based on a state of at least one of the plurality of traffic lights, at least one detected traffic lane, and a trajectory of at least one detected vehicle.
Chen from the same or similar field of endeavor discloses wherein the plurality of image frames also depict a plurality of traffic lights ([0014], multiple traffic lights are imaged), the operations further including determining a relevant traffic light of the plurality of traffic lights based on a state of at least one of the plurality of traffic lights ([0015], [0031], the color of the traffic light is determined), at least one detected traffic lane ([0023], the traveling lane of the vehicle and an orientation of the vehicle relative to the traffic light is determined), and a trajectory of at least one detected vehicle ([0023], the traveling lane of the vehicle and an orientation of the vehicle relative to the traffic light is determined).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen into the teachings of Wang in view of Sikka for more accurately controlling a vehicle based only on relevant traffic information.
Regarding claims 6 and 14, Chen further discloses wherein determining the relevant traffic light includes: determining at least one association between the plurality of traffic lights and the at least one detected traffic lane ([0023], [0040], the traveling lane of the vehicle and an orientation of the vehicle relative to the traffic lights is determined); and determining at least one association between the plurality of traffic lights and the at least one detected vehicle ([0023], [0040], the traveling lane of the vehicle and an orientation of the vehicle relative to the traffic lights is determined).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen into the teachings of Wang in view of Sikka for more accurately controlling a vehicle based only on relevant traffic information.
Claim(s) 5 and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (Wang) (US 2021/0374547) in view of Sikka (US 2022/0391689) in view of Chen et al. (Chen) (US 2023/0343109), and further in view of Eldar et al. (Eldar) (US 2021/0064057).
Regarding claims 5 and 13, Wang in view of Sikka in view of Chen discloses the apparatus of claim 12 (see claim 12 above).
Wang in view of Sikka in view of Chen is silent about wherein determining the relevant traffic light includes determining a graph-based data structure representative of an intersection, the at least one detected traffic lane, and the at least one detected vehicle, wherein the intersection includes the plurality of traffic lights.
Eldar from the same or similar field of endeavor discloses wherein determining the relevant traffic light ([0473], a relevant traffic light is determined) includes determining a graph-based data structure (FIG. 37A) representative of an intersection (FIG. 36A), the at least one detected traffic lane (FIG. 37A, 3611A), and the at least one detected vehicle (3601) ([0503], FIG. 37A illustrates a possible relation between the time-dependent navigational information of an autonomous vehicle (e.g., vehicle 3601) traveling in lane 3611A as shown in FIG. 36A, and time-dependent state identifier for a traffic light), wherein the intersection includes the plurality of traffic lights (FIG. 36A, [0487], lights 3630A-3630C).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Eldar into the teachings of Wang in view of Sikka in view of Chen for more accurately controlling a vehicle based only on relevant traffic information.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Das et al. (Das) (US 2024/0185000) ([0031], [0120], natural language descriptors of an input object, including a traffic sign are generated).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEFFERY A WILLIAMS whose telephone number is (571)270-7579. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sath Perungavoor can be reached at 571-272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEFFERY A WILLIAMS/Primary Examiner, Art Unit 2488
Prosecution Insights