Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Response to Amendments
The amendment filed 07/20/2026 has been entered.
Claims 1, 3, 13, and 15 have been amended.
Claims 6-8 have been cancelled.
Claims 1-5 and 9-15 remain pending within the application.
The amendments filed 07/20/2026 are sufficient to overcome the 35 U.S.C.
112(b) rejection of claim 13. The previous rejection has been withdrawn.
The amendments filed 07/20/2026 are sufficient to overcome the claim objection of claim 3. The previous objection has been withdrawn.
Response to Arguments
Argument 1, regarding the 112(b) rejections, applicant argues that the rejections should be withdrawn in view of amendments made to claim 13. Examiner agrees and the rejections have been withdrawn.
Argument 2, regarding the claim objection, applicant argues that the objection to claim 3 should be withdrawn in view of amendments. Examiner agrees and the objection has been withdrawn.
Argument 3, regarding the prior art rejections, applicant argues that none of the cited art teaches “for each detected area in the set of digital images, the class scores are
determined and a maximum class score for the respective area is selected as one
score per area, the mean and the standard deviation are calculated from the respective
per-area maximum class scores; and the threshold is determined by subtracting a parameter-weighted standard deviation from the mean, with the second score meeting the condition when the second score is equal to or greater than the threshold”. Applicant argues that Mohanty teaches generic candidate-prediction confidence statistics but does not teach determining statistics from the maximum class scores assigned to respective detected areas or a threshold based on the mean minus a parameter-weighted standard deviation.
Examiner notes that Dogra (Pub. No.: US 20240006051 A1) teaches classifying data based on a mean minus standard deviation calculation, (see Dogra P0079, P0109, P0114).
The full prior art rejections are outlined below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4 and 9-15 are rejected under 35 U.S.C. 103 as being unpatentable over Yu et al (Pub. No.: US 12456055 B2), hereafter Yu, in view of Ekambaram (Pub. No.: US 11487949 B2), hereafter Ekambaram, Mohanty et al (Pub. No.: US 11562554 B1), hereafter Mohanty, and Dogra (Pub. No.: US 20240006051 A1).
Regarding claims 1, 13, and 15, Yu teaches a computer-implemented method for operating a machine (the present invention may be a method for operating autonomous vehicles, C59:L7-10, C59:34-37), a device for operating a machine (Method for operating autonomous vehicles may include the use of PPU 3200 which is a GPU (graphics processing unit) and display device, C59:L18-23), and a non-transitory computer-readable medium on which is stored a computer program for operating a machine comprising (non-transitory computer-readable storage medium may be used to carry out processes of the current invention, C69:L67, C70:L1-5): providing a digital image (“detect object in images including digital representations of those objects”, abstract); providing a structured representation of a question (users may submit requests using client device 702, C11:L55-58); predicting with an object detector and depending on at least a part of the digital image an area of the digital image, wherein an object is depicted in the digital image (“classification and location information can be used to detect and/or segment objects 182 represented in an image”, C2:L47-49); predicting, with a classifier and depending on at least a part of the digital image within the area, a first score indicating a likelihood that the object is of a first class and a second score indicating a likelihood that the object is of a second class, wherein the first score indicates a higher likelihood than the second score (Different elements of a matrix may have a confidence score associated with a class. The confidence score indicates the confidence that an object in an image belongs to a given class, C6:L62-65);… determining if the second score meets the condition or not (Scores are compared to a threshold value, with being above the threshold being interpreted as meeting the condition, C7:L13-19);… adding at least one fact to the answer set programming program depending on the structured representation of the question (“request can include, for example, input data to be processed using a neural network to obtain one or more inferences or other output values, classifications, or predictions…Relevant data, which may include at least some of input or inference data, may also be stored to a local database 720 for processing future requests”, C12:L34-36, C12:L52-55. Relevant data, including inference or input data based on the request, may be stored in a local database to be used by the machine learning model responsible for answering/responding to requests. The relevant data is interpreted as at least one fact, the request is interpreted as the structured representation of the question, and the machine learning model is interpreted as the answer set programming program)… determining, with an answer set solver, an answer to the answer set programming program (Machine learning model may be used to respond to a user request with one or more inferences, classifications, or predictions, C12:L34-46),… operating the machine depending on the answer (Machine learning model may analyze training data, including input, prediction, and answer data, and navigate a vehicle or device on behalf of a user based on the inferences made, C11:L55-58, C14:L3-10).
Yu does not appear to explicitly teach “providing at least one attribute value for the first class; adding to an answer set programming program a first rule including the at least one attribute value of the first class and/or a first constraint including the at least one attribute value of the first class,… based on the second score meeting the condition, providing at least one attribute value for the second class, and adding, to the answer set programming program, a second rule including the at least one attribute value of the second class and/or a second constraint including the at least one attribute value of the second class;…wherein the answer includes at least one attribute value of the first class and/or at least one attribute value of the second class;”.
Ekambaram teaches providing at least one attribute value for the first class (Properties are associated with classification labels. Under the broadest reasonable interpretation, the properties are interpreted as attributes belonging to a class based on their classification label, C9:L29-31); adding to an answer set programming program a first rule including the at least one attribute value of the first class and/or a first constraint including the at least one attribute value of the first class (Examiner notes that the term “and/or” is interpreted as “or” in this context and in future limitations. A word web is updated including a set of answers corresponding to a set of properties, the properties being classified. Nodes are added to this word web in accordance to various embedding scores and belief scores. Under the broadest reasonable interpretation, the basis of which nodes are added are interpreted as rules for adding answers to the answer set. C9:L31-64),… based on the second score meeting the condition, providing at least one attribute value for the second class, and adding, to the answer set programming program, a second rule including the at least one attribute value of the second class and/or a second constraint including the at least one attribute value of the second class (Context 108 may be updated and image labels corresponding to a specific class or properties may be updated based on the context. If a confidence score is equal to or exceeds a threshold value, an object is determined to belong to a specific class, C5:L22-35); … wherein the answer includes at least one attribute value of the first class and/or at least one attribute value of the second class (If an answer has a belief score above a threshold value, at least one of the first or second classification labels are removed and the answer is verified. C9:L15-26).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Yu and Ekambaram before them, to include Ekambaram’s specific teachings of associating classification labels with properties, updating an answer set based on properties associated with classes, determining an object belongs to a class based on a confidence score exceeding a threshold value, and verifying an answer has appropriate classification labels in Yu’s system of Weakly-supervised Object Detection Using One Or More Neural Networks. One would have been motivated to make such a combination of determining an object belongs to a class based on a confidence score exceeding a threshold value, updating context and image labels according to the classification (see Ekambaram C5:L22-35), and using a softmax function to evaluate whether or not elements of an image belong to a specific class (see Yu C6:L62-65) for improved outputs to users (see Ekambaram C16:L38-40).
Yu in view of Ekambaram does not appear to explicitly teach “…with a condition determined depending on a mean value and a standard deviation of a distribution of scores that indicate for a plurality of classes their respective likelihood that an object of the respective class is detected by the object detector… providing a set of classes including the first class and the second class; providing a set of digital images; determining with the object detector for the digital images in the set of digital images their respective area; determining with the classifier for areas of the digital images in the set of digital images, respective scores for the classes in the set of classes, wherein each of the scores indicates a likelihood that an object that is depicted in the respective area is of one of the set of classes; determining the mean value depending on a sum of one score per area, that are assigned to the classes in the area; determining for the one scores per area their respective difference to the mean, and determining the standard deviation depending on the differences; and determining a threshold depending on a difference between the mean and the standard deviation weighted with a parameter, and determining that the condition is met, when the second score is equal to or larger than the threshold, wherein: for each detected area in the set of digital images, the class scores are determined and a maximum class score for the respective area is selected as one score per area, the mean and the standard deviation are calculated from the respective per-area maximum class scores… with the second score meeting the condition when the second score is equal to or greater than the threshold”.
Mohanty teaches …with a condition determined depending on a mean value and a standard deviation of a distribution of scores that indicate for a plurality of classes their respective likelihood that an object of the respective class is detected by the object detector (Mean and standard deviation may be calculated corresponding to candidate predictions, C11:L61-67, C12:L1-12. A classifier then operates on the remaining candidate predictions to determine the likelihood that an object belongs to a specific class, C12:L37-46)… providing a set of classes including the first class and the second class (“A classifier can then operate on the remaining candidate predictions to assign a class to each of the remaining candidate predictions. For example, in object detection applications, the classifier can determine which class of objects that each of the remaining candidate regions contains”, C12:L40-46); providing a set of digital images (“The data stores can include permanent or transitory data used and/or operated on by the operating system, application programs, or drivers. Examples of such data include… images”, C21:L33-36); determining with the object detector for the digital images in the set of digital images their respective area (Regions of images may be processed to determine the likelihood of a specific object being present, C12:L46-59); determining with the classifier for areas of the digital images in the set of digital images, respective scores for the classes in the set of classes (A classifier operates on the remaining candidate predictions to determine the likelihood that an object in a specific area belongs to a specific class, C12:L37-46), wherein each of the scores indicates a likelihood that an object that is depicted in the respective area is of one of the set of classes (Candidate regions may be associated with regions of images indicating the likelihood of a specific object is present, C12:L46-59); determining the mean value depending on a sum of one score per area, that are assigned to the classes in the area (The mean of confidence scores corresponding to candidate regions of an image is determined, C12:L60-67, C13:L1-8); determining for the one scores per area their respective difference to the mean, and determining the standard deviation depending on the differences (Standard deviation of scores are calculated based on the difference between scores and the calculated mean, C13:L9-19); and determining a threshold depending on a difference between the mean and the standard deviation weighted with a parameter, and determining that the condition is met, when the second score is equal to or larger than the threshold (A threshold is determined based on the difference between the mean and standard deviation multiplied by a multiplier factor. Under the broadest reasonable interpretation, the standard deviation multiplied by a multiplied factor is interpreted as a weighted standard deviation. Confidence scores that are beneath the calculated threshold are discarded, meaning only scores above the threshold meet the condition. C13:L20-27), wherein: for each detected area in the set of digital images, the class scores are determined and a maximum class score for the respective area is selected as one score per area, the mean and the standard deviation are calculated from the respective per-area maximum class scores (statistical analysis may be performed on the top, high score, tail end candidates. Analysis includes determining mean and standard deviation, C5:L4-30); … with the second score meeting the condition when the second score is equal to or greater than the threshold (“Candidate predictions having a confidence score x.sub.i below the confidence score threshold (represented as the region x.sub.i<μ+(ρ×σ) in FIG. 2) can be discarded, and candidate predictions having a confidence score x.sub.i equal to or above the confidence score threshold (represented as x.sub.i≥μ+(ρ×σ) in FIG. 2) can be kept for the subsequent IoU analysis”, C5:L19-25).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Yu, Ekambaram, and Mohanty before them, to include Mohanty’s specific teachings of mean and standard deviation being calculated corresponding to candidate predictions to determine the likelihood that an object belongs to a specific class in Yu’s system of Weakly-supervised Object Detection Using One Or More Neural Networks. One would have been motivated to make such a combination of mean and standard deviation being calculated corresponding to candidate predictions to determine the likelihood that an object belongs to a specific class (see Mohanty C11:L61-67, C12:L1-12, C12:L37-46), and using a softmax function to evaluate whether or not elements of an image belong to a specific class (see Yu C6:L62-65) for improved computation time (see Mohanty C2:L66-67, C3:L1-4).
Yu in view of Ekambaram and Mohanty do not appear to explicitly teach “the threshold is determined by subtracting a parameter-weighted standard deviation from the mean”.
Dogra teaches the threshold is determined by subtracting a parameter-weighted standard deviation from the mean (data may be classified based on a mean minus standard deviation calculation, P0079, P0109, P0114).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Yu, Ekambaram, Mohanty, and Dogra before them, to include Dogra’s specific teaching of classifying data based on a mean minus standard deviation calculation in Yu’s system of Weakly-supervised Object Detection Using One Or More Neural Networks. One would have been motivated to make such a combination of classifying data based on a mean minus standard deviation calculation (see Dogra P0 P0079, P0109, P0114) and using a softmax function to evaluate whether or not elements of an image belong to a specific class (see Yu C6:L62-65) for improved classification (see Dogra P0001).
Regarding claim 2, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 1 as outlined above. Yu further teaches wherein the machine is a robot and/or a vehicle, and wherein the method further comprises detecting the digital image with at least one sensor, the at least one sensor including a camera, or a radar sensor, or a lidar sensor, or a ultrasonic sensor, or an infrared sensor, or a motion sensor (“PPU 3200 is configured to accelerate deep learning systems and applications including following non-limiting examples: autonomous vehicle platforms, … robotics”, C59:L34-37, 41. PPU may include or be coupled to a digital camera, C68:L24-28).
Regarding claim 3, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 1 as outlined above. Yu further teaches wherein the digital image includes at least one object representing a traffic sign, or a traffic surface, or a pedestrian, or a vehicle (Deep neural network analyzes an input image that may include automobiles, pedestrians, road hazards, C10:L61-67, C11:L1-14).
Ekambaram further teaches wherein the at least one attribute value of the first class and the at least one attribute value of the second class indicates a type thereof (Objects may be classified according to more broad properties. For example, a tiger may receive the classification labels for both being a mammal and for having four legs. Figure 2, C4:L11-23), and wherein the structured representation of the question includes at least one attribute value of the at least one attribute value of the first class and the at least one attribute value of the second class (Questions are formed based on selected entity nodes which include various properties and classifications. Figure 2, C4:L11-28).
Regarding claim 4, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 3 as outlined above. Yu further teaches determining an action depending on the at least one attribute value that the answer includes (Trained neural network may provide instructions for navigating a vehicle or device based on inferences or predictions made in response to user input, C11:L55-58).
Regarding claim 9, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 1 as outlined above. Ekambaram further teaches adding the second rule and/or the second constraint to the answer set programming program based on the second score failing to meet the condition and based on the second score being within a predetermined set of scores (In response to confidence values being below a threshold value, the wordweb adds properties to more accurately determine which class an object belongs to, C9:L3-8. Properties are adjusted in the wordweb based on the confidence score values corresponding to entity nodes, C4:L48-59).
Regarding claim 10, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 9 as outlined above. Mohanty further teaches determining a plurality of scores for the plurality of classes, wherein each score indicates a likelihood that an object that is depicted in the area is of one of the classes (“The convolution operation 412 may include generating candidate predictions (e.g., candidate regions such as boundary boxes in the case of object detection), and assessing for each candidate prediction a confidence score (e.g., representing the likelihood that an object of interest is in a candidate region)”, C9:L56-61); and adding to the set of scores an amount of highest scores from the plurality of scores (Candidate predictions associated with high confidence scores are kept for analysis with remaining candidate predictions associated with lower scores discarded, C4:L54-67, C5:L1-3).
Regarding claim 11, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 1 as outlined above. Yu further teaches … and determining the answer depending on the first constraint weighted with the first weight; … and determining the answer depending on the second constraint weighted with the second weight (Scores may be calculated using weighted values or weighted functions corresponding to objects. Objects may be classified as belonging to a relevant class depending on whether or not their corresponding score is above or below a confidence threshold. C8:L56-67) Mohanty further teaches providing the first constraint with a first weight for weighting the first constraint… and/or providing the second constraint with a second weight for weighting the second constraint (K candidate predictions have a corresponding multiplier factor used to weight standard deviations, C3:L11-19).
Regarding claim 12, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 11 as outlined above. Mohanty further teaches determining the first weight depending on the first score and/or determining the second weight depending on the second score (Multiplier factors may be dependent upon K number candidate predictions that are above a threshold, with the candidate predictions having confidence scores being above the threshold. C3:L11-19).
Regarding claim 14, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 13 as outlined above. Yu further teaches at least one sensor configured to capture the digital images and/or at least one actuator configured to operate the machine according to the instructions (PPU may include or be coupled to a digital camera, C68:L24-28. “PPU 3200 is configured to accelerate deep learning systems and applications including following non-limiting examples: autonomous vehicle platforms, … robotics”, C59:L34-37, 41).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Ekambaram Mohanty, and Dogra and further in view of Martin et al (Pub. No.: US 11194330 B1), hereafter Martin.
Regarding Claim 5, Yu in view of Ekambaram, Mohanty, and Dogra teaches the limitations of claim 4 as outlined above. Yu further teaches wherein the at least one object includes the traffic sign or the pedestrian (Objects in digital images may include pedestrians, C11:L12-13).
Yu in view of Ekambaram and Mohanty does not appear to explicitly teach “…and the action includes performing a stop, when the attribute value of the object representing the sign indicates that the sign is a stop sign or the attribute value of the object representing the pedestrian indicates that the pedestrian is a child”.
Martin teaches …and the action includes performing a stop, when the attribute value of the object representing the sign indicates that the sign is a stop sign or the attribute value of the object representing the pedestrian indicates that the pedestrian is a child (“a drone or other autonomous vehicle may be controlled to move based on the classification. For example, if implemented in an autonomous vehicle, the vehicle may be caused to maneuver based on a particular classification, non-limiting examples of which include driving away from “explosive” sounds, or applying brakes to slow down when a classification such as “children playing” is detected”, C10:L20-27).
Accordingly, it would have been obvious to a person having ordinary skill in the
art before the effective filing date of the claimed invention, having the teachings of
Yu, Ekambaram, Mohanty, Dogra, and Martin before them, to include Martin’s specific teachings of stopping a vehicle in response to detecting children in Yu’s system of Weakly-supervised Object Detection Using One Or More Neural Networks. One would have been motivated to make such a combination of stopping a vehicle in response to detecting children (see Martin C10:L20-27) and using a deep neural network to identify and classify objects including pedestrians (see Yu C11:L5-14) for improved deep learning performance of a deep learning network (see Martin C6:L34-35).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISHAN MOUNDI whose telephone number is (703)756-1547. The examiner can normally be reached 8:30 A.M. - 5 P.M..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.M./Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights