DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-20 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 8-14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Claim 8 is drawn to functional descriptive material recorded on a “computer program product”. Normally, the claim would be statutory. However, the broadest reasonable interpretation of a claim drawn to a “computer program product” typically covers forms of non-transitory tangible media as well as transitory propagating signals per se, making the recited claim language directed towards non-statutory subject matter such as a “signal”.
“A transitory, propagating signal … is not a “process, machine, manufacture, or composition of matter.” Those four categories define the explicit scope and reach of subject matter patentable under 35 U.S.C. § 101; thus, such a signal cannot be patentable subject matter.” (In re Nuijten, 84 USPQ2d 1495 (Fed. Cir. 2007)).
Likewise, claims 9-14 are dependent upon Claim [Insert Claim #] and fail to overcome the problem recited for claim 8. Because the full scope of the claim as properly read in light of the disclosure appears to encompass non-statutory subject matter (i.e., because the specification is silent to the exact embodiment of a computer readable medium, it is interpreted as including the ordinary and customary meaning of computer readable medium covering both non-transitory media and transitory propagating signals, etc.) the claim as a whole is non-statutory. In view of the USPTO's Interim Examination Instructions for Evaluating Subject Matter Eligibility under 35 U.S.C. 101 (the "Guidelines"), and the Official Gazette Notice (1351 OG 212, made available February 23, 2010), the examiner suggests amending the claim to include the limitation "non-transitory" in order to exclude any non-statutory subject matter. Any amendment to the claim should be commensurate with its corresponding disclosure.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 8 and 15 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Turkelson et al. (US PGPub US 20200193206 A1, hereby referred to as “Turkelson”).
Consider Claims 1, 8 and 15.
Turkelson teaches:
1. A method of detecting one or more objects in an image, the method comprising: / 8. A computer program product comprising: a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:/ 15. A system comprising: one or more processors; and a memory storing instructions that when executed by the one or more processors enable performance of an operation of detecting one or more objects in an image, the operation comprising: (Turkelson: abstract, [0007] Some aspects also include a process including: obtaining, with a computer system, (i) an image captured by a mobile computing device and (ii) coordinates indicating an input location of an input detected on a display screen of the mobile computing device, wherein: the input caused the image to be captured, the input location is a location in pixel-space of the image, and the image depicts a first object located at a first location in the image; obtaining, with the computer system, a computer-vision object recognition model trained using a training data set including images depicting objects, wherein: each image of the training data set is labeled with an object identifier, each object identifier indicates an object in an object ontology depicted by a corresponding image, and the object ontology includes the first object; detecting, with the computer system, with the computer-vision object recognition model, the first object based on: a first distance in a feature space of the computer-vision object recognition model between an image feature vector of the image and a first feature vector of the first object in the computer-vision object recognition model; and a first distance in the pixel-space of the image between the input location of the input and the first location of the first object; and causing, with the computer system, a first object identifier of the first object from the object ontology to be stored in memory. [0028]-[0032], [0140])
1. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 8. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 15. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; (Turkleson: [0140] B3. The method of any one of embodiments B1-B2, wherein the object ontology further comprises a third object not depicted in the image, detecting the first object further comprises: detecting, with the computer system, the first object based on a distance in the feature space of the computer-vision object recognition model between the image vector of the image and a third feature vector of the third object in the computer-vision object recognition model. B4. The method of any one of embodiments B1-B3, wherein the distances in the feature space comprise Euclidean distances, Minkowski distances, or cosine distances.
B5. The method of any one of embodiments B1-B4, wherein causing the first object identifier of the first object to be stored in the memory comprises: causing, with the computer system, in response to the first object being detected, the first object identifier of the first object to be stored in the memory, wherein: the first object identifier of the first object is stored in the memory in association with the first image, one or more features extracted from the first image, or the first image and the one or more features extracted from the first image.)
1. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 8. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 15. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; (Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting).)
1. and classifying the object using a classification model, / 8. and classifying the object using a classification model, / 15. and classifying the object using a classification model, (Turkleson: [0053] In some embodiments, computer system 102 may include a context classification subsystem 112, an object recognition subsystem 114, a model subsystem 116, a visual search subsystem 118, an input determination subsystem 120, a distance determination subsystem 122, and other components. As mentioned above, some or all of the aforementioned subsystems (e.g., subsystems 112-122) may be offloaded to a mobile computing device (e.g., mobile computing device 104), computer system 102 may be a mobile computing device, or both. Therefore, while the functionalities of each subsystem may be described in the context of being performed client-side or server-side, the functionalities of these subsystems are not restricted to be performed only client-side or only server-side. [0054] In some embodiments, context classification subsystem 112 may be configured to classify a context of an image based on a context classification model. As described herein, a context of an image may include, but is not limited to, a scene depicted by an image, geographical information regarding where an image was captured (e.g., from one or more location sensors resident on a device used to capture the image), temporal information indicating a time that an image was captured, input information regarding inputs detected by a device used to capture an image, user information related to a user operating a device used to capture the image, and so on. In some embodiments, scene classification may refer to a process whereby objects depicted by an image, the layout of those objects within the image, and ambient contextual information, are used to determine a scene of an image. A “scene,” as defined herein, may refer to a view of a real-world environment that includes multiple surfaces and objects, which may be organized in a meaningful way. A scene may represent one type of context, and may refer to a physical place (e.g., a geographical location, such as a landmark, address, point of interest, etc.), a type of place (e.g., a home, a school, an office, etc.), a sub-type of place (e.g., a bedroom within a home, a garage of a home, a classroom within a school, etc.), background information (e.g., trees, snow, bodies of water), or any other information, or any combination thereof. For example, context classification subsystem 112 may be configured to determine, based on an input image and a scene classification model, that the image depicts a snow-covered field. [0055]-[0056])
1. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 8. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 15. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. (Turkleson: [0056] One example of a CNN used to perform automatic scene classification is AlexNet. The AlexNet architecture includes five convolutional layers and three fully connected layers, and a Softmax layer following the last fully connected layer to output a classification distribution, with a ReLU non-linearity applied to the output of every convolutional layer and every fully connected layer. However, the scene classification model used by context classification subsystem 112 may include a same, fewer, or more convolutional layers and fully connected layers. In some embodiments, the kernels may be grouped together as residual blocks, and the kernels may be 1×1, 3×3, 5×5, or other sizes. [0057] In some embodiments, context classification subsystem 112 may output a classification vector including weights representative of the contexts determined for a given input image based on the context classification model. In some embodiments, the classification vector may upweight (e.g., if all weights are initialized to zero) features of the classification vector that were determined to be represented by the image. For example, an image depicting a snow-covered field may have a vector element associated with winter scenes upweighted (e.g., to a non-zero positive value), whereas a vector element associated with a desert or a beach may remain at its initialized value (e.g., zero). In some embodiments, additional features may be added to a feature vector for object recognition based on the contexts identified by context classification subsystem 112. [0058] In some embodiments, context classification subsystem 112 may receive an image and output a context classification vector indicative of a confidence that the image depicts a particular context. For example, an image captured by an image capture component (e.g., a camera) of computer system 102, mobile computing device 104, or kiosk device 106 may be provided to context classification subsystem 112. Upon receiving the image, context classification subsystem 112 may retrieve a context classification model from model database 138 (e.g., a scene classification model), input the image to the context classification model, and obtain an output from the context classification model of a context classification vector. For instance, context classification subsystem 112 may use a scene classification model to output a context classification vector indicative of a confidence that the image depicts a particular scene)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-6, 8-13 and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Turkelson et al. (US PGPub US 20200193206 A1, hereby referred to as “Turkelson”), in view of Wu et al. (US PGPub US 20230351769), hereby referred to as “Wu”.
Consider Claims 1, 8 and 15.
Turkelson teaches:
1. A method of detecting one or more objects in an image, the method comprising: / 8. A computer program product comprising: a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:/ 15. A system comprising: one or more processors; and a memory storing instructions that when executed by the one or more processors enable performance of an operation of detecting one or more objects in an image, the operation comprising: (Turkelson: abstract, [0007] Some aspects also include a process including: obtaining, with a computer system, (i) an image captured by a mobile computing device and (ii) coordinates indicating an input location of an input detected on a display screen of the mobile computing device, wherein: the input caused the image to be captured, the input location is a location in pixel-space of the image, and the image depicts a first object located at a first location in the image; obtaining, with the computer system, a computer-vision object recognition model trained using a training data set including images depicting objects, wherein: each image of the training data set is labeled with an object identifier, each object identifier indicates an object in an object ontology depicted by a corresponding image, and the object ontology includes the first object; detecting, with the computer system, with the computer-vision object recognition model, the first object based on: a first distance in a feature space of the computer-vision object recognition model between an image feature vector of the image and a first feature vector of the first object in the computer-vision object recognition model; and a first distance in the pixel-space of the image between the input location of the input and the first location of the first object; and causing, with the computer system, a first object identifier of the first object from the object ontology to be stored in memory. [0028]-[0032], [0140])
1. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 8. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 15. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; (Turkleson: [0140] B3. The method of any one of embodiments B1-B2, wherein the object ontology further comprises a third object not depicted in the image, detecting the first object further comprises: detecting, with the computer system, the first object based on a distance in the feature space of the computer-vision object recognition model between the image vector of the image and a third feature vector of the third object in the computer-vision object recognition model. B4. The method of any one of embodiments B1-B3, wherein the distances in the feature space comprise Euclidean distances, Minkowski distances, or cosine distances.
B5. The method of any one of embodiments B1-B4, wherein causing the first object identifier of the first object to be stored in the memory comprises: causing, with the computer system, in response to the first object being detected, the first object identifier of the first object to be stored in the memory, wherein: the first object identifier of the first object is stored in the memory in association with the first image, one or more features extracted from the first image, or the first image and the one or more features extracted from the first image.)
1. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 8. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 15. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; (Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting).)
1. and classifying the object using a classification model, / 8. and classifying the object using a classification model, / 15. and classifying the object using a classification model, (Turkleson: [0053] In some embodiments, computer system 102 may include a context classification subsystem 112, an object recognition subsystem 114, a model subsystem 116, a visual search subsystem 118, an input determination subsystem 120, a distance determination subsystem 122, and other components. As mentioned above, some or all of the aforementioned subsystems (e.g., subsystems 112-122) may be offloaded to a mobile computing device (e.g., mobile computing device 104), computer system 102 may be a mobile computing device, or both. Therefore, while the functionalities of each subsystem may be described in the context of being performed client-side or server-side, the functionalities of these subsystems are not restricted to be performed only client-side or only server-side. [0054] In some embodiments, context classification subsystem 112 may be configured to classify a context of an image based on a context classification model. As described herein, a context of an image may include, but is not limited to, a scene depicted by an image, geographical information regarding where an image was captured (e.g., from one or more location sensors resident on a device used to capture the image), temporal information indicating a time that an image was captured, input information regarding inputs detected by a device used to capture an image, user information related to a user operating a device used to capture the image, and so on. In some embodiments, scene classification may refer to a process whereby objects depicted by an image, the layout of those objects within the image, and ambient contextual information, are used to determine a scene of an image. A “scene,” as defined herein, may refer to a view of a real-world environment that includes multiple surfaces and objects, which may be organized in a meaningful way. A scene may represent one type of context, and may refer to a physical place (e.g., a geographical location, such as a landmark, address, point of interest, etc.), a type of place (e.g., a home, a school, an office, etc.), a sub-type of place (e.g., a bedroom within a home, a garage of a home, a classroom within a school, etc.), background information (e.g., trees, snow, bodies of water), or any other information, or any combination thereof. For example, context classification subsystem 112 may be configured to determine, based on an input image and a scene classification model, that the image depicts a snow-covered field. [0055]-[0056])
1. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 8. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 15. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. (Turkleson: [0056] One example of a CNN used to perform automatic scene classification is AlexNet. The AlexNet architecture includes five convolutional layers and three fully connected layers, and a Softmax layer following the last fully connected layer to output a classification distribution, with a ReLU non-linearity applied to the output of every convolutional layer and every fully connected layer. However, the scene classification model used by context classification subsystem 112 may include a same, fewer, or more convolutional layers and fully connected layers. In some embodiments, the kernels may be grouped together as residual blocks, and the kernels may be 1×1, 3×3, 5×5, or other sizes. [0057] In some embodiments, context classification subsystem 112 may output a classification vector including weights representative of the contexts determined for a given input image based on the context classification model. In some embodiments, the classification vector may upweight (e.g., if all weights are initialized to zero) features of the classification vector that were determined to be represented by the image. For example, an image depicting a snow-covered field may have a vector element associated with winter scenes upweighted (e.g., to a non-zero positive value), whereas a vector element associated with a desert or a beach may remain at its initialized value (e.g., zero). In some embodiments, additional features may be added to a feature vector for object recognition based on the contexts identified by context classification subsystem 112. [0058] In some embodiments, context classification subsystem 112 may receive an image and output a context classification vector indicative of a confidence that the image depicts a particular context. For example, an image captured by an image capture component (e.g., a camera) of computer system 102, mobile computing device 104, or kiosk device 106 may be provided to context classification subsystem 112. Upon receiving the image, context classification subsystem 112 may retrieve a context classification model from model database 138 (e.g., a scene classification model), input the image to the context classification model, and obtain an output from the context classification model of a context classification vector. For instance, context classification subsystem 112 may use a scene classification model to output a context classification vector indicative of a confidence that the image depicts a particular scene)
Turkelson does not teach dependent limitations from claims 2, 9 and 16:
wherein the second portion is larger than the first portion and fully overlaps the first portion
Wu teaches:
1. A method of detecting one or more objects in an image, the method comprising: / 8. A computer program product comprising: a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:/ 15. A system comprising: one or more processors; and a memory storing instructions that when executed by the one or more processors enable performance of an operation of detecting one or more objects in an image, the operation comprising: (Wu: abstract, In various examples, systems and methods for machine learning based hazard detection for autonomous machine applications using stereo disparity are presented. Disparity between a stereo pair of images is used to generate a path disparity model. Using the path disparity model, a machine learning model can recognize when a pixel in the first image corresponds to a pixel in the second image even though the pixel in the two images does not have identical characteristics. Similarities in extracted feature vectors can be computed and represented by a vector similarity metric that is input to a machine learning classifier, along with feature information extracted from the stereo image pair, to differentiate hazard pixels from non-hazard pixels. In some embodiments, a V-space disparity map, where a first axis corresponds to disparity values and the second axis corresponds to pixel rows, may be used to simplify estimation of the path disparity model. [0045]-[0056], Figure 1, Figures 9A-D, Figures 10-11)
1. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 8. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 15. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; (Wu: [0055] With reference to FIG. 1 , FIG. 1 is an example data flow diagram illustrating the interconnection of components and flow of information or data for machine learning based hazard detection using stereo disparity for an ego-machine (such as autonomous vehicle 900 discussed below with respect to FIG. 9A), in accordance with some embodiments of the present disclosure. In some embodiments, the systems, methods, and processes described herein may be executed using similar components, features, and/or functionality to those of example autonomous vehicle 900 of FIGS. 9A-9D, example computing device 1000 of FIG. 10 , and/or example data center 1100 of FIG. 11 . [0056] As shown in FIG. 1 , the hazard detection system executing the process 100 includes a stereo disparity machine learning hazard detector 102 that receives, as input, stereo sensor data 108 captured by a sensor pair 110. The stereo disparity machine learning hazard detector 102 processes the stereo sensor data 108 to ascertain a path disparity model 104 from one or more disparity maps 106. The stereo disparity machine learning hazard detector 102 processes the stereo sensor data 108 with one or more neural network models 105 to compute pairs of feature vectors for features extracted from the stereo sensor data 108. As further explained herein, stereo disparity machine learning hazard detector 102 utilizes the combination of feature vectors and the path disparity model 104 to detect hazards on the surface of the path, or caused by defects in the path itself. In this example, the stereo sensor data 108 may be derived from one or more on-board sensor pairs 110 of an ego-machine (e.g., ego-machine 900 of FIGS. 9A-9D).)
1. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 8. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 15. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; (Wu: [0038] In one embodiment, a first set of one or more feature vectors is computed for features extracted from the first image of the stereo image pair, and a second set of one or more features is computed for features extracted from the second image of the stereo image pair. For example, a Deep Neural Network (DNN) model may be used to extract a feature map of the first image (for example, the left image), and extract a feature map of the second image (for example, the right image). In this way, the DNN computes feature vectors for one or more pixels (e.g., each pixel) of the respective first and second images. [0041], [0069] In some embodiments, the machine learning feature vector evaluator 318 comprises a feature vector processing function 330 and an element classification function 332, each of which may be implemented at least in part by a DNN model. The feature vector processing function 330 inputs the stereo image pair (e.g. from the stereo image data 108) to extract a feature map of the first image (for example, the left image), and extract a feature map of the second image (for example, the right image). A first set of feature vectors is computed for features extracted from the first image of the stereo image pair, and a second set of features is computed for features extracted from the second image of the stereo image pair. Pixels that represent hazards appearing in the stereo image pair are expected have a distinguishable texture and pattern characteristics as compared to pixels representing the surface of the path. These distinguishing characteristics can be captured and described by the feature vectors computed by the DNN model implementing the feature vector processing function 330, which may be trained to recognize such semantic information and texture information. For example, for a given pixel (P) of the first image of the stereo image pair, a corresponding pixel of the second image of the stereo image pair is determined based on the path disparity model and/or path disparity map. A first feature vector (FV1) is extracted for the pixel (P) of the first image and a second feature vector (FV2) is extracted for the pixel determined to be the corresponding pixel to pixel (P) in the second image.)
1. and classifying the object using a classification model, / 8. and classifying the object using a classification model, / 15. and classifying the object using a classification model, (Wu: [0069] In some embodiments, the machine learning feature vector evaluator 318 comprises a feature vector processing function 330 and an element classification function 332, each of which may be implemented at least in part by a DNN model. The feature vector processing function 330 inputs the stereo image pair (e.g. from the stereo image data 108) to extract a feature map of the first image (for example, the left image), and extract a feature map of the second image (for example, the right image). A first set of feature vectors is computed for features extracted from the first image of the stereo image pair, and a second set of features is computed for features extracted from the second image of the stereo image pair. Pixels that represent hazards appearing in the stereo image pair are expected have a distinguishable texture and pattern characteristics as compared to pixels representing the surface of the path. These distinguishing characteristics can be captured and described by the feature vectors computed by the DNN model implementing the feature vector processing function 330, which may be trained to recognize such semantic information and texture information. For example, for a given pixel (P) of the first image of the stereo image pair, a corresponding pixel of the second image of the stereo image pair is determined based on the path disparity model and/or path disparity map. A first feature vector (FV1) is extracted for the pixel (P) of the first image and a second feature vector (FV2) is extracted for the pixel determined to be the corresponding pixel to pixel (P) in the second image. [0079]-[0080])
1. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 8. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 15. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. (Wu: [0079] The method 500 is drawn to detecting real-world hazardous objects in the path of an ego-machine based on extracted feature vectors and a path disparity model. Generally, the method 500 comprises determining a location of one or more pixels corresponding to one or more hazard objects on a path of an ego-machine based at least in part on classifying the one or more pixels using a first feature vector corresponding to a first image and a second feature vector corresponding to a second image, the first feature vector and the second feature vector determined to correspond to one another based at least in part on a path disparity model computed using a disparity map generated using the first image and the second image. [0080] The method 500 begins at B510 with generating a disparity map indicative of disparities between a first image generated using a first sensor and a second image generated using a second sensor. The first sensor and the second sensor have at least partially overlapping fields of view including at least a portion of a path of an ego-machine. In some embodiments, the method includes receiving a stereo image pair for a region of interest in a path of an ego-machine. The first and second sensors produce the stereo image pair. For example, FIG. 6 at 610 illustrates an image 612 from one of the stereo image pair showing the path 614 traveled by the ego-machine, and a delineated ROI 616 within which hazard detection is performed. At 620, an image space disparity map 622 is shown for the delineated ROI 616. Each pixel of the image space disparity map 622 within the ROI 616 represents the computed disparity for that pixel between the left and right images of the stereo image pair. As indicated by the image space disparity map 622, disparity of the surface of the path 614 inherently increases as points on the surface draw closer to the ego-machine. [0081])
2. The method of claim 1, wherein the second portion is larger than the first portion and fully overlaps the first portion. / 9. The computer program product of claim 8, wherein the second portion is larger than the first portion and fully overlaps the first portion. /16. The system of claim 15, wherein the second portion is larger than the first portion and fully overlaps the first portion. (Wu: [0057] For example, with reference to FIG. 2 , FIG. 2 illustrates an overlapping field of view (FOV) for a pair of cameras of vehicle 900, in accordance with some embodiments of the present disclosure. FIG. 2 includes vehicle 900, camera 202, camera 204 (which together form the sensor pair 110), field of view (FOV) 206, field of view (FOV) 208, and overlapping field of view (FOV) region 210. For example, cameras 202 and 204 may be synchronized stereo cameras that may be mounted on the vehicle 900—such as on a windshield or other area of the vehicle 900. The two cameras 202 and 204 may execute a cross-camera optical flow (OF) tracking algorithm to extract stereo disparity information from pairwise images. The camera 202 may provide the FOV 206 to the system and the camera 204 may provide the FOV 208 to the system. In some embodiments, cross-camera OF tracking may be executed at least in the overlapping FOV region 210 of the cameras 202 and 204. The overlapping FOV region 210 covers the defined region of interest (ROI) that includes the path of the ego-machine. Based on a disparity between pixels in the camera image data in the overlapping FOV region 210 for a particular location, the system may determine both the path disparity model 104 and the disparity maps 106. [0080] The method 500 begins at B510 with generating a disparity map indicative of disparities between a first image generated using a first sensor and a second image generated using a second sensor. The first sensor and the second sensor have at least partially overlapping fields of view including at least a portion of a path of an ego-machine. In some embodiments, the method includes receiving a stereo image pair for a region of interest in a path of an ego-machine. The first and second sensors produce the stereo image pair. For example, FIG. 6 at 610 illustrates an image 612 from one of the stereo image pair showing the path 614 traveled by the ego-machine, and a delineated ROI 616 within which hazard detection is performed. At 620, an image space disparity map 622 is shown for the delineated ROI 616. Each pixel of the image space disparity map 622 within the ROI 616 represents the computed disparity for that pixel between the left and right images of the stereo image pair. As indicated by the image space disparity map 622, disparity of the surface of the path 614 inherently increases as points on the surface draw closer to the ego-machine. [0081])
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Turkelson’s method and system for image analysis and processing for scene and user-input context aided visual search and object classification with Wu’s ML models for disparity maps in the realm of object classification. Both references are in the same overall field of endeavor, and the determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify Turkelson in order to ensure that it leverages the teaching of Wu for a refined machine learning model that relies on enhanced accuracy with the use of disparity maps of overlapping regions for object detection and classification. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of Turkelson, while the teaching of Wu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of an improved machine learning model for accurate object detection and classification. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 2, 9 and 16.
The combination of Turkelson and Wu teaches:
2. The method of claim 1, wherein the second portion is larger than the first portion and fully overlaps the first portion. / 9. The computer program product of claim 8, wherein the second portion is larger than the first portion and fully overlaps the first portion. /16. The system of claim 15, wherein the second portion is larger than the first portion and fully overlaps the first portion. (Wu: [0057] For example, with reference to FIG. 2 , FIG. 2 illustrates an overlapping field of view (FOV) for a pair of cameras of vehicle 900, in accordance with some embodiments of the present disclosure. FIG. 2 includes vehicle 900, camera 202, camera 204 (which together form the sensor pair 110), field of view (FOV) 206, field of view (FOV) 208, and overlapping field of view (FOV) region 210. For example, cameras 202 and 204 may be synchronized stereo cameras that may be mounted on the vehicle 900—such as on a windshield or other area of the vehicle 900. The two cameras 202 and 204 may execute a cross-camera optical flow (OF) tracking algorithm to extract stereo disparity information from pairwise images. The camera 202 may provide the FOV 206 to the system and the camera 204 may provide the FOV 208 to the system. In some embodiments, cross-camera OF tracking may be executed at least in the overlapping FOV region 210 of the cameras 202 and 204. The overlapping FOV region 210 covers the defined region of interest (ROI) that includes the path of the ego-machine. Based on a disparity between pixels in the camera image data in the overlapping FOV region 210 for a particular location, the system may determine both the path disparity model 104 and the disparity maps 106. [0080] The method 500 begins at B510 with generating a disparity map indicative of disparities between a first image generated using a first sensor and a second image generated using a second sensor. The first sensor and the second sensor have at least partially overlapping fields of view including at least a portion of a path of an ego-machine. In some embodiments, the method includes receiving a stereo image pair for a region of interest in a path of an ego-machine. The first and second sensors produce the stereo image pair. For example, FIG. 6 at 610 illustrates an image 612 from one of the stereo image pair showing the path 614 traveled by the ego-machine, and a delineated ROI 616 within which hazard detection is performed. At 620, an image space disparity map 622 is shown for the delineated ROI 616. Each pixel of the image space disparity map 622 within the ROI 616 represents the computed disparity for that pixel between the left and right images of the stereo image pair. As indicated by the image space disparity map 622, disparity of the surface of the path 614 inherently increases as points on the surface draw closer to the ego-machine. [0081])
Consider Claims 3, 10 and 17.
The combination of Turkelson and Wu teaches:
3. The method of claim 1, wherein the object is depicted in the second portion. / 10. The computer program product of claim 8, wherein the object is depicted in the second portion. / 17. The system of claim 15, wherein the object is depicted in the second portion. (Wu: [0057] For example, with reference to FIG. 2 , FIG. 2 illustrates an overlapping field of view (FOV) for a pair of cameras of vehicle 900, in accordance with some embodiments of the present disclosure. FIG. 2 includes vehicle 900, camera 202, camera 204 (which together form the sensor pair 110), field of view (FOV) 206, field of view (FOV) 208, and overlapping field of view (FOV) region 210. For example, cameras 202 and 204 may be synchronized stereo cameras that may be mounted on the vehicle 900—such as on a windshield or other area of the vehicle 900. The two cameras 202 and 204 may execute a cross-camera optical flow (OF) tracking algorithm to extract stereo disparity information from pairwise images. The camera 202 may provide the FOV 206 to the system and the camera 204 may provide the FOV 208 to the system. In some embodiments, cross-camera OF tracking may be executed at least in the overlapping FOV region 210 of the cameras 202 and 204. The overlapping FOV region 210 covers the defined region of interest (ROI) that includes the path of the ego-machine. Based on a disparity between pixels in the camera image data in the overlapping FOV region 210 for a particular location, the system may determine both the path disparity model 104 and the disparity maps 106. [0080] The method 500 begins at B510 with generating a disparity map indicative of disparities between a first image generated using a first sensor and a second image generated using a second sensor. The first sensor and the second sensor have at least partially overlapping fields of view including at least a portion of a path of an ego-machine. In some embodiments, the method includes receiving a stereo image pair for a region of interest in a path of an ego-machine. The first and second sensors produce the stereo image pair. For example, FIG. 6 at 610 illustrates an image 612 from one of the stereo image pair showing the path 614 traveled by the ego-machine, and a delineated ROI 616 within which hazard detection is performed. At 620, an image space disparity map 622 is shown for the delineated ROI 616. Each pixel of the image space disparity map 622 within the ROI 616 represents the computed disparity for that pixel between the left and right images of the stereo image pair. As indicated by the image space disparity map 622, disparity of the surface of the path 614 inherently increases as points on the surface draw closer to the ego-machine. [0081] Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting).)
Consider Claims 4, 11 and 18.
The combination of Turkelson and Wu teaches:
4. The method of claim 1, further comprising: receiving positional information indicating a position of the first portion relative to at least the second portion, wherein classifying the object further comprises applying the positional information to the classification model. / 11. The computer program product of claim 8, the operation further comprising: receiving positional information indicating a position of the first portion relative to at least the second portion, wherein classifying the object further comprises applying the positional information to the classification model./ 18. The system of claim 15, the operation further comprising: receiving positional information indicating a position of the first portion relative to at least the second portion, wherein classifying the object further comprises applying the positional information to the classification model. (Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting). Wu: [0036] In some embodiments, to simplify estimation of the path disparity model, the image space disparity map is transformed to an updated (or “V”) disparity map. That is, in an image space disparity map, the pixel columns are represented on a first axis (which may referred to at the U axis) and pixel rows are represented on the second axis (which may be referred to as the V axis) and a disparity value at coordinates (u,v) of the disparity map may indicate a disparity associated with an image pixel at coordinates (x,y) of the image as captured by one of the sensors (e.g., either sensor O or O′). In V-disparity space a first axis of the disparity map instead corresponds to disparity values while the second axis again corresponds to pixel rows (the V axis). Each of the one or more elements in the V-disparity map indicates a count of disparity elements in the row of the original image space disparity map. [0057] In some embodiments, context classification subsystem 112 may output a classification vector including weights representative of the contexts determined for a given input image based on the context classification model. In some embodiments, the classification vector may upweight (e.g., if all weights are initialized to zero) features of the classification vector that were determined to be represented by the image. For example, an image depicting a snow-covered field may have a vector element associated with winter scenes upweighted (e.g., to a non-zero positive value), whereas a vector element associated with a desert or a beach may remain at its initialized value (e.g., zero). In some embodiments, additional features may be added to a feature vector for object recognition based on the contexts identified by context classification subsystem 112.)
Consider Claims 5, 12 and 19.
The combination of Turkelson and Wu teaches:
5. The method of claim 4, wherein the second portion is the image, and wherein the positional information comprises one of coordinates of the first portion within the image, and a position vector of the first portion within the image. / 12. The computer program product of claim 11, wherein the second portion is the image, and wherein the positional information comprises one of coordinates of the first portion within the image, and a position vector of the first portion within the image. / 19. The system of claim 18, wherein the second portion is the image, and wherein the positional information comprises one of coordinates of the first portion within the image, and a position vector of the first portion within the image. (Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting). Wu: [0036] In some embodiments, to simplify estimation of the path disparity model, the image space disparity map is transformed to an updated (or “V”) disparity map. That is, in an image space disparity map, the pixel columns are represented on a first axis (which may referred to at the U axis) and pixel rows are represented on the second axis (which may be referred to as the V axis) and a disparity value at coordinates (u,v) of the disparity map may indicate a disparity associated with an image pixel at coordinates (x,y) of the image as captured by one of the sensors (e.g., either sensor O or O′). In V-disparity space a first axis of the disparity map instead corresponds to disparity values while the second axis again corresponds to pixel rows (the V axis). Each of the one or more elements in the V-disparity map indicates a count of disparity elements in the row of the original image space disparity map. [0057] In some embodiments, context classification subsystem 112 may output a classification vector including weights representative of the contexts determined for a given input image based on the context classification model. In some embodiments, the classification vector may upweight (e.g., if all weights are initialized to zero) features of the classification vector that were determined to be represented by the image. For example, an image depicting a snow-covered field may have a vector element associated with winter scenes upweighted (e.g., to a non-zero positive value), whereas a vector element associated with a desert or a beach may remain at its initialized value (e.g., zero). In some embodiments, additional features may be added to a feature vector for object recognition based on the contexts identified by context classification subsystem 112.)
Consider Claims 6, 13 and 20.
The combination of Turkelson and Wu teaches:
6. The method of claim 1, wherein one or both of the first feature extractor and the second feature extractor have pretrained fixed parameters, the method further comprising: training the classification model using outputs from the first feature extractor and the second feature extractor. / 13. The computer program product of claim 8, wherein one or both of the first feature extractor and the second feature extractor have pretrained fixed parameters, the operation further comprising: training the classification model using outputs from the first feature extractor and the second feature extractor. / 20. The system of claim 15, wherein one or both of the first feature extractor and the second feature extractor have pretrained fixed parameters, the operation further comprising: training the classification model using outputs from the first feature extractor and the second feature extractor. (Turkelson: In some embodiments, the feature extraction process may use deep learning processing to extract features from an image. For example, a deep convolution neural network (CNN), trained on a large set of training data (e.g., the AlexNet architecture, which includes 5 convolutional layers and 3 fully connected layers, trained using the ImageNet dataset) may be used to extract features from an image. In some embodiments, to perform feature extraction, context classification subsystem 112 and object recognition subsystem 112 may obtain a pre-trained machine learning model from model database 138, which may be used for performing feature extraction for images from a set of images provided to computer system 102. In some embodiments, a support vector machine (SVM) may be trained with a training data to obtain a trained model for performing feature extraction. In some embodiments, a classifier may be trained using extracted features from an earlier layer of the machine learning model. In some embodiments, preprocessing may be performed to an input image prior to the feature extraction being performed. For example, preprocessing may include resizing, normalizing, cropping, etc., to each image to allow that image to serve as an input to the pre-trained model. Example pre-trained networks may include AlexNet, GoogLeNet, MobileNet-v2, and others. The preprocessing input images may be fed to the pre-trained model, which may extract features, and those features may then be used to train a classifier (e.g., SVM). In some embodiments, the input images, the features extracted from each of the input images, an identifier labeling each of the input image, or any other aspect capable of being used to describe each input image, or a combination thereof, may be stored in memory (e.g., within training data database 136 as an update to training data set for training an object recognition model, a context classification model, etc.). In some embodiments, a feature vector describing visual features extracted from an image may be output from context classification subsystem 112 and object recognition subsystem 114, which may describe one or more contexts of the image and one or more objects determined to be depicted by the image. In some embodiments, the feature vector, the input image, or both, may be used as an input to a visual search system (e.g., visual search subsystem 124) for performing a visual search to obtain information related to objects depicted within the image (e.g., products that a user may purchase). [0066] In some embodiments, model subsystem 116 may be configured to retrieve models stored within model database 138, provide the retrieved models to one or more subsystems for analyzing an image or set of images (e.g., to context classification subsystem 112, object recognition subsystem 114, etc.), as well as to train one or more models and generate training data for training the one or more models. For example, model subsystem 116 may be configured to train a context classification model to be used by context classification subsystem 112, an object recognition model to be used by object recognition subsystem 114, and the like. In some embodiments, model subsystem 116 may build or assist in the build of a given model.)
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Turkelson et al. (US PGPub US 20200193206 A1, hereby referred to as “Turkelson”), in view of Wu et al. (US PGPub US 20230351769), hereby referred to as “Wu”, further in view of Georgeson et al. (US PGPub US 2014/0184786, hereby referred to as “Georgeson”)
Consider Claims 7 and 14.
The combination of Turkelson and Wu teaches:
7. The method of claim 1, wherein the image depicts external surfaces of a plurality of sections of a surface, wherein the first portion of image depicts an external surface of a first section of the plurality of sections, wherein classifying the object comprises distinguishing the first section. / 14. The computer program product of claim 8, wherein the image depicts external surfaces of a plurality of sections of a surface, wherein the first portion of image depicts an external surface of a first section of the plurality of sections, wherein classifying the object comprises distinguishing the first section. (Wu: [0003] Another existing hazard detection technology leverages a Deep Neural Network (DNN) inference engine. The DNN is trained on the appearance and shapes of hazards to detect when a hazard appears in a captured image. However, given the wide variety of hazards that might be present on a road surface, large amounts of training data are needed, and obtaining such training data is not trivial due to the unknown and unlimited types of potential hazards. [0004], [0034] In some embodiments, blockwise division is used to subdivide the disparity map into a plurality of smaller disparity maps, each corresponding to a block of pixels of the disparity map. Typically, a road traveled by an ego-machine is not a flat plane. Rather, it can be expected to slope to either side (e.g., to allow for rain runoff), and have a surface that is not perfectly parallel to the row axis of the captured images. Both of these characteristics contribute to greater road surface disparity. By subdividing and treating each of these blocks as an individual processing unit for hazard detection, the disparity of the road surface itself within each block is considerably less than if the disparity of the entire road surface as captured by the stereo images were considered. After hazard detection is performed within each block, the results can then be correlated back to an image of the road surface within the ROI. [0051] The path travelled by the ego-machine is not limited to any one type of path or surface and may include paths such as a paced road, an unpaved road, a highway, a driveway, a portion of a parking lot, a trail, a track, a walking path, a delineated portion of an environment, or an aircraft runway or landing pad, for example. It should also be understood that the path travelled by the ego-machine may be inside a building or other facility and comprise a hallway, corridor or isle, for example. Turkelson: [0076] In some embodiments, pressure may also be detected by a touch-sensitive display screen. As an example, pressure sensitive device or pressure may be configured to determine an amount of pressure applied to a surface. Based on an amount of pressure, a characteristic of the input may be determined. In some embodiments, an amount of force that an input has may indicate a depth of focus of the image. For example, a hard touch may indicate a larger area of interest, whereas a soft touch may indicate a smaller area of interest, or vice versa. The amount of force, and the corresponding area that the input was detected by, may indicate what a user sought to select within an image, the user's focus or interest within an image, and the like. [0077] In some embodiments, eye gaze may also be used as an input channel for determining a location of an input to a display screen. In some embodiments, eye gaze may include tracking a position and movement of an individual's eyes to determine a location on a display screen (or other surface) that an individual's focus is directed towards.)
The combination of Turkelson and Wu does not teach the dependent claims from claims 7 and 14:
wherein the image depicts external surfaces of a plurality of sections of an aircraft
Georgeson teaches:
7. The method of claim 1, wherein the image depicts external surfaces of a plurality of sections of an aircraft, wherein the first portion of image depicts an external surface of a first section of the plurality of sections, wherein classifying the object comprises distinguishing the first section. / 14. The computer program product of claim 8, wherein the image depicts external surfaces of a plurality of sections of an aircraft, wherein the first portion of image depicts an external surface of a first section of the plurality of sections, wherein classifying the object comprises distinguishing the first section. (Georgeson: abstract, A system for stand-off inspection comprising local positioning system hardware and a nondestructive evaluation instrument supported by a pan-tilt mechanism. The system further comprises a computer system that is programmed to perform the following operations: (a) directing the local positioning system hardware toward an area of a surface on a target object by control of the pan-tilt mechanism; (b) activating the local positioning system hardware to acquire an image; (c) processing the image to determine whether an anomaly is present in the area; (d) if an anomaly is present, determining coordinates of a position of the anomaly in a coordinate system of the target object; and (e) directing the nondestructive evaluation instrument toward a position corresponding to the coordinates. Optionally, the computer system is further programmed to measure one or more characteristics of the anomaly. [0034] Three-dimensional localization software may be loaded into computer 8. For example, the three-dimensional localization software may be of a type that uses multiple calibration points 15 at a distance on the target object 14 (such as a surface on an aircraft) to define the location (position and orientation) of video camera 2 relative to target object 14. In some applications, the three-dimensional localization software may utilize a minimum of three calibration points 15 on the target object 14, in combination with pan and tilt data from the pan-tilt mechanism 3, to define the relative position and orientation of the video camera 2 with respect to the local coordinate system 27 of the target object 14. The calibration points 15 may be visible features of known position in the local coordinate system 27 of the target object 14 as determined from a three-dimensional database of feature positions (e.g., a CAD model) or other measurement technique. The calibration points 15 may be used in coordination with the azimuth and elevation angles from the pan-tilt mechanism 3 to solve for the camera position and orientation relative to the target object 14.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the combination of Turkelson and Wu for a method and system for image analysis and processing for use of ML models in the realm of object classification context aided visual search and object classification with Georgeson’s specific model for target object detection algorithm for aircraft surfaces. Both references are in the same overall field of endeavor, and the determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the combination of Turkelson and Wu in order to ensure that it leverages the teaching of Georgeson for a refined machine learning model that can be applied for for object detection and classification on aircraft surfaces. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of the combination of Turkelson and Wu, while the teaching of Georgeson continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of an improved machine learning model for accurate object detection and classification for aircraft surfaces. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image1.png
274
894
media_image1.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
April 9, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights