DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is in response to the applicant’s reply filed July 1, 2026. In the applicant’s reply; claims 1, 8-9 and 15 were amended, claims 3, 10 and 17 were cancelled, and claims 21-23 were newly added. Claims 1-2, 4-9, 11-16 and 18-23 are pending in this application.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Examiner’s Responses to Applicant’s Remark
Applicants' amendments filed on July 1, 2026 have been fully considered. The amendments overcome the following rejections set forth in the office action mailed on April 17, 2026.
Applicants’ amendments overcome the objection to the title of the specification, and the objection is hereby withdrawn.
Applicant’s amendments overcome the rejections of claims 8-14 under 35 U.S.C. 101 for being directed to non-statutory subject matter, and the rejection is hereby withdrawn.
Applicant's arguments with respect to the claims have been considered but are moot in view of the new ground(s) of rejection, presented below, and necessitated by applicant’s amendments.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1, 8 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Turkelson et al. (US PGPub US 20200193206 A1, hereby referred to as “Turkelson”) in view of Wang et al. (US PGPub US 2011/0069152).
Consider Claims 1, 8 and 15.
Turkelson teaches:
1. A method of detecting one or more objects in an image, the method comprising: / 8. A computer program product comprising: a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:/ 15. A system comprising: one or more processors; and a memory storing instructions that when executed by the one or more processors enable performance of an operation of detecting one or more objects in an image, the operation comprising: (Turkelson: abstract, [0007] Some aspects also include a process including: obtaining, with a computer system, (i) an image captured by a mobile computing device and (ii) coordinates indicating an input location of an input detected on a display screen of the mobile computing device, wherein: the input caused the image to be captured, the input location is a location in pixel-space of the image, and the image depicts a first object located at a first location in the image; obtaining, with the computer system, a computer-vision object recognition model trained using a training data set including images depicting objects, wherein: each image of the training data set is labeled with an object identifier, each object identifier indicates an object in an object ontology depicted by a corresponding image, and the object ontology includes the first object; detecting, with the computer system, with the computer-vision object recognition model, the first object based on: a first distance in a feature space of the computer-vision object recognition model between an image feature vector of the image and a first feature vector of the first object in the computer-vision object recognition model; and a first distance in the pixel-space of the image between the input location of the input and the first location of the first object; and causing, with the computer system, a first object identifier of the first object from the object ontology to be stored in memory. [0028]-[0032], [0140])
1. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 8. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 15. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; (Turkleson: [0140] B3. The method of any one of embodiments B1-B2, wherein the object ontology further comprises a third object not depicted in the image, detecting the first object further comprises: detecting, with the computer system, the first object based on a distance in the feature space of the computer-vision object recognition model between the image vector of the image and a third feature vector of the third object in the computer-vision object recognition model. B4. The method of any one of embodiments B1-B3, wherein the distances in the feature space comprise Euclidean distances, Minkowski distances, or cosine distances.
B5. The method of any one of embodiments B1-B4, wherein causing the first object identifier of the first object to be stored in the memory comprises: causing, with the computer system, in response to the first object being detected, the first object identifier of the first object to be stored in the memory, wherein: the first object identifier of the first object is stored in the memory in association with the first image, one or more features extracted from the first image, or the first image and the one or more features extracted from the first image.)
1. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 8. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 15. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; (Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting).)
1. and classifying the object using a classification model, / 8. and classifying the object using a classification model, / 15. and classifying the object using a classification model, (Turkleson: [0053] In some embodiments, computer system 102 may include a context classification subsystem 112, an object recognition subsystem 114, a model subsystem 116, a visual search subsystem 118, an input determination subsystem 120, a distance determination subsystem 122, and other components. As mentioned above, some or all of the aforementioned subsystems (e.g., subsystems 112-122) may be offloaded to a mobile computing device (e.g., mobile computing device 104), computer system 102 may be a mobile computing device, or both. Therefore, while the functionalities of each subsystem may be described in the context of being performed client-side or server-side, the functionalities of these subsystems are not restricted to be performed only client-side or only server-side. [0054] In some embodiments, context classification subsystem 112 may be configured to classify a context of an image based on a context classification model. As described herein, a context of an image may include, but is not limited to, a scene depicted by an image, geographical information regarding where an image was captured (e.g., from one or more location sensors resident on a device used to capture the image), temporal information indicating a time that an image was captured, input information regarding inputs detected by a device used to capture an image, user information related to a user operating a device used to capture the image, and so on. In some embodiments, scene classification may refer to a process whereby objects depicted by an image, the layout of those objects within the image, and ambient contextual information, are used to determine a scene of an image. A “scene,” as defined herein, may refer to a view of a real-world environment that includes multiple surfaces and objects, which may be organized in a meaningful way. A scene may represent one type of context, and may refer to a physical place (e.g., a geographical location, such as a landmark, address, point of interest, etc.), a type of place (e.g., a home, a school, an office, etc.), a sub-type of place (e.g., a bedroom within a home, a garage of a home, a classroom within a school, etc.), background information (e.g., trees, snow, bodies of water), or any other information, or any combination thereof. For example, context classification subsystem 112 may be configured to determine, based on an input image and a scene classification model, that the image depicts a snow-covered field. [0055]-[0056])
1. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 8. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 15. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. (Turkleson: [0056] One example of a CNN used to perform automatic scene classification is AlexNet. The AlexNet architecture includes five convolutional layers and three fully connected layers, and a Softmax layer following the last fully connected layer to output a classification distribution, with a ReLU non-linearity applied to the output of every convolutional layer and every fully connected layer. However, the scene classification model used by context classification subsystem 112 may include a same, fewer, or more convolutional layers and fully connected layers. In some embodiments, the kernels may be grouped together as residual blocks, and the kernels may be 1×1, 3×3, 5×5, or other sizes. [0057] In some embodiments, context classification subsystem 112 may output a classification vector including weights representative of the contexts determined for a given input image based on the context classification model. In some embodiments, the classification vector may upweight (e.g., if all weights are initialized to zero) features of the classification vector that were determined to be represented by the image. For example, an image depicting a snow-covered field may have a vector element associated with winter scenes upweighted (e.g., to a non-zero positive value), whereas a vector element associated with a desert or a beach may remain at its initialized value (e.g., zero). In some embodiments, additional features may be added to a feature vector for object recognition based on the contexts identified by context classification subsystem 112. [0058] In some embodiments, context classification subsystem 112 may receive an image and output a context classification vector indicative of a confidence that the image depicts a particular context. For example, an image captured by an image capture component (e.g., a camera) of computer system 102, mobile computing device 104, or kiosk device 106 may be provided to context classification subsystem 112. Upon receiving the image, context classification subsystem 112 may retrieve a context classification model from model database 138 (e.g., a scene classification model), input the image to the context classification model, and obtain an output from the context classification model of a context classification vector. For instance, context classification subsystem 112 may use a scene classification model to output a context classification vector indicative of a confidence that the image depicts a particular scene)
Even if Turkelson does not teach the amended features for: wherein the object is in the second portion, andwherein the second portion has a downscaled pixel resolution compared to the first portion;
Wang teaches:
1. A method of detecting one or more objects in an image, the method comprising: / 8. A computer program product comprising: a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:/ 15. A system comprising: one or more processors; and a memory storing instructions that when executed by the one or more processors enable performance of an operation of detecting one or more objects in an image, the operation comprising: (Wang: abstract, A method for real-time 2D to 3D video conversion includes receiving a decoded 2D video frame having an original resolution, downscaling the decoded 2D video frame into an associated 2D video frame having a lower resolution, and segmenting objects present in the downscaled 2D video frame into background objects and foreground objects. The method also includes generating a background depth map and a foreground depth map for the downscaled 2D video frame based on the segmented background and foreground objects, and deriving a frame depth map in the original resolution based on the background depth map and the foreground depth map. The method further includes providing a 3D video frame for display at a real-time playback rate. The 3D video frame is generated in the original resolution based on the frame depth map. [0017] FIG. 1 illustrates a block diagram of an exemplary system 100. Exemplary system 100 can be any type of system that provides video content over a local connection or a network, such as a wireless network, Internet, broadcast network, and etc. Exemplary system 100 can include, among other things, 2D video content sources such as a video storage medium 102, a media server 104, and/or network 106, a user device 108, a video decoder 110, a 2D-to-3D video converter 112, and a display 114. [0018] Video storage medium 102 can be any medium storing video content. For example, video storage medium 102 can be provided as a video CD, DVD, Blu-ray disc, hard disk, magnetic tape, flash memory card/drive, volatile or non-volatile memory, holographic data storage, and any other storage medium. Video storage medium 102 can be located within user device 108, local to user device 108, or remote from user device 108. [0019] Media server 104 is a computer server that receives a request for video content from user device 108, processes the request, and provides video content to user device 108 through, in some embodiments, network 106.)
1. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 8. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 15. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; (Examiner Note: Wang’s teachings for a background depth map qualifies as a first feature vector as the depth map in the broadest reasonable interpretation is analogous in scope to a first feature vector; Wang: [0016] The method and system disclosed herein process video frame depth estimation by using background tracking. Instead of seeking for the geometric structure for a whole frame, geometric structure of background objects is detected, while foreground objects are not considered in the structure analysis process, thereby enhancing execution speed and correctness of point of view. In exemplary embodiments, the foreground objects are not considered to contribute to the frame geometric structure if there is a background in the image. To the contrary, if the foreground objects have complicated texture or edge components inside, their consideration may unnecessarily increase computational complexity. In exemplary embodiments, a foreground object depth map is generated using a different model than used for background objects. After all depth maps are obtained, a final frame depth map is derived at the same level of resolution as that of the original image. The methods and systems discussed herein provide real-time 2D-to-3D video conversion, which derives depth maps from lower-resolution 2D frames and generates 3D video frames in the original resolution at a real-time playback rate on a user device. [0046] FIG. 3 is a functional diagram illustrating an exemplary communication flow for real-time 3D video depth map generation by background tracking and structure analysis in exemplary system 200 of FIG. 2. After receiving (109) a 2D video bitstream, video decoder 110 can decode the video bitstream into 2D video frames, and store (111) the decoded 2D video frames in video frame database 210. More particularly, video frame database 210 can store in buffer 212 the decoded 2D video frames in their original resolution. Each decoded 2D video frame can include a unique identifier (frame ID) for later retrieval and association purpose., [0047] Downscaler 220 can retrieve (302) from buffer 212 the decoded 2D video frames in original resolution, downscale the decoded 2D video frames to lower resolution, and store (302) the downscaled 2D video frames in buffer 214. Alternatively, downscaled 2D video frames can be passed in communication pipeline flows and be processed on the fly without being stored in buffer 214. Each downscaled 2D video frame is associated with a decoded 2D video frame with original resolution and retains the frame ID of the corresponding decoded 2D video frame with original resolution. The association can be based on the frame ID.)
1. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion, wherein the object is in the second portion, and wherein the second portion has a downscaled pixel resolution compared to the first portion; / 8. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion, wherein the object is in the second portion, and wherein the second portion has a downscaled pixel resolution compared to the first portion; / 15. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion, wherein the object is in the second portion, and wherein the second portion has a downscaled pixel resolution compared to the first portion; (Examiner Note: Wang’s teachings for a background depth map qualifies as a first feature vector as the depth map in the broadest reasonable interpretation is analogous in scope to a first feature vector; Wang: [0047] Downscaler 220 can retrieve (302) from buffer 212 the decoded 2D video frames in original resolution, downscale the decoded 2D video frames to lower resolution, and store (302) the downscaled 2D video frames in buffer 214. Alternatively, downscaled 2D video frames can be passed in communication pipeline flows and be processed on the fly without being stored in buffer 214. Each downscaled 2D video frame is associated with a decoded 2D video frame with original resolution and retains the frame ID of the corresponding decoded 2D video frame with original resolution. The association can be based on the frame ID. [0054] A 2D video bitstream is first received (109) and decoded by video decoder 110 to provide decoded video frames 402 in original resolution that can be stored in video frame database 210 for future display. In addition to the decoded video frames 402 in the original resolution (for example, 640 by 480), an associated set of lower-resolution frames 406 (for example 240 by 135) can be generated (404) and stored in video frame database 210 for accelerated background tracking (410) and depth map estimation. Alternatively, the associated set of lower-resolution frames 406 can be passed in process pipeline flows without being stored in video frame database 210. By tracking moving objects in the lower-resolution frames 406, the background and foreground objects can be segmented from each other, as described above. [0055] Next, the background and foreground objects are subjected to separate depth map estimation process. A background depth map is generated based on, among other things, background structure analysis (420) and background depth map estimation (430), as described above. A foreground depth map is generated based on, among other things, foreground skeleton depth estimation (440) and foreground depth map estimation (450), as described above. [0056] The final frame depth map is generated (460) by fusing background and foreground depth maps in the original resolution, as described above. Using DIBR algorithms (470), multi-view (3D) video frames 408 for 3D displaying are generated from the frame depth map and images of frames in original resolution, as described above. 3D video frames 408 can be stored in video frame database 210 and provided (113) to display 114 for displaying.)
1. and classifying the object using a classification model, / 8. and classifying the object using a classification model, / 15. and classifying the object using a classification model, (Wang: [0057] FIG. 5 is a flowchart representing an exemplary method of background object segmentation by tracking moving objects in a lower-resolution video frame. It will now be appreciated by one of ordinary skill in the art that the illustrated procedure can be altered to delete steps, change the order of steps, or include additional steps. After initial start step 500, downscaler 220 of 2D-to-3D video converter 112 receives (502) a 2D video bitstream from, for example, user device 108, or video storage medium 102, or media server 104 through network 106. Then, video decoder 110 decodes (504) the video bitstream into a set of video frames in original resolution. Each decoded video frame can have a unique identifier (frame ID).)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the context-based machine learning classification model of Turkelson with Wang’s multi-resolution image analytics and processing in order to leverage image context data in a well-established and conventional manner. The determination of obviousness is predicated upon the following findings: Wang’s teachings for the processing of image data to establish foreground and background data in overlapping areas is a well-established known technique that could easily be used and applied to contextually data and uses similar image analytics of processing data from different resolutions. One skilled in the art would have been motivated to modify Turkeslon in this manner in order to improve the machine learning model to rely on well-known and established in 3D image processing for foreground and background separation and using those principles on contextual data for object classification. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of Turkelson, while the teaching of Wang continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of improving the overall machine learning model for object classification to leverage multi-resolution image processing in a known manner for context-based applications. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Claims 2-6, 9-13 and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Turkelson and Wang, further in view of Wu et al. (US PGPub US 20230351769), hereby referred to as “Wu”.
Consider Claims 2, 9 and 16.
The combination of Turkelson and Wang teaches: The method of Claim 1/ A computer program product of Claim 8/ 15. A system of Claim 15, as presented above.
The combination of Turkelson and Wang does not teach dependent limitations from claims 2, 9 and 16:
-; wherein the second portion is larger than the first portion and fully overlaps the first portion
Wu teaches:
1. A method of detecting one or more objects in an image, the method comprising: / 8. A computer program product comprising: a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising:/ 15. A system comprising: one or more processors; and a memory storing instructions that when executed by the one or more processors enable performance of an operation of detecting one or more objects in an image, the operation comprising: (Wu: abstract, In various examples, systems and methods for machine learning based hazard detection for autonomous machine applications using stereo disparity are presented. Disparity between a stereo pair of images is used to generate a path disparity model. Using the path disparity model, a machine learning model can recognize when a pixel in the first image corresponds to a pixel in the second image even though the pixel in the two images does not have identical characteristics. Similarities in extracted feature vectors can be computed and represented by a vector similarity metric that is input to a machine learning classifier, along with feature information extracted from the stereo image pair, to differentiate hazard pixels from non-hazard pixels. In some embodiments, a V-space disparity map, where a first axis corresponds to disparity values and the second axis corresponds to pixel rows, may be used to simplify estimation of the path disparity model. [0045]-[0056], Figure 1, Figures 9A-D, Figures 10-11)
1. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 8. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; / 15. receiving a first portion of the image at a first feature extractor to provide a first feature vector, the first portion having an object depicted therein; (Wu: [0055] With reference to FIG. 1 , FIG. 1 is an example data flow diagram illustrating the interconnection of components and flow of information or data for machine learning based hazard detection using stereo disparity for an ego-machine (such as autonomous vehicle 900 discussed below with respect to FIG. 9A), in accordance with some embodiments of the present disclosure. In some embodiments, the systems, methods, and processes described herein may be executed using similar components, features, and/or functionality to those of example autonomous vehicle 900 of FIGS. 9A-9D, example computing device 1000 of FIG. 10 , and/or example data center 1100 of FIG. 11 . [0056] As shown in FIG. 1 , the hazard detection system executing the process 100 includes a stereo disparity machine learning hazard detector 102 that receives, as input, stereo sensor data 108 captured by a sensor pair 110. The stereo disparity machine learning hazard detector 102 processes the stereo sensor data 108 to ascertain a path disparity model 104 from one or more disparity maps 106. The stereo disparity machine learning hazard detector 102 processes the stereo sensor data 108 with one or more neural network models 105 to compute pairs of feature vectors for features extracted from the stereo sensor data 108. As further explained herein, stereo disparity machine learning hazard detector 102 utilizes the combination of feature vectors and the path disparity model 104 to detect hazards on the surface of the path, or caused by defects in the path itself. In this example, the stereo sensor data 108 may be derived from one or more on-board sensor pairs 110 of an ego-machine (e.g., ego-machine 900 of FIGS. 9A-9D).)
1. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 8. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; / 15. receiving a second portion of the image at a second feature extractor to provide a second feature vector, the second portion being different from the first portion; (Wu: [0038] In one embodiment, a first set of one or more feature vectors is computed for features extracted from the first image of the stereo image pair, and a second set of one or more features is computed for features extracted from the second image of the stereo image pair. For example, a Deep Neural Network (DNN) model may be used to extract a feature map of the first image (for example, the left image), and extract a feature map of the second image (for example, the right image). In this way, the DNN computes feature vectors for one or more pixels (e.g., each pixel) of the respective first and second images. [0041], [0069] In some embodiments, the machine learning feature vector evaluator 318 comprises a feature vector processing function 330 and an element classification function 332, each of which may be implemented at least in part by a DNN model. The feature vector processing function 330 inputs the stereo image pair (e.g. from the stereo image data 108) to extract a feature map of the first image (for example, the left image), and extract a feature map of the second image (for example, the right image). A first set of feature vectors is computed for features extracted from the first image of the stereo image pair, and a second set of features is computed for features extracted from the second image of the stereo image pair. Pixels that represent hazards appearing in the stereo image pair are expected have a distinguishable texture and pattern characteristics as compared to pixels representing the surface of the path. These distinguishing characteristics can be captured and described by the feature vectors computed by the DNN model implementing the feature vector processing function 330, which may be trained to recognize such semantic information and texture information. For example, for a given pixel (P) of the first image of the stereo image pair, a corresponding pixel of the second image of the stereo image pair is determined based on the path disparity model and/or path disparity map. A first feature vector (FV1) is extracted for the pixel (P) of the first image and a second feature vector (FV2) is extracted for the pixel determined to be the corresponding pixel to pixel (P) in the second image.)
1. and classifying the object using a classification model, / 8. and classifying the object using a classification model, / 15. and classifying the object using a classification model, (Wu: [0069] In some embodiments, the machine learning feature vector evaluator 318 comprises a feature vector processing function 330 and an element classification function 332, each of which may be implemented at least in part by a DNN model. The feature vector processing function 330 inputs the stereo image pair (e.g. from the stereo image data 108) to extract a feature map of the first image (for example, the left image), and extract a feature map of the second image (for example, the right image). A first set of feature vectors is computed for features extracted from the first image of the stereo image pair, and a second set of features is computed for features extracted from the second image of the stereo image pair. Pixels that represent hazards appearing in the stereo image pair are expected have a distinguishable texture and pattern characteristics as compared to pixels representing the surface of the path. These distinguishing characteristics can be captured and described by the feature vectors computed by the DNN model implementing the feature vector processing function 330, which may be trained to recognize such semantic information and texture information. For example, for a given pixel (P) of the first image of the stereo image pair, a corresponding pixel of the second image of the stereo image pair is determined based on the path disparity model and/or path disparity map. A first feature vector (FV1) is extracted for the pixel (P) of the first image and a second feature vector (FV2) is extracted for the pixel determined to be the corresponding pixel to pixel (P) in the second image. [0079]-[0080])
1. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 8. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. / 15. wherein classifying the object comprises applying at least the first feature vector and the second feature vector to the classification model. (Wu: [0079] The method 500 is drawn to detecting real-world hazardous objects in the path of an ego-machine based on extracted feature vectors and a path disparity model. Generally, the method 500 comprises determining a location of one or more pixels corresponding to one or more hazard objects on a path of an ego-machine based at least in part on classifying the one or more pixels using a first feature vector corresponding to a first image and a second feature vector corresponding to a second image, the first feature vector and the second feature vector determined to correspond to one another based at least in part on a path disparity model computed using a disparity map generated using the first image and the second image. [0080] The method 500 begins at B510 with generating a disparity map indicative of disparities between a first image generated using a first sensor and a second image generated using a second sensor. The first sensor and the second sensor have at least partially overlapping fields of view including at least a portion of a path of an ego-machine. In some embodiments, the method includes receiving a stereo image pair for a region of interest in a path of an ego-machine. The first and second sensors produce the stereo image pair. For example, FIG. 6 at 610 illustrates an image 612 from one of the stereo image pair showing the path 614 traveled by the ego-machine, and a delineated ROI 616 within which hazard detection is performed. At 620, an image space disparity map 622 is shown for the delineated ROI 616. Each pixel of the image space disparity map 622 within the ROI 616 represents the computed disparity for that pixel between the left and right images of the stereo image pair. As indicated by the image space disparity map 622, disparity of the surface of the path 614 inherently increases as points on the surface draw closer to the ego-machine. [0081])
2. The method of claim 1, wherein the second portion is larger than the first portion and fully overlaps the first portion. / 9. The computer program product of claim 8, wherein the second portion is larger than the first portion and fully overlaps the first portion. /16. The system of claim 15, wherein the second portion is larger than the first portion and fully overlaps the first portion. (Wu: [0057] For example, with reference to FIG. 2 , FIG. 2 illustrates an overlapping field of view (FOV) for a pair of cameras of vehicle 900, in accordance with some embodiments of the present disclosure. FIG. 2 includes vehicle 900, camera 202, camera 204 (which together form the sensor pair 110), field of view (FOV) 206, field of view (FOV) 208, and overlapping field of view (FOV) region 210. For example, cameras 202 and 204 may be synchronized stereo cameras that may be mounted on the vehicle 900—such as on a windshield or other area of the vehicle 900. The two cameras 202 and 204 may execute a cross-camera optical flow (OF) tracking algorithm to extract stereo disparity information from pairwise images. The camera 202 may provide the FOV 206 to the system and the camera 204 may provide the FOV 208 to the system. In some embodiments, cross-camera OF tracking may be executed at least in the overlapping FOV region 210 of the cameras 202 and 204. The overlapping FOV region 210 covers the defined region of interest (ROI) that includes the path of the ego-machine. Based on a disparity between pixels in the camera image data in the overlapping FOV region 210 for a particular location, the system may determine both the path disparity model 104 and the disparity maps 106. [0080] The method 500 begins at B510 with generating a disparity map indicative of disparities between a first image generated using a first sensor and a second image generated using a second sensor. The first sensor and the second sensor have at least partially overlapping fields of view including at least a portion of a path of an ego-machine. In some embodiments, the method includes receiving a stereo image pair for a region of interest in a path of an ego-machine. The first and second sensors produce the stereo image pair. For example, FIG. 6 at 610 illustrates an image 612 from one of the stereo image pair showing the path 614 traveled by the ego-machine, and a delineated ROI 616 within which hazard detection is performed. At 620, an image space disparity map 622 is shown for the delineated ROI 616. Each pixel of the image space disparity map 622 within the ROI 616 represents the computed disparity for that pixel between the left and right images of the stereo image pair. As indicated by the image space disparity map 622, disparity of the surface of the path 614 inherently increases as points on the surface draw closer to the ego-machine. [0081])
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify Turkelson’s method and system for image analysis and processing for scene and user-input context aided visual search and object classification with Wu’s ML models for disparity maps in the realm of object classification. Both references are in the same overall field of endeavor, and the determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify Turkelson in order to ensure that it leverages the teaching of Wu for a refined machine learning model that relies on enhanced accuracy with the use of disparity maps of overlapping regions for object detection and classification. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of Turkelson, while the teaching of Wu continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of an improved machine learning model for accurate object detection and classification. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claims 2, 9 and 16.
The combination of Turkelson, Wang and Wu teaches:
2. The method of claim 1, wherein the second portion is larger than the first portion and fully overlaps the first portion. / 9. The computer program product of claim 8, wherein the second portion is larger than the first portion and fully overlaps the first portion. /16. The system of claim 15, wherein the second portion is larger than the first portion and fully overlaps the first portion. (Wu: [0057] For example, with reference to FIG. 2 , FIG. 2 illustrates an overlapping field of view (FOV) for a pair of cameras of vehicle 900, in accordance with some embodiments of the present disclosure. FIG. 2 includes vehicle 900, camera 202, camera 204 (which together form the sensor pair 110), field of view (FOV) 206, field of view (FOV) 208, and overlapping field of view (FOV) region 210. For example, cameras 202 and 204 may be synchronized stereo cameras that may be mounted on the vehicle 900—such as on a windshield or other area of the vehicle 900. The two cameras 202 and 204 may execute a cross-camera optical flow (OF) tracking algorithm to extract stereo disparity information from pairwise images. The camera 202 may provide the FOV 206 to the system and the camera 204 may provide the FOV 208 to the system. In some embodiments, cross-camera OF tracking may be executed at least in the overlapping FOV region 210 of the cameras 202 and 204. The overlapping FOV region 210 covers the defined region of interest (ROI) that includes the path of the ego-machine. Based on a disparity between pixels in the camera image data in the overlapping FOV region 210 for a particular location, the system may determine both the path disparity model 104 and the disparity maps 106. [0080] The method 500 begins at B510 with generating a disparity map indicative of disparities between a first image generated using a first sensor and a second image generated using a second sensor. The first sensor and the second sensor have at least partially overlapping fields of view including at least a portion of a path of an ego-machine. In some embodiments, the method includes receiving a stereo image pair for a region of interest in a path of an ego-machine. The first and second sensors produce the stereo image pair. For example, FIG. 6 at 610 illustrates an image 612 from one of the stereo image pair showing the path 614 traveled by the ego-machine, and a delineated ROI 616 within which hazard detection is performed. At 620, an image space disparity map 622 is shown for the delineated ROI 616. Each pixel of the image space disparity map 622 within the ROI 616 represents the computed disparity for that pixel between the left and right images of the stereo image pair. As indicated by the image space disparity map 622, disparity of the surface of the path 614 inherently increases as points on the surface draw closer to the ego-machine. [0081])
Consider Claims 3, 10 and 17. CANCELLED
Consider Claims 4, 11 and 18.
The combination of Turkelson, Wang and Wu teaches:
4. The method of claim 1, further comprising: receiving positional information indicating a position of the first portion relative to at least the second portion, wherein classifying the object further comprises applying the positional information to the classification model. / 11. The computer program product of claim 8, the operation further comprising: receiving positional information indicating a position of the first portion relative to at least the second portion, wherein classifying the object further comprises applying the positional information to the classification model./ 18. The system of claim 15, the operation further comprising: receiving positional information indicating a position of the first portion relative to at least the second portion, wherein classifying the object further comprises applying the positional information to the classification model. (Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting). Wu: [0036] In some embodiments, to simplify estimation of the path disparity model, the image space disparity map is transformed to an updated (or “V”) disparity map. That is, in an image space disparity map, the pixel columns are represented on a first axis (which may referred to at the U axis) and pixel rows are represented on the second axis (which may be referred to as the V axis) and a disparity value at coordinates (u,v) of the disparity map may indicate a disparity associated with an image pixel at coordinates (x,y) of the image as captured by one of the sensors (e.g., either sensor O or O′). In V-disparity space a first axis of the disparity map instead corresponds to disparity values while the second axis again corresponds to pixel rows (the V axis). Each of the one or more elements in the V-disparity map indicates a count of disparity elements in the row of the original image space disparity map. [0057] In some embodiments, context classification subsystem 112 may output a classification vector including weights representative of the contexts determined for a given input image based on the context classification model. In some embodiments, the classification vector may upweight (e.g., if all weights are initialized to zero) features of the classification vector that were determined to be represented by the image. For example, an image depicting a snow-covered field may have a vector element associated with winter scenes upweighted (e.g., to a non-zero positive value), whereas a vector element associated with a desert or a beach may remain at its initialized value (e.g., zero). In some embodiments, additional features may be added to a feature vector for object recognition based on the contexts identified by context classification subsystem 112.)
Consider Claims 5, 12 and 19.
The combination of Turkelson, Wang and Wu teaches:
5. The method of claim 4, wherein the second portion is the image, and wherein the positional information comprises one of coordinates of the first portion within the image, and a position vector of the first portion within the image. / 12. The computer program product of claim 11, wherein the second portion is the image, and wherein the positional information comprises one of coordinates of the first portion within the image, and a position vector of the first portion within the image. / 19. The system of claim 18, wherein the second portion is the image, and wherein the positional information comprises one of coordinates of the first portion within the image, and a position vector of the first portion within the image. (Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting). Wu: [0036] In some embodiments, to simplify estimation of the path disparity model, the image space disparity map is transformed to an updated (or “V”) disparity map. That is, in an image space disparity map, the pixel columns are represented on a first axis (which may referred to at the U axis) and pixel rows are represented on the second axis (which may be referred to as the V axis) and a disparity value at coordinates (u,v) of the disparity map may indicate a disparity associated with an image pixel at coordinates (x,y) of the image as captured by one of the sensors (e.g., either sensor O or O′). In V-disparity space a first axis of the disparity map instead corresponds to disparity values while the second axis again corresponds to pixel rows (the V axis). Each of the one or more elements in the V-disparity map indicates a count of disparity elements in the row of the original image space disparity map. [0057] In some embodiments, context classification subsystem 112 may output a classification vector including weights representative of the contexts determined for a given input image based on the context classification model. In some embodiments, the classification vector may upweight (e.g., if all weights are initialized to zero) features of the classification vector that were determined to be represented by the image. For example, an image depicting a snow-covered field may have a vector element associated with winter scenes upweighted (e.g., to a non-zero positive value), whereas a vector element associated with a desert or a beach may remain at its initialized value (e.g., zero). In some embodiments, additional features may be added to a feature vector for object recognition based on the contexts identified by context classification subsystem 112.)
Consider Claims 6, 13 and 20.
The combination of Turkelson, Wang and Wu teaches:
6. The method of claim 1, wherein one or both of the first feature extractor and the second feature extractor have pretrained fixed parameters, the method further comprising: training the classification model using outputs from the first feature extractor and the second feature extractor. / 13. The computer program product of claim 8, wherein one or both of the first feature extractor and the second feature extractor have pretrained fixed parameters, the operation further comprising: training the classification model using outputs from the first feature extractor and the second feature extractor. / 20. The system of claim 15, wherein one or both of the first feature extractor and the second feature extractor have pretrained fixed parameters, the operation further comprising: training the classification model using outputs from the first feature extractor and the second feature extractor. (Turkelson: In some embodiments, the feature extraction process may use deep learning processing to extract features from an image. For example, a deep convolution neural network (CNN), trained on a large set of training data (e.g., the AlexNet architecture, which includes 5 convolutional layers and 3 fully connected layers, trained using the ImageNet dataset) may be used to extract features from an image. In some embodiments, to perform feature extraction, context classification subsystem 112 and object recognition subsystem 112 may obtain a pre-trained machine learning model from model database 138, which may be used for performing feature extraction for images from a set of images provided to computer system 102. In some embodiments, a support vector machine (SVM) may be trained with a training data to obtain a trained model for performing feature extraction. In some embodiments, a classifier may be trained using extracted features from an earlier layer of the machine learning model. In some embodiments, preprocessing may be performed to an input image prior to the feature extraction being performed. For example, preprocessing may include resizing, normalizing, cropping, etc., to each image to allow that image to serve as an input to the pre-trained model. Example pre-trained networks may include AlexNet, GoogLeNet, MobileNet-v2, and others. The preprocessing input images may be fed to the pre-trained model, which may extract features, and those features may then be used to train a classifier (e.g., SVM). In some embodiments, the input images, the features extracted from each of the input images, an identifier labeling each of the input image, or any other aspect capable of being used to describe each input image, or a combination thereof, may be stored in memory (e.g., within training data database 136 as an update to training data set for training an object recognition model, a context classification model, etc.). In some embodiments, a feature vector describing visual features extracted from an image may be output from context classification subsystem 112 and object recognition subsystem 114, which may describe one or more contexts of the image and one or more objects determined to be depicted by the image. In some embodiments, the feature vector, the input image, or both, may be used as an input to a visual search system (e.g., visual search subsystem 124) for performing a visual search to obtain information related to objects depicted within the image (e.g., products that a user may purchase). [0066] In some embodiments, model subsystem 116 may be configured to retrieve models stored within model database 138, provide the retrieved models to one or more subsystems for analyzing an image or set of images (e.g., to context classification subsystem 112, object recognition subsystem 114, etc.), as well as to train one or more models and generate training data for training the one or more models. For example, model subsystem 116 may be configured to train a context classification model to be used by context classification subsystem 112, an object recognition model to be used by object recognition subsystem 114, and the like. In some embodiments, model subsystem 116 may build or assist in the build of a given model.)
Consider Claims 21.
The combination of Turkelson, Wang and Wu teaches:
21. (New) The method of claim 1, wherein the first portion is a first crop of the image having a first size and the second portion is a second crop of the image having a different size.
(Wu: [0057] For example, with reference to FIG. 2 , FIG. 2 illustrates an overlapping field of view (FOV) for a pair of cameras of vehicle 900, in accordance with some embodiments of the present disclosure. FIG. 2 includes vehicle 900, camera 202, camera 204 (which together form the sensor pair 110), field of view (FOV) 206, field of view (FOV) 208, and overlapping field of view (FOV) region 210. For example, cameras 202 and 204 may be synchronized stereo cameras that may be mounted on the vehicle 900—such as on a windshield or other area of the vehicle 900. The two cameras 202 and 204 may execute a cross-camera optical flow (OF) tracking algorithm to extract stereo disparity information from pairwise images. The camera 202 may provide the FOV 206 to the system and the camera 204 may provide the FOV 208 to the system. In some embodiments, cross-camera OF tracking may be executed at least in the overlapping FOV region 210 of the cameras 202 and 204. The overlapping FOV region 210 covers the defined region of interest (ROI) that includes the path of the ego-machine. Based on a disparity between pixels in the camera image data in the overlapping FOV region 210 for a particular location, the system may determine both the path disparity model 104 and the disparity maps 106. [0080] The method 500 begins at B510 with generating a disparity map indicative of disparities between a first image generated using a first sensor and a second image generated using a second sensor. The first sensor and the second sensor have at least partially overlapping fields of view including at least a portion of a path of an ego-machine. In some embodiments, the method includes receiving a stereo image pair for a region of interest in a path of an ego-machine. The first and second sensors produce the stereo image pair. For example, FIG. 6 at 610 illustrates an image 612 from one of the stereo image pair showing the path 614 traveled by the ego-machine, and a delineated ROI 616 within which hazard detection is performed. At 620, an image space disparity map 622 is shown for the delineated ROI 616. Each pixel of the image space disparity map 622 within the ROI 616 represents the computed disparity for that pixel between the left and right images of the stereo image pair. As indicated by the image space disparity map 622, disparity of the surface of the path 614 inherently increases as points on the surface draw closer to the ego-machine. [0081] Turkleson: [0042] The image displayed on the UI may include an object or objects with which the user would like to search for and obtain information. In some embodiments, the UI may not include an explicit image capture button. In other words, the entire UI may display the contents viewed by the camera, and no capture image button physically or virtually may be available. In some embodiments, multiple objects may be recognized as being present within the input image, and the coordinate location of the user input (e.g., tap input) may be used to select one of the objects as being reflective of the user's intent based on the location of the user input in pixel coordinates (e.g., selecting the object having a centroid with a closest location to the touch location in pixel coordinates. In some embodiments, a distance from an input's coordinate location with respect to one or more bounding boxes may serve as an additional input for determining an object of interest for the user. [0043] In some embodiments, a visual search system may obtain the image and coordinate location information. The visual search system may extract features from the image and determine, based on locations of objects depicted by the image and the coordinate location information, a likely object (or other type of object) of interest for which the user is searching. Each object detected within the image may be reverse weighted with respect to the coordinate location information, and the object that is most proximate to the coordinate location information may be selected as the likely object of interest. For example, scores of an object recognition model may be adjusted based on the reverse weighting such that a first object for which the model indicates a lower confidence (indicated by a lower) score is selected over a second object with a higher confidence based on the first object being depicted in the image closer to the touch location than the second object (e.g., by multiplying the score by the reverse weighting).)
Claims 7, 14, 21 and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Turkelson et al. (US PGPub US 20200193206 A1, hereby referred to as “Turkelson”), in view of Wang (US PGPub 2011/0069152) in view of Wu et al. (US PGPub US 20230351769), hereby referred to as “Wu”, further in view of Georgeson et al. (US PGPub US 2014/0184786, hereby referred to as “Georgeson”).
Consider Claims 7 and 14.
The combination of Turkelson, Wang and Wu teaches:
7. The method of claim 1, wherein the image depicts external surfaces of a plurality of sections of a surface, wherein the first portion of image depicts an external surface of a first section of the plurality of sections, wherein classifying the object comprises distinguishing the first section. / 14. The computer program product of claim 8, wherein the image depicts external surfaces of a plurality of sections of a surface, wherein the first portion of image depicts an external surface of a first section of the plurality of sections, wherein classifying the object comprises distinguishing the first section./ 22. (New) The computer program product of claim 8, wherein first portion depicts an external surface of at least a first section of the object. / 23. (New) The system of claim 15, wherein the image depicts external surfaces of the object. (Wu: [0003] Another existing hazard detection technology leverages a Deep Neural Network (DNN) inference engine. The DNN is trained on the appearance and shapes of hazards to detect when a hazard appears in a captured image. However, given the wide variety of hazards that might be present on a road surface, large amounts of training data are needed, and obtaining such training data is not trivial due to the unknown and unlimited types of potential hazards. [0004], [0034] In some embodiments, blockwise division is used to subdivide the disparity map into a plurality of smaller disparity maps, each corresponding to a block of pixels of the disparity map. Typically, a road traveled by an ego-machine is not a flat plane. Rather, it can be expected to slope to either side (e.g., to allow for rain runoff), and have a surface that is not perfectly parallel to the row axis of the captured images. Both of these characteristics contribute to greater road surface disparity. By subdividing and treating each of these blocks as an individual processing unit for hazard detection, the disparity of the road surface itself within each block is considerably less than if the disparity of the entire road surface as captured by the stereo images were considered. After hazard detection is performed within each block, the results can then be correlated back to an image of the road surface within the ROI. [0051] The path travelled by the ego-machine is not limited to any one type of path or surface and may include paths such as a paced road, an unpaved road, a highway, a driveway, a portion of a parking lot, a trail, a track, a walking path, a delineated portion of an environment, or an aircraft runway or landing pad, for example. It should also be understood that the path travelled by the ego-machine may be inside a building or other facility and comprise a hallway, corridor or isle, for example. Turkelson: [0076] In some embodiments, pressure may also be detected by a touch-sensitive display screen. As an example, pressure sensitive device or pressure may be configured to determine an amount of pressure applied to a surface. Based on an amount of pressure, a characteristic of the input may be determined. In some embodiments, an amount of force that an input has may indicate a depth of focus of the image. For example, a hard touch may indicate a larger area of interest, whereas a soft touch may indicate a smaller area of interest, or vice versa. The amount of force, and the corresponding area that the input was detected by, may indicate what a user sought to select within an image, the user's focus or interest within an image, and the like. [0077] In some embodiments, eye gaze may also be used as an input channel for determining a location of an input to a display screen. In some embodiments, eye gaze may include tracking a position and movement of an individual's eyes to determine a location on a display screen (or other surface) that an individual's focus is directed towards.)
The combination of Turkelson and Wu does not teach the dependent claims from claims 7 and 14:
wherein the image depicts external surfaces of a plurality of sections of an aircraft
Georgeson teaches:
7. The method of claim 1, wherein the image depicts external surfaces of a plurality of sections of an aircraft, wherein the first portion of image depicts an external surface of a first section of the plurality of sections, wherein classifying the object comprises distinguishing the first section. / 14. The computer program product of claim 8, wherein the image depicts external surfaces of a plurality of sections of an aircraft, wherein the first portion of image depicts an external surface of a first section of the plurality of sections, wherein classifying the object comprises distinguishing the first section./ 22. (New) The computer program product of claim 8, wherein first portion depicts an external surface of at least a first section of the object. / 23. (New) The system of claim 15, wherein the image depicts external surfaces of the object. (Georgeson: abstract, A system for stand-off inspection comprising local positioning system hardware and a nondestructive evaluation instrument supported by a pan-tilt mechanism. The system further comprises a computer system that is programmed to perform the following operations: (a) directing the local positioning system hardware toward an area of a surface on a target object by control of the pan-tilt mechanism; (b) activating the local positioning system hardware to acquire an image; (c) processing the image to determine whether an anomaly is present in the area; (d) if an anomaly is present, determining coordinates of a position of the anomaly in a coordinate system of the target object; and (e) directing the nondestructive evaluation instrument toward a position corresponding to the coordinates. Optionally, the computer system is further programmed to measure one or more characteristics of the anomaly. [0034] Three-dimensional localization software may be loaded into computer 8. For example, the three-dimensional localization software may be of a type that uses multiple calibration points 15 at a distance on the target object 14 (such as a surface on an aircraft) to define the location (position and orientation) of video camera 2 relative to target object 14. In some applications, the three-dimensional localization software may utilize a minimum of three calibration points 15 on the target object 14, in combination with pan and tilt data from the pan-tilt mechanism 3, to define the relative position and orientation of the video camera 2 with respect to the local coordinate system 27 of the target object 14. The calibration points 15 may be visible features of known position in the local coordinate system 27 of the target object 14 as determined from a three-dimensional database of feature positions (e.g., a CAD model) or other measurement technique. The calibration points 15 may be used in coordination with the azimuth and elevation angles from the pan-tilt mechanism 3 to solve for the camera position and orientation relative to the target object 14.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the combination of Turkelson, Wang and Wu for a method and system for image analysis and processing for use of ML models in the realm of object classification context aided visual search and object classification with Georgeson’s specific model for target object detection algorithm for aircraft surfaces. Both references are in the same overall field of endeavor, and the determination of obviousness is predicated upon the following findings: One skilled in the art would have been motivated to modify the combination of Turkelson, Wang and Wu in order to ensure that it leverages the teaching of Georgeson for a refined machine learning model that can be applied for for object detection and classification on aircraft surfaces. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and programming techniques, without changing a “fundamental” operating principle of the combination of Turkelson, Wang and Wu, while the teaching of Georgeson continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of an improved machine learning model for accurate object detection and classification for aircraft surfaces. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA N ANSARI whose telephone number is (571)270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O' NEAL MISTRY can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
TAHMINA N. ANSARI
Examiner
Art Unit 2672
2672
August 31, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights