DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-11 and 14-23 are pending for examination in the application filed 02/26/2026. Claims 1-11 and 14-20 have been amended, claims 12-13 are cancelled, and claims 21-23 are new.
Priority
Acknowledgement is made of Applicant’s claim to priority of provisional application 63/409,048, filing date 09/22/2022.
Response to Arguments and Amendments
Applicant’s arguments with respect to claim 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument, as facilitated by the newly added amendments. Applicant arguments on page 10 of the Remarks filed 02/28/2026 that Theverapperuma fails to overcome the above-identified deficiencies of Rybakov but does not provide any explanation. Applicant argues that the alleged combination of references do not even recognize the problems addressed by the claims, let alone teach or suggest a solution similar to the claims and that there is no motivation or suggestion in the references to combine the references. The remarks do not provide any specific reasons as to why either the findings of fact or the legal conclusion of obviousness is allegedly in error and thus do not comply with 37 CFR 1.111(b) and MPEP § 714.02. However, Applicant’s reply is considered to be a bona fide attempt at a response and is being accepted as a complete response. In order to establish a prima facie case of obviousness, Examiner must set forth (a) the relevant teachings of the prior art relied upon, (b) the differences between the prior art in the claim and the applied references, (c) the proposed modification of the applied references necessary to arrive at the claimed subject matter, and (d) an explanation as to why the claimed invention would have been obvious to one of ordinary skill in the art at the relevant time. See MPEP 2142. Here, Examiner has mapped the Rybakov reference to the claim, explained the deficiencies of the Rybakov reference, proposed a modification of the Rybakov reference with the Theverapperuma reference, and provided a motivation for the combination. Therefore, a prima facie case of obviousness has been made. Please see the updated 35 USC § 103 rejections in view of the newly added amendments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-6, 8-9, and 11, 14-21 are rejected under 35 U.S.C. 103 as being unpatentable over Rybakov (US9171195B1) in view of Theverapperuma (US20220024485A1).
Regarding claim 1, Rybakov teaches a computer implemented method for identifying points of interest in an image, the method comprising (Fig. 3-4. [col. 2 ln. 61-62] The server 104 identifies features points of the query image (134)):
obtaining a two-dimensional input image of a scene ([col. 2 ln. 34-41] At runtime, the server 104 may receive a communication from a user device, such as smartphone 102 over a network 106 or through another communication method. The communication may include a request to identify a specific object seen by a camera(s) of the device 102. The device 102 may send video/image data from its camera(s) to the server 104. The image data may be sent as a stream of images taken from the camera(s));
generating, using a feature generation model, a feature map for the input image ([col. 2 ln. 9-15] For example, several images of a particular object may be taken from multiple points of view (for example, left view, right view, top view, bottom view, etc.). These images from the multiple points of view may then be used to train a planar recognizer. [col. 10 ln. 20-25] The system may then determine feature points, feature vectors, a keyword, or other representations (such as texture, color, etc.) of the contents of the query image (406). The keyword may be determined based on a mathematical function related to the feature points and/or feature vector of the query image);
comparing the plurality of feature vectors included in the feature map with reference feature vectors generated by the feature generation model based on reference points within a reference image; and based on the comparing, identifying at least one point of interest ([col. 5 ln. 40-57] A server 104 receives multiple images, each image showing a different angle of an object (302). The server may associate images showing angles of a same object together (304) either as part of a tree (discussed below) or separately. An object identifier (which may be a unique alpha-numeric code) corresponding to a displayed image may be associated with an image showing the object. The database images may include only one object per image (which assists in image mapping) or, in certain instances, may include multiple objects in one image. In that case, the particular image may be associated with the multiple object identifiers corresponding to the displayed images. The server 104 may determine feature points and/or pyramid images (306) for the database images. These feature points and pyramid images are associated with their corresponding images. The server may also determine feature vectors, keywords, and/or other representations or features of the contents of the images (308). [col. 6 ln. 38-46] The image matching algorithm of the present invention finds a match of query image from among multiple database images. As explained below, the image matching algorithm may operate by representing images in terms of feature points, orientations and feature vectors. After a representation of the query image and database images has been created, the feature points, orientations and feature vectors of the images are used to determine a match between the images. [col. 7 ln. 27-30] The pair of putatively matching images can be determined to be a match when a sufficient number of feature points match the corresponding feature points of the other image both visually and geometrically).
Rybakov does not explicitly teach the input image comprising a plurality of pixels; using a machine learning based feature generation model, the feature map comprising a plurality of feature vectors each associated with a respective one of the plurality of pixels.
Theverapperuma, in the same field of endeavor of feature generation, teaches the input image comprising a plurality of pixels ([0007] In certain embodiments, a method involves receiving, by a controller system of an autonomous vehicle, sensor data from a plurality of sensors. The sensor data comprises at least one camera image of a physical environment and a first three-dimensional (3D) representation of the physical environment. The method further involves extracting, by the controller system, a set of features from the at least one camera image. [0095] (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404);
using a machine learning based feature generation model ([0006] In certain embodiments, at least some of the modules in the surface identification subsystem are implemented using a machine learning model (e.g., a convolutional neural network or CNN). [0007] The method further involves extracting, by the controller system, a set of features from the at least one camera image. The extracting comprises inputting the at least one camera image to a neural network trained to infer values of the set of features from image data),
the feature map comprising a plurality of feature vectors each associated with a respective one of the plurality of pixels ([0095] the inclusion of the feature extractor 422 in the embodiment of FIG. 4 increases computational efficiency by reducing the dimensionality of the input image space (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404). [0094] Feature extractor 422 operates as a backbone network for the extraction of image features. In particular, the feature extractor 422 is configured to extract values for a set of features represented in the data from the cameras 402, 404. The feature extractor 422 can be implemented as a neural network that has been trained (e.g., through supervised learning and backpropagation) to generate a vector or multi-dimensional tensor for input to each of the modules 424, 426, and 428. The vector or multi-dimensional tensor is an abstract representation of a 2D image that combines information from the individual camera images).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Theverapperuma to use machine learning for "generating an augmented image or a segmented image comprises inputting values of a set of features to a neural network trained using images of surface deformations associated with drivable surfaces" [Theverapperuma 0009] and for the feature map to comprise a plurality of feature vectors associated with a plurality of pixels because "the inclusion of the feature extractor 422 in the embodiment of FIG. 4 increases computational efficiency by reducing the dimensionality of the input image space (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404)" [Theverapperuma 0095] and "the segmented image can be an RGB formatted 2D image in which each pixel has been assigned a class of “road” or a class of “non-road”. Thus, the segmented image can represent the result of performing classification on the extracted features, possibly classification that divides regions in the segmented image into one of two types of surfaces: potentially drivable and non-drivable" [Theverapperuma 0098].
Regarding claim 2, Rybakov and Theverapperuma teach the method of claim 1. Rybakov further teaches wherein the two-dimensional input image is obtained using a camera ([col. 2 ln. 45-47] the stream may be a series of images taken from a camera of a mobile device as a user holds or moves the mobile device near an object) and includes two-dimensional color data or grayscale data ([col. 2 ln. 61-64] The server 104 identifies features points of the query image (134). The features may include feature points of the image (as described below), textures in the image, colors in the image, etc.), and the feature generation model generates the feature map in absence of depth data for the pixels of the two-dimensional input image ([col. 2 ln. 9-15] For example, several images of a particular object may be taken from multiple points of view (for example, left view, right view, top view, bottom view, etc.). These images from the multiple points of view may then be used to train a planar recognizer).
Rybakov does not teach two-dimensional color data or grayscale data arranged in an array of pixels, each pixel corresponding to respective physical location within the scene.
Theverapperuma teaches two-dimensional color data or grayscale data arranged in an array of pixels, each pixel corresponding to respective physical location within the scene ([0098] Surface segmentation module 426 is configured to generate, using the extracted features, a segmented image that is divided into different surfaces. The segmented image is a 2D image indicating which areas correspond to potentially drivable surfaces (e.g., road surfaces) and which areas correspond to non-drivable surfaces (e.g., grass, hills, or other terrain). For example, the segmented image can be an RGB formatted 2D image in which each pixel has been assigned a class of “road” or a class of “non-road”. [0109] RGB-D images are another example of a quasi-3D representation in which, for any given pixel at image coordinates (x, y), only one depth value is assigned to the pixel. Irrespective of whether the output representation 450 is 3D or quasi-3D, each elementary unit (e.g., an individual voxel or grid tile) in the output representation 450 can be assigned a label indicating whether the corresponding location in the physical environment is drivable or potentially drivable).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Theverapperuma to arrange the 2D color data in an array of pixels where each pixel corresponds to a respective physical location for "indicating whether the corresponding location in the physical environment is drivable or potentially drivable" [Theverapperuma 0109].
Regarding claim 3, Rybakov and Theverapperuma teach the method of claim 1. Rybakov further teaches training the feature generation model, the training comprising: obtaining a set of training images that includes a plurality of corresponding image sets, each corresponding image set including multiple non-identical images wherein a plurality of same physical points are represented at different respective locations across the multiple non-identical images ([col. 3 ln. 65 - col. 4 ln. 14] To build a database of object images, with multiple objects per image, a number of different images of an object may be taken from different viewpoints. From those images, feature points may be extracted and pyramid images constructed. Multiple images from different points of view of each particular object may be taken and linked within the database (for example within a tree structure described below). The multiple images may correspond to different viewpoints of the object sufficient to identify the object from any later angle that may be included in a user's query image. For example, a shoe may look very different from a bottom view than from a top view than from a side view. For certain objects, this number of different image angles may be 6 (top, bottom, left side, right side, front, back), for other objects this may be more or less depending on various factors, including how many images should be taken to ensure the object may be recognized in an incoming query image);
generating an image label set for the set of training images, the image label set identifying, by respective locations, groups of the same physical points across the multiple images included in each corresponding image set ([col. 2 ln. 21-33] As shown, a server 104 that performs image recognition may, during a training period, receive multiple images showing different angles of an object (120). The server 104 may build an image database using the multiple angles (122). The server 104 may also perform pre-processing on these images to identify feature points and keywords (as described below) to potentially match images in the database with incoming images. The database images may be associated with certain objects or object IDs so images of incoming objects may be recognized. For example, different images 160 of shoes may be taken from different angles, processed, and stored in an image database on the server 104 or elsewhere);
and using the set of training images and the image label set to train the feature generation model to generate feature vectors with an objective of generating identical feature vectors for locations that correspond to the same physical points ([col. 6 ln. 38-46] The image matching algorithm of the present invention finds a match of query image from among multiple database images. As explained below, the image matching algorithm may operate by representing images in terms of feature points, orientations and feature vectors. After a representation of the query image and database images has been created, the feature points, orientations and feature vectors of the images are used to determine a match between the images).
Rybakov does not teach generating pixel feature vectors for pixel locations.
Theverapperuma teaches generating pixel feature vectors for pixel locations ([0095] the inclusion of the feature extractor 422 in the embodiment of FIG. 4 increases computational efficiency by reducing the dimensionality of the input image space (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404). [0096] Depth estimation module 424 is configured to generate a depth image, e.g., an RGB-D (red, green, blue, and depth) image, based on the features extracted by the feature extractor 422. Each pixel in the depth image is assigned a depth value indicating the depth at the location represented by the pixel).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Theverapperuma to generate pixel feature vectors for pixel locations (Rybakov teaches generating feature vectors for locations but does not specify on a pixel-level) for "indicating whether the corresponding location in the physical environment is drivable or potentially drivable" [Theverapperuma 0109].
Regarding claim 4, Rybakov and Theverapperuma teach the method of claim 3. Rybakov further teaches said training the feature generation model ([col. 3 ln. 65 - col. 4 ln. 14] To build a database of object images, with multiple objects per image, a number of different images of an object may be taken from different viewpoints. From those images, feature points may be extracted and pyramid images constructed. Multiple images from different points of view of each particular object may be taken and linked within the database (for example within a tree structure described below). The multiple images may correspond to different viewpoints of the object sufficient to identify the object from any later angle that may be included in a user's query image. For example, a shoe may look very different from a bottom view than from a top view than from a side view. For certain objects, this number of different image angles may be 6 (top, bottom, left side, right side, front, back), for other objects this may be more or less depending on various factors, including how many images should be taken to ensure the object may be recognized in an incoming query image).
Rybakov does not teach wherein training the feature generation model comprises generating depth information for each of the corresponding image sets using a machine learning based depth generating model, wherein the generated depth information is used together with the set of training images and the image label set to train the feature generation model.
Theverapperuma teaches wherein training the feature generation model comprises generating depth information for each of the corresponding image sets using a machine learning based depth generating model, wherein the generated depth information is used together with the set of training images and the image label set to train the feature generation model ([0096] Depth estimation module 424 is configured to generate a depth image, e.g., an RGB-D (red, green, blue, and depth) image, based on the features extracted by the feature extractor 422. Each pixel in the depth image is assigned a depth value indicating the depth at the location represented by the pixel. [0097] Training of the depth estimation module 424 may involve providing the depth estimation module 424 with training images depicting surfaces and/or objects, at different distances away from the camera that captured the training image. The depth images generated as a result of processing the training images can then be compared to corresponding ground truth depth information (e.g., the correct depth value for each pixel in a training image) to adjust the CNN by changing weights and/or bias values for one or more layers of the CNN such that a loss function is minimized).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Theverapperuma to generate depth information and use it in training the model because "If the camera data, as represented in the features extracted by the feature extractor 422, captures a drivable surface, then the depth values for the drivable surface (e.g., the depth at various points along the drivable surface) will have been determined by virtue of estimating the depth for each pixel in the depth image" [Theverapperuma 0096].
Regarding claim 5, Rybakov and Theverapperuma teach the method of claim 4. Rybakov further teaches obtaining, in addition to the two-dimensional input image, one or more further two-dimensional input images of the scene, the two-dimensional input image and the one or more further two-dimensional input images each corresponding to a different respective camera view of the scene ([col. 3 ln. 65 - col. 4 ln. 14] To build a database of object images, with multiple objects per image, a number of different images of an object may be taken from different viewpoints. From those images, feature points may be extracted and pyramid images constructed. Multiple images from different points of view of each particular object may be taken and linked within the database (for example within a tree structure described below). The multiple images may correspond to different viewpoints of the object sufficient to identify the object from any later angle that may be included in a user's query image. For example, a shoe may look very different from a bottom view than from a top view than from a side view. For certain objects, this number of different image angles may be 6 (top, bottom, left side, right side, front, back), for other objects this may be more or less depending on various factors, including how many images should be taken to ensure the object may be recognized in an incoming query image);
generating, using the feature generation model, a respective feature map of respective feature vectors for each of the one or more further two-dimensional input images; and further comparing the further feature vectors included in the further feature maps with the reference feature vectors; wherein said identifying the at least one point of interest in the two-dimensional input image is also based on the further comparing ([col. 2 ln. 21-33] As shown, a server 104 that performs image recognition may, during a training period, receive multiple images showing different angles of an object (120). The server 104 may build an image database using the multiple angles (122). The server 104 may also perform pre-processing on these images to identify feature points and keywords (as described below) to potentially match images in the database with incoming images. The database images may be associated with certain objects or object IDs so images of incoming objects may be recognized. For example, different images 160 of shoes may be taken from different angles, processed, and stored in an image database on the server 104 or elsewhere. [col. 6 ln. 38-46] The image matching algorithm of the present invention finds a match of query image from among multiple database images. As explained below, the image matching algorithm may operate by representing images in terms of feature points, orientations and feature vectors. After a representation of the query image and database images has been created, the feature points, orientations and feature vectors of the images are used to determine a match between the images. [col. 7 ln. 27-30] The pair of putatively matching images can be determined to be a match when a sufficient number of feature points match the corresponding feature points of the other image both visually and geometrically).
Rybakov does not teach using a machine learning based feature generation model.
Theverapperuma teaches using a machine learning based feature generation model ([0006] In certain embodiments, at least some of the modules in the surface identification subsystem are implemented using a machine learning model (e.g., a convolutional neural network or CNN). [0007] The method further involves extracting, by the controller system, a set of features from the at least one camera image. The extracting comprises inputting the at least one camera image to a neural network trained to infer values of the set of features from image data).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Theverapperuma to use machine learning for "generating an augmented image or a segmented image comprises inputting values of a set of features to a neural network trained using images of surface deformations associated with drivable surfaces" [Theverapperuma 0009].
Regarding claim 6, Rybakov and Theverapperuma teach the method of claim 5. Rybakov further teaches obtaining the reference feature vectors, including: obtaining the reference image and one or more further reference images, the reference image and the one or more further reference images each corresponding to a different respective camera view ([col. 3 ln. 65 - col. 4 ln. 14] To build a database of object images, with multiple objects per image, a number of different images of an object may be taken from different viewpoints. From those images, feature points may be extracted and pyramid images constructed. Multiple images from different points of view of each particular object may be taken and linked within the database (for example within a tree structure described below). The multiple images may correspond to different viewpoints of the object sufficient to identify the object from any later angle that may be included in a user's query image. For example, a shoe may look very different from a bottom view than from a top view than from a side view. For certain objects, this number of different image angles may be 6 (top, bottom, left side, right side, front, back), for other objects this may be more or less depending on various factors, including how many images should be taken to ensure the object may be recognized in an incoming query image);
identifying, for each of the reference points, a corresponding set of points across the reference image and the one or more further reference images; for each of the reference points, using the feature generation model to generate respective corresponding point feature vectors for each of the points included in the set of points corresponding to the reference point; and for each reference point, generating a respective one of the reference feature vectors based on the respective corresponding point feature vectors generated for each of the points included in the set of points corresponding to the reference point ([col. 5 ln. 40-57] A server 104 receives multiple images, each image showing a different angle of an object (302). The server may associate images showing angles of a same object together (304) either as part of a tree (discussed below) or separately. An object identifier (which may be a unique alpha-numeric code) corresponding to a displayed image may be associated with an image showing the object. The database images may include only one object per image (which assists in image mapping) or, in certain instances, may include multiple objects in one image. In that case, the particular image may be associated with the multiple object identifiers corresponding to the displayed images. The server 104 may determine feature points and/or pyramid images (306) for the database images. These feature points and pyramid images are associated with their corresponding images. The server may also determine feature vectors, keywords, and/or other representations or features of the contents of the images (308). [col. 6 ln. 38-46] The image matching algorithm of the present invention finds a match of query image from among multiple database images. As explained below, the image matching algorithm may operate by representing images in terms of feature points, orientations and feature vectors. After a representation of the query image and database images has been created, the feature points, orientations and feature vectors of the images are used to determine a match between the images. [col. 7 ln. 27-30] The pair of putatively matching images can be determined to be a match when a sufficient number of feature points match the corresponding feature points of the other image both visually and geometrically).
Regarding claim 8, Rybakov and Theverapperuma teach the method of claim 6. Rybakov further teaches wherein one or more cameras that are capable of capturing two-dimensional images but not enabled to capture an image depth dimension are used to obtain each of the two-dimensional input image, the one or more further two-dimensional input images, the reference image and the one or more further reference images (([col. 2 ln. 45-47] the stream may be a series of images taken from a camera of a mobile device as a user holds or moves the mobile device near an object. [col. 2 ln. 61-64] The server 104 identifies features points of the query image (134). The features may include feature points of the image (as described below), textures in the image, colors in the image, etc. [col. 2 ln. 9-15] For example, several images of a particular object may be taken from multiple points of view (for example, left view, right view, top view, bottom view, etc.). These images from the multiple points of view may then be used to train a planar recognizer. [col. 4 ln. 19-26] This process may be repeated for multiple objects. For large databases, such as an online shopping database where a user may submit an image of an object to be purchased (or simply identified), this process may be repeated thousands, if not millions of times to construct a database of images and data for image matching. The database also may continually be updated and/or refined to account for a changing catalog of objects to be recognized).
Regarding claim 9, Rybakov and Theverapperuma teach the method of claim 3. Rybakov further teaches further comprising retraining the feature generation model based on an updated set of training images that include one or more images previously obtained as two-dimensional input images of the scene ([col. 4 ln. 19-26] This process may be repeated for multiple objects. For large databases, such as an online shopping database where a user may submit an image of an object to be purchased (or simply identified), this process may be repeated thousands, if not millions of times to construct a database of images and data for image matching. The database also may continually be updated and/or refined to account for a changing catalog of objects to be recognized).
Regarding claim 11, Rybakov and Theverapperuma teach the method of claim 3. Rybakov further teaches wherein said obtaining the set of training images comprises, for each of the corresponding image sets: obtaining at least a first image and a second image using different camera views ([col. 2 ln. 21-33] As shown, a server 104 that performs image recognition may, during a training period, receive multiple images showing different angles of an object (120). The server 104 may build an image database using the multiple angles (122). The server 104 may also perform pre-processing on these images to identify feature points and keywords (as described below) to potentially match images in the database with incoming images. The database images may be associated with certain objects or object IDs so images of incoming objects may be recognized. For example, different images 160 of shoes may be taken from different angles, processed, and stored in an image database on the server 104 or elsewhere).
Regarding claim 14, Rybakov teaches a processing system comprising one or more processing devices and one or more memories coupled to the one or more processing devices (Fig. 5), the processing system being configured for identifying points of interest in an image by ([col. 1 ln. 20-22] FIG. 1 illustrates an overview of a system for recognizing three dimensional objects according to aspects of the present disclosure. [col. 2 ln. 61-62] The server 104 identifies features points of the query image (134)):
obtaining a two-dimensional input image of a scene ([col. 2 ln. 34-41] At runtime, the server 104 may receive a communication from a user device, such as smartphone 102 over a network 106 or through another communication method. The communication may include a request to identify a specific object seen by a camera(s) of the device 102. The device 102 may send video/image data from its camera(s) to the server 104. The image data may be sent as a stream of images taken from the camera(s));
generating, using a feature generation model, a feature map for the input image ([col. 2 ln. 9-15] For example, several images of a particular object may be taken from multiple points of view (for example, left view, right view, top view, bottom view, etc.). These images from the multiple points of view may then be used to train a planar recognizer. [col. 10 ln. 20-25] The system may then determine feature points, feature vectors, a keyword, or other representations (such as texture, color, etc.) of the contents of the query image (406). The keyword may be determined based on a mathematical function related to the feature points and/or feature vector of the query image);
comparing the plurality of feature vectors included in the feature map with reference feature vectors generated by the feature generation model based on reference points within a reference image; and based on the comparing, identifying at least one point of interest ([col. 5 ln. 40-57] A server 104 receives multiple images, each image showing a different angle of an object (302). The server may associate images showing angles of a same object together (304) either as part of a tree (discussed below) or separately. An object identifier (which may be a unique alpha-numeric code) corresponding to a displayed image may be associated with an image showing the object. The database images may include only one object per image (which assists in image mapping) or, in certain instances, may include multiple objects in one image. In that case, the particular image may be associated with the multiple object identifiers corresponding to the displayed images. The server 104 may determine feature points and/or pyramid images (306) for the database images. These feature points and pyramid images are associated with their corresponding images. The server may also determine feature vectors, keywords, and/or other representations or features of the contents of the images (308). [col. 6 ln. 38-46] The image matching algorithm of the present invention finds a match of query image from among multiple database images. As explained below, the image matching algorithm may operate by representing images in terms of feature points, orientations and feature vectors. After a representation of the query image and database images has been created, the feature points, orientations and feature vectors of the images are used to determine a match between the images. [col. 7 ln. 27-30] The pair of putatively matching images can be determined to be a match when a sufficient number of feature points match the corresponding feature points of the other image both visually and geometrically).
Rybakov does not explicitly teach the input image comprising a plurality of pixels; using a machine learning based feature generation model, the feature map comprising a plurality of feature vectors each associated with a respective one of the plurality of pixels.
Theverapperuma, in the same field of endeavor of feature generation, teaches the input image comprising a plurality of pixels ([0007] In certain embodiments, a method involves receiving, by a controller system of an autonomous vehicle, sensor data from a plurality of sensors. The sensor data comprises at least one camera image of a physical environment and a first three-dimensional (3D) representation of the physical environment. The method further involves extracting, by the controller system, a set of features from the at least one camera image. [0095] (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404);
using a machine learning based feature generation model ([0006] In certain embodiments, at least some of the modules in the surface identification subsystem are implemented using a machine learning model (e.g., a convolutional neural network or CNN). [0007] The method further involves extracting, by the controller system, a set of features from the at least one camera image. The extracting comprises inputting the at least one camera image to a neural network trained to infer values of the set of features from image data),
the feature map comprising a plurality of feature vectors each associated with a respective one of the plurality of pixels ([0095] the inclusion of the feature extractor 422 in the embodiment of FIG. 4 increases computational efficiency by reducing the dimensionality of the input image space (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404). [0094] Feature extractor 422 operates as a backbone network for the extraction of image features. In particular, the feature extractor 422 is configured to extract values for a set of features represented in the data from the cameras 402, 404. The feature extractor 422 can be implemented as a neural network that has been trained (e.g., through supervised learning and backpropagation) to generate a vector or multi-dimensional tensor for input to each of the modules 424, 426, and 428. The vector or multi-dimensional tensor is an abstract representation of a 2D image that combines information from the individual camera images).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Rybakov with the teachings of Theverapperuma to use machine learning for "generating an augmented image or a segmented image comprises inputting values of a set of features to a neural network trained using images of surface deformations associated with drivable surfaces" [Theverapperuma 0009] and for the feature map to comprise a plurality of feature vectors associated with a plurality of pixels because "the inclusion of the feature extractor 422 in the embodiment of FIG. 4 increases computational efficiency by reducing the dimensionality of the input image space (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404)" [Theverapperuma 0095] and "the segmented image can be an RGB formatted 2D image in which each pixel has been assigned a class of “road” or a class of “non-road”. Thus, the segmented image can represent the result of performing classification on the extracted features, possibly classification that divides regions in the segmented image into one of two types of surfaces: potentially drivable and non-drivable" [Theverapperuma 0098].
Regarding claim 15, Rybakov and Theverapperuma teach the system of claim 14. Rybakov further teaches wherein the two-dimensional input image is obtained using a camera ([col. 2 ln. 45-47] the stream may be a series of images taken from a camera of a mobile device as a user holds or moves the mobile device near an object) and includes two-dimensional color data or grayscale data ([col. 2 ln. 61-64] The server 104 identifies features points of the query image (134). The features may include feature points of the image (as described below), textures in the image, colors in the image, etc.), and the feature generation model generates the feature map in absence of depth data for the pixels of the two-dimensional input image ([col. 2 ln. 9-15] For example, several images of a particular object may be taken from multiple points of view (for example, left view, right view, top view, bottom view, etc.). These images from the multiple points of view may then be used to train a planar recognizer).
Rybakov does not teach two-dimensional color data or grayscale data arranged in an array of pixels, each pixel corresponding to respective physical location within the scene.
Theverapperuma teaches two-dimensional color data or grayscale data arranged in an array of pixels, each pixel corresponding to respective physical location within the scene ([0098] Surface segmentation module 426 is configured to generate, using the extracted features, a segmented image that is divided into different surfaces. The segmented image is a 2D image indicating which areas correspond to potentially drivable surfaces (e.g., road surfaces) and which areas correspond to non-drivable surfaces (e.g., grass, hills, or other terrain). For example, the segmented image can be an RGB formatted 2D image in which each pixel has been assigned a class of “road” or a class of “non-road”. [0109] RGB-D images are another example of a quasi-3D representation in which, for any given pixel at image coordinates (x, y), only one depth value is assigned to the pixel. Irrespective of whether the output representation 450 is 3D or quasi-3D, each elementary unit (e.g., an individual voxel or grid tile) in the output representation 450 can be assigned a label indicating whether the corresponding location in the physical environment is drivable or potentially drivable).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Rybakov with the teachings of Theverapperuma to arrange the 2D color data in an array of pixels where each pixel corresponds to a respective physical location for "indicating whether the corresponding location in the physical environment is drivable or potentially drivable" [Theverapperuma 0109].
Regarding claim 16, Rybakov and Theverapperuma teach the system of claim 14. Rybakov further teaches wherein the processing system is further configured to train the feature generation model, the training comprising: obtaining a set of training images that includes a plurality of corresponding image sets, each corresponding image set including multiple non-identical images wherein a plurality of same physical points are represented at different respective locations across the multiple non-identical images ([col. 3 ln. 65 - col. 4 ln. 14] To build a database of object images, with multiple objects per image, a number of different images of an object may be taken from different viewpoints. From those images, feature points may be extracted and pyramid images constructed. Multiple images from different points of view of each particular object may be taken and linked within the database (for example within a tree structure described below). The multiple images may correspond to different viewpoints of the object sufficient to identify the object from any later angle that may be included in a user's query image. For example, a shoe may look very different from a bottom view than from a top view than from a side view. For certain objects, this number of different image angles may be 6 (top, bottom, left side, right side, front, back), for other objects this may be more or less depending on various factors, including how many images should be taken to ensure the object may be recognized in an incoming query image);
generating an image label set for the set of training images, the image label set identifying, by respective locations, groups of the same physical points across the multiple images included in each corresponding image set ([col. 2 ln. 21-33] As shown, a server 104 that performs image recognition may, during a training period, receive multiple images showing different angles of an object (120). The server 104 may build an image database using the multiple angles (122). The server 104 may also perform pre-processing on these images to identify feature points and keywords (as described below) to potentially match images in the database with incoming images. The database images may be associated with certain objects or object IDs so images of incoming objects may be recognized. For example, different images 160 of shoes may be taken from different angles, processed, and stored in an image database on the server 104 or elsewhere);
and using the set of training images and the image label set to train the feature generation model to generate feature vectors with an objective of generating identical feature vectors for locations that correspond to the same physical points ([col. 6 ln. 38-46] The image matching algorithm of the present invention finds a match of query image from among multiple database images. As explained below, the image matching algorithm may operate by representing images in terms of feature points, orientations and feature vectors. After a representation of the query image and database images has been created, the feature points, orientations and feature vectors of the images are used to determine a match between the images).
Rybakov does not teach generating pixel feature vectors for pixel locations.
Theverapperuma teaches generating pixel feature vectors for pixel locations ([0095] the inclusion of the feature extractor 422 in the embodiment of FIG. 4 increases computational efficiency by reducing the dimensionality of the input image space (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404). [0096] Depth estimation module 424 is configured to generate a depth image, e.g., an RGB-D (red, green, blue, and depth) image, based on the features extracted by the feature extractor 422. Each pixel in the depth image is assigned a depth value indicating the depth at the location represented by the pixel).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Rybakov with the teachings of Theverapperuma to generate pixel feature vectors for pixel locations (Rybakov teaches generating feature vectors for locations but does not specify on a pixel-level) for "indicating whether the corresponding location in the physical environment is drivable or potentially drivable" [Theverapperuma 0109].
Regarding claim 17, Rybakov and Theverapperuma teach the system of claim 16. Rybakov further teaches said training the feature generation model ([col. 3 ln. 65 - col. 4 ln. 14] To build a database of object images, with multiple objects per image, a number of different images of an object may be taken from different viewpoints. From those images, feature points may be extracted and pyramid images constructed. Multiple images from different points of view of each particular object may be taken and linked within the database (for example within a tree structure described below). The multiple images may correspond to different viewpoints of the object sufficient to identify the object from any later angle that may be included in a user's query image. For example, a shoe may look very different from a bottom view than from a top view than from a side view. For certain objects, this number of different image angles may be 6 (top, bottom, left side, right side, front, back), for other objects this may be more or less depending on various factors, including how many images should be taken to ensure the object may be recognized in an incoming query image).
Rybakov does not teach wherein training the feature generation model comprises generating depth information for each of the corresponding image sets using a machine learning based depth generating model, wherein the generated depth information is used together with the set of training images and the image label set to train the feature generation model.
Theverapperuma teaches wherein training the feature generation model comprises generating depth information for each of the corresponding image sets using a machine learning based depth generating model, wherein the generated depth information is used together with the set of training images and the image label set to train the feature generation model ([0096] Depth estimation module 424 is configured to generate a depth image, e.g., an RGB-D (red, green, blue, and depth) image, based on the features extracted by the feature extractor 422. Each pixel in the depth image is assigned a depth value indicating the depth at the location represented by the pixel. [0097] Training of the depth estimation module 424 may involve providing the depth estimation module 424 with training images depicting surfaces and/or objects, at different distances away from the camera that captured the training image. The depth images generated as a result of processing the training images can then be compared to corresponding ground truth depth information (e.g., the correct depth value for each pixel in a training image) to adjust the CNN by changing weights and/or bias values for one or more layers of the CNN such that a loss function is minimized).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Rybakov with the teachings of Theverapperuma to generate depth information and use it in training the model because "If the camera data, as represented in the features extracted by the feature extractor 422, captures a drivable surface, then the depth values for the drivable surface (e.g., the depth at various points along the drivable surface) will have been determined by virtue of estimating the depth for each pixel in the depth image" [Theverapperuma 0096].
Regarding claim 18, Rybakov and Theverapperuma teach the system of claim 17. Rybakov further teaches wherein the processing system is further configured to: obtain, in addition to the two-dimensional input image, one or more further two-dimensional input images of the scene, the two-dimensional input image and the one or more further two-dimensional input images each corresponding to a different respective camera view of the scene ([col. 3 ln. 65 - col. 4 ln. 14] To build a database of object images, with multiple objects per image, a number of different images of an object may be taken from different viewpoints. From those images, feature points may be extracted and pyramid images constructed. Multiple images from different points of view of each particular object may be taken and linked within the database (for example within a tree structure described below). The multiple images may correspond to different viewpoints of the object sufficient to identify the object from any later angle that may be included in a user's query image. For example, a shoe may look very different from a bottom view than from a top view than from a side view. For certain objects, this number of different image angles may be 6 (top, bottom, left side, right side, front, back), for other objects this may be more or less depending on various factors, including how many images should be taken to ensure the object may be recognized in an incoming query image);
generate, using the feature generation model, a respective feature map of respective feature vectors for each of the one or more further two-dimensional input images; and further compare the further feature vectors included in the further feature maps with the reference feature vectors; wherein said identifying the at least one point of interest in the two-dimensional input image is also based on the further comparison ([col. 2 ln. 21-33] As shown, a server 104 that performs image recognition may, during a training period, receive multiple images showing different angles of an object (120). The server 104 may build an image database using the multiple angles (122). The server 104 may also perform pre-processing on these images to identify feature points and keywords (as described below) to potentially match images in the database with incoming images. The database images may be associated with certain objects or object IDs so images of incoming objects may be recognized. For example, different images 160 of shoes may be taken from different angles, processed, and stored in an image database on the server 104 or elsewhere).
Rybakov does not teach using a machine learning based feature generation model.
Theverapperuma teaches using a machine learning based feature generation model ([0006] In certain embodiments, at least some of the modules in the surface identification subsystem are implemented using a machine learning model (e.g., a convolutional neural network or CNN). [0007] The method further involves extracting, by the controller system, a set of features from the at least one camera image. The extracting comprises inputting the at least one camera image to a neural network trained to infer values of the set of features from image data).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the system of Rybakov with the teachings of Theverapperuma to use machine learning for "generating an augmented image or a segmented image comprises inputting values of a set of features to a neural network trained using images of surface deformations associated with drivable surfaces" [Theverapperuma 0009].
Regarding claim 19, Rybakov and Theverapperuma teach the system of claim 14. Rybakov further teaches wherein the processing system is further configured to cause a physical action to be performed in respect of a target object based on the comparing ([col. 3 ln. 17-22] If the results of the second comparison are above a threshold, the represented object may be determined to be a match. The server 104 may then return results to the user based on that object identifier. The results may include identification information for the object, the option to purchase the object, etc.).
Regarding claim 20, Rybakov teaches a computer readable medium storing a set of non-transitory executable software instructions that, when executed by one or more processing devices, configure the one or more processing devices to perform a method of identifying points of interest in an image ([col. 14 ln. 45-51] Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture such as a memory device or non-transitory computer readable storage medium. The computer readable storage medium may be readable by a computer and may comprise instructions for causing a computer or other device to perform processes described in the present disclosure. [col. 2 ln. 61-62] The server 104 identifies features points of the query image (134)), comprising:
obtaining a two-dimensional input image of a scene ([col. 2 ln. 34-41] At runtime, the server 104 may receive a communication from a user device, such as smartphone 102 over a network 106 or through another communication method. The communication may include a request to identify a specific object seen by a camera(s) of the device 102. The device 102 may send video/image data from its camera(s) to the server 104. The image data may be sent as a stream of images taken from the camera(s));
generating, using a feature generation model, a feature map for the input image ([col. 2 ln. 9-15] For example, several images of a particular object may be taken from multiple points of view (for example, left view, right view, top view, bottom view, etc.). These images from the multiple points of view may then be used to train a planar recognizer. [col. 10 ln. 20-25] The system may then determine feature points, feature vectors, a keyword, or other representations (such as texture, color, etc.) of the contents of the query image (406). The keyword may be determined based on a mathematical function related to the feature points and/or feature vector of the query image);
comparing the plurality of feature vectors included in the feature map with reference feature vectors generated by the feature generation model based on reference points within a reference image; and based on the comparing, identifying at least one point of interest ([col. 5 ln. 40-57] A server 104 receives multiple images, each image showing a different angle of an object (302). The server may associate images showing angles of a same object together (304) either as part of a tree (discussed below) or separately. An object identifier (which may be a unique alpha-numeric code) corresponding to a displayed image may be associated with an image showing the object. The database images may include only one object per image (which assists in image mapping) or, in certain instances, may include multiple objects in one image. In that case, the particular image may be associated with the multiple object identifiers corresponding to the displayed images. The server 104 may determine feature points and/or pyramid images (306) for the database images. These feature points and pyramid images are associated with their corresponding images. The server may also determine feature vectors, keywords, and/or other representations or features of the contents of the images (308). [col. 6 ln. 38-46] The image matching algorithm of the present invention finds a match of query image from among multiple database images. As explained below, the image matching algorithm may operate by representing images in terms of feature points, orientations and feature vectors. After a representation of the query image and database images has been created, the feature points, orientations and feature vectors of the images are used to determine a match between the images. [col. 7 ln. 27-30] The pair of putatively matching images can be determined to be a match when a sufficient number of feature points match the corresponding feature points of the other image both visually and geometrically).
Rybakov does not explicitly teach the input image comprising a plurality of pixels; using a machine learning based feature generation model, the feature map comprising a plurality of feature vectors each associated with a respective one of the plurality of pixels.
Theverapperuma, in the same field of endeavor of feature generation, teaches the input image comprising a plurality of pixels ([0007] In certain embodiments, a method involves receiving, by a controller system of an autonomous vehicle, sensor data from a plurality of sensors. The sensor data comprises at least one camera image of a physical environment and a first three-dimensional (3D) representation of the physical environment. The method further involves extracting, by the controller system, a set of features from the at least one camera image. [0095] (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404);
using a machine learning based feature generation model ([0006] In certain embodiments, at least some of the modules in the surface identification subsystem are implemented using a machine learning model (e.g., a convolutional neural network or CNN). [0007] The method further involves extracting, by the controller system, a set of features from the at least one camera image. The extracting comprises inputting the at least one camera image to a neural network trained to infer values of the set of features from image data),
the feature map comprising a plurality of feature vectors each associated with a respective one of the plurality of pixels ([0095] the inclusion of the feature extractor 422 in the embodiment of FIG. 4 increases computational efficiency by reducing the dimensionality of the input image space (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404). [0094] Feature extractor 422 operates as a backbone network for the extraction of image features. In particular, the feature extractor 422 is configured to extract values for a set of features represented in the data from the cameras 402, 404. The feature extractor 422 can be implemented as a neural network that has been trained (e.g., through supervised learning and backpropagation) to generate a vector or multi-dimensional tensor for input to each of the modules 424, 426, and 428. The vector or multi-dimensional tensor is an abstract representation of a 2D image that combines information from the individual camera images).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the medium of Rybakov with the teachings of Theverapperuma to use machine learning for "generating an augmented image or a segmented image comprises inputting values of a set of features to a neural network trained using images of surface deformations associated with drivable surfaces" [Theverapperuma 0009] and for the feature map to comprise a plurality of feature vectors associated with a plurality of pixels because "the inclusion of the feature extractor 422 in the embodiment of FIG. 4 increases computational efficiency by reducing the dimensionality of the input image space (e.g., an N-dimensional space corresponding to N number of pixels in a given image captured by camera 402 or camera 404)" [Theverapperuma 0095] and "the segmented image can be an RGB formatted 2D image in which each pixel has been assigned a class of “road” or a class of “non-road”. Thus, the segmented image can represent the result of performing classification on the extracted features, possibly classification that divides regions in the segmented image into one of two types of surfaces: potentially drivable and non-drivable" [Theverapperuma 0098].
Regarding claim 21, Rybakov and Theverapperuma teach the method of claim 1. Rybakov further teaches receiving an identification of a given point within the two-dimensional input image, said comparing comprising comparing a given one of the plurality of feature vectors associated with the given point within the two-dimensional input image with the reference feature vectors, and said identifying comprising when the given one of the plurality of feature vectors matches one of the reference feature vectors, identifying the given point as being a point of interest ([col. 2 ln. 61-66] The server 104 identifies features points of the query image (134). The features may include feature points of the image (as described below), textures in the image, colors in the image, etc. The server 104 then identifies one or more putative matching database image(s) for the query image (136). To test the putative matches, the server 104 compares a representation of the first image to a representation of the second image. For example, the server 104 compares features between the putative matches and the query image (138). [col. 6 ln. 38-46] The image matching algorithm of the present invention finds a match of query image from among multiple database images. As explained below, the image matching algorithm may operate by representing images in terms of feature points, orientations and feature vectors. After a representation of the query image and database images has been created, the feature points, orientations and feature vectors of the images are used to determine a match between the images. [col. 7 ln. 27-30] The pair of putatively matching images can be determined to be a match when a sufficient number of feature points match the corresponding feature points of the other image both visually and geometrically).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Rybakov in view of Theverapperuma and Chen (US20220284691A1).
Regarding claim 7, Rybakov and Theverapperuma teach the method of claim 6. Chen, in the same field of endeavor of detecting a target object using feature information, teaches receiving, through a user interface, user inputs selecting locations on the reference image as the reference points, as part of a configuration phase ([0139] For example, as illustrated in FIG. 8A, a first monitoring image 800 may include a first object 801 and a first object 802. The controller 103 may determine a reference point A in the first monitoring image 800, determine position information associated with the reference point A, and designate the position information associated with the reference point A as a target monitoring position of the second capture device 102. Then the controller 103 may direct the second capture device 102 to acquire a second monitoring image 850 based on the target monitoring position. As illustrated in FIG. 8B, the second monitoring image 850 may include a second object 803 and a second object 804 corresponding to the first object 801 and the first object 802, respectively. [0094] In some embodiments, the reference point may be determined manually, automatically, or semi-automatically. For example, the first monitoring image may be transmitted to a terminal device (e.g., the terminal device 140) for display and a user may annotate the reference point in the first monitoring image via a user interface implemented on the terminal device. [0049] The capture device 110 may be configured to acquire (or capture) images or videos associated with one or more objects. In some embodiments, the images or videos may be two-dimensional (2D)).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Chen for the user to select reference points so that "the object detection model may be generated by training a preliminary model using at least one training sample. Each of the at least one training sample may include a sample monitoring image and object(s) annotated in the sample monitoring image" [Chen 0134].
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Rybakov in view of Theverapperuma and Lee (US20220061816A1).
Regarding claim 10, Rybakov and Theverapperuma teach the method of claim 3. Rybakov further teaches the training is performed without depth information ([col. 2 ln. 9-15] For example, several images of a particular object may be taken from multiple points of view (for example, left view, right view, top view, bottom view, etc.). These images from the multiple points of view may then be used to train a planar recognizer).
Rybakov does not teach the training includes generating depth information for each of the corresponding image sets using a machine learning based depth generating model and the generated depth information is used together with the set of training images and the image label set to train the feature generation model.
Theverapperuma teaches the training includes generating depth information for each of the corresponding image sets using a machine learning based depth generating model and the generated depth information is used together with the set of training images and the image label set to train the feature generation model ([0096] Depth estimation module 424 is configured to generate a depth image, e.g., an RGB-D (red, green, blue, and depth) image, based on the features extracted by the feature extractor 422. Each pixel in the depth image is assigned a depth value indicating the depth at the location represented by the pixel. [0097] Training of the depth estimation module 424 may involve providing the depth estimation module 424 with training images depicting surfaces and/or objects, at different distances away from the camera that captured the training image. The depth images generated as a result of processing the training images can then be compared to corresponding ground truth depth information (e.g., the correct depth value for each pixel in a training image) to adjust the CNN by changing weights and/or bias values for one or more layers of the CNN such that a loss function is minimized).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Theverapperuma to generate depth information and use it in training the model because "If the camera data, as represented in the features extracted by the feature extractor 422, captures a drivable surface, then the depth values for the drivable surface (e.g., the depth at various points along the drivable surface) will have been determined by virtue of estimating the depth for each pixel in the depth image" [Theverapperuma 0096].
Rybakov does not teach presenting, using a user interface, selectable training mode options including a 3D scene learning mode and a 2D scene learning mode; and receiving a user input selecting one of the training mode options, wherein: (i) the user input selects the 3D scene learning mode, the and (ii) the user input selects the 2D scene learning mode.
Lee, in the same field of endeavor of machine learning image analysis teaches presenting, using a user interface, selectable training mode options including a 3D scene learning mode and a 2D scene learning mode; and receiving a user input selecting one of the training mode options, wherein: (i) the user input selects the 3D scene learning mode, the and (ii) the user input selects the 2D scene learning mode ([0103] In order to improve resolution of the 2D image with scan data obtained from a 1D transducer, such as higher resolution in an elevation direction, and with reduced complexity of scanning and without bulkiness of additional mechanical controls, a trained resolution mapping algorithm may be deployed. [0102] Depending on rendering mode (2D or 3D, which may be based on user selection), 2D images with higher elevation resolution or 3D images may be reconstructed from the plurality of 2D cross-sectional images).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Lee to use user input for selection of a 2D or 3D learning so that a "first ultrasound sound image is fed into the trained resolution mapping neural network algorithm to obtain a second ultrasound image with a higher resolution profile" [Lee 0103].
Claims 22-23 are rejected under 35 U.S.C. 103 as being unpatentable over Rybakov in view of Theverapperuma and Ajika (JP2012123608A).
Regarding claim 22, Rybakov and Theverapperuma teach the method of claim 1. Rybakov does not teach wherein the plurality of feature vectors comprises fused feature vectors each obtained based a first feature vector associated with the respective one of the plurality of pixels and at least one second feature vector associated with at least one neighboring pixel within the two-dimensional input image. Ajika, in the same field of endeavor of feature vector comparison, teaches wherein the plurality of feature vectors comprises fused feature vectors each obtained based a first feature vector associated with the respective one of the plurality of pixels and at least one second feature vector associated with at least one neighboring pixel within the two-dimensional input image ([pg. 3-4] Each behavior vector is obtained for the pixel coordinates. Note that the vector normally has two elements of magnitude and direction, but the behavior vector obtained in this embodiment has only the direction element. The method of obtaining the behavior vector is not particularly limited, but may be obtained by connecting pixel coordinates adjacent to each other with a straight line. [pg. 4 para. 8] FIG. 5A shows the finger position 1 to 10 and the behavior vectors of the respective finger positions when the user draws the trajectory of the letter “L” with his fingertips in the free space. Here, the behavior vectors at the finger positions 1 and 2 are “pattern 3”, the behavior vectors at the finger positions 3 to 6 are “stationary”, and the behavior vectors at the finger positions 7 to 9 are “pattern 1”. And the pixel coordinate 10 is “still state”. In the actual behavior vector data structure, the behavior vectors at the positions of the respective fingers are arranged in parallel in the order of positions 1 to 10 with one data block. First, when data blocks indicating the same behavior vector are adjacent to each other, they are combined into a single data block. That is, as shown in FIG. 5B, the finger positions 1 and 2, 3 to 6, and 7 to 9 are each set as a single data block).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Ajika for the feature vectors to comprise fused feature vectors because "'feature points' are a group of data blocks that are adjacent and exhibit the same behavior vector, and data blocks that are identified as 'still state'…Therefore, when the amount of movement between adjacent pixel coordinates is below a certain level, it is considered to be in a 'still state'" [pg. 4 para. 7 and 4].
Regarding claim 23, Rybakov, Theverapperuma, and Ajika teach the method of claim 22. Rybakov does not teach wherein the reference feature vectors comprise reference fused feature vectors each obtained based on a third feature vector associated with a respective one of the reference points and at least one fourth feature vector associated with at least one neighboring point within the reference image. Ajika teaches wherein the reference feature vectors comprise reference fused feature vectors each obtained based on a third feature vector associated with a respective one of the reference points and at least one fourth feature vector associated with at least one neighboring point within the reference image ([pg. 5 para. 4] The pattern matching unit 60 has a function of specifying a trajectory drawn by the user's fingertip based on the data received from the feature point analysis unit 50, that is, a fingertip gesture. Specifically, it is searched whether there is any data that matches the data received from the feature point analysis unit 50 among a plurality of gesture pattern data created in advance and recorded in the data storage unit 90. If there is a matching gesture pattern, it is determined that the gesture pattern is a gesture made by the user. [pg. 5 para. 9] The pattern matching unit 60 first searches for the end point with reference to the data received from the feature point analysis unit 50. Referring to Table 1, it can be seen that an end point exists in the data string I. Therefore, paying attention to the data string H immediately before the end point, a gesture pattern in which the behavior vector of the last data string (block) is “pattern 8” is searched. Next, a gesture pattern whose second behavior vector is “pattern 7” is searched. This search process is repeated to recognize that the trajectory of the data strings B to H corresponds to the gesture “U”).
Therefore, it would have been obvious to a person of ordinary skill in the art at the time that the invention was made to modify the method of Rybakov with the teachings of Ajika for the reference feature vectors to comprise reference fused feature vectors because "'feature points' are a group of data blocks that are adjacent and exhibit the same behavior vector, and data blocks that are identified as 'still state'…"Therefore, when the amount of movement between adjacent pixel coordinates is below a certain level, it is considered to be in a 'still state'" [pg. 4 para. 7 and 4] so that "it is searched whether there is any data that matches the data received from the feature point analysis unit 50 among a plurality of gesture pattern data created in advance and recorded in the data storage unit 90" [pg. 3 para. 4].
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights