DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8-11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 8 recites “obtaining a current image frame, wherein the current image frame is an image obtained after the to-be-processed image is obtained; and determining, …a second pose of the target in the current image frame.”, its unclear how a target can have two poses in the same frame. This makes the claim indefinite.
Claim 9-11 depend on claim 8 and therefore, are rejected as well.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-10, 12-14 and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Grabner et al (US Pub. 2019/0147221).
With respect to claim 1, Grabner discloses A method comprising:
receiving, [from a first electronic device], a localization request comprising a to-be-processed image, (see paragraph 0093, wherein …For the real world query image (input image 202 “a to-be-processed image”), an image descriptor (e.g., descriptor 226) is extracted…);
recognizing a target in the to-be-processed image, (see paragraph 0005, wherein …the techniques and systems described herein can detect one or more target objects in an input image…);
searching a pose estimation model library for a target pose estimation model corresponding to the target, wherein the pose estimation model library comprises pose estimation models corresponding to objects, (see paragraph 0060, wherein …The pose estimation system 108 [this is read as a pose estimation model library] can process each of the input images 102 to detect one or more target objects in the input images 102. The input image 202 shown in FIG. 2A is an example of an image with an airplane object…);
obtaining, based on the to-be-processed image and the target pose estimation model, a pose of the target, (se figure 2A, the Predicted 3D pose numerical 232); and
sending, [to the first electronic device], the pose, (see figure 2A, the numerical 232 predicted 3D pose is sent for the rendering), as claimed.
However, Grabner fails to explicitly disclose receiving, from a first electronic device, a localization request comprising a to-be-processed image; and sending, to the first electronic device, the pose, as claimed.
But, Grabner in paragraph 0058 teaches “…The video source can include an image capture device (e.g., a camera, a camera phone, a video camera, a tablet device with a built-in camera…” a camera phone is read as “a first electronic device” and paragraph 0137 teaches “…, the camera device can include communications transceiver and/or a video codec…” communications is read as “sending to the first electronic device”, as claimed.
Therefore, it would be obvious to one ordinary skilled in the art at the effective date of invention to utilize the teaching of the Grabner for use of a camera phone or smart phone in order to capture an image for the query in the system to yield the predictable results for searching a pose model for an object in an image.
With respect to claim 3, Grabner further discloses wherein recognizing the target comprises: extracting a first feature vector from the to-be-processed image; searching a feature vector library for a second feature vector having a highest similarity to the first feature vector, wherein the feature vector library comprises feature vectors corresponding to identifiers of the objects; and determining, based on a first identifier corresponding to the second feature vector, the target, (see paragraph 0060, wherein …an object can be detected by extracting features of the object from the image, and using a learning algorithm to recognize an object category based on the extracted features…; see paragprah 0081, wherein …a CNN architecture which jointly predicts the 2D image locations of the projections of the eight 3D bounding box corners (16 values) can be used as well as the 3D bounding box spatial dimensions (3 values). The architecture shown in FIG. 2B supports joint optimization for 3D object pose estimation and 3D model retrieval…; and paragraph 0096, wherein …The similarity loss can then be used to select the retrieved 3D model for use in representing a target object. For example, the candidate 3D model (represented by a 3D mesh) that has the lease similarity loss, with respect to a target object's descriptor, can be selected as the 3D model that will be retrieved for the target object), as claimed.
With respect to claim 4, Grabner further discloses wherein obtaining the pose comprises: inputting the to-be-processed image into a key point recognition model of the target pose estimation model to obtain two-dimensional coordinates of at least four key points corresponding to the target; determining, based on a three-dimensional model corresponding to the target three-dimensional coordinates of the at least four key points; and obtaining, based on the two-dimensional coordinates the three-dimensional coordinates and a perspective-n-point algorithm of the target pose estimation model, the pose, (see paragraph 0079-0081, and the keypoints are read as the coordinates of the bounding box), as claimed.
With respect to claim 5, Grabner further discloses wherein before sending the pose the method further comprises: reprojecting, based on the pose a three-dimensional model corresponding to the target onto the to-be-processed image to obtain a rendered image; performing M optimization processes, wherein M is a positive integer, and wherein each of the M optimization processes comprises: calculating an optimized pose based on the rendered image and the to- be-processed image; and calculating a pose error based on the to-be-processed image and the optimized pose; updating the optimized pose to the pose when the pose error is less than a preset error value; and when the pose error is not less than the preset error value; updating, based on the optimized pose, the rendered image; and performing the M optimization processes, (see paragraph 0160, wherein …In some examples, the refiner method includes training a regressor (e.g., a convolutional neural network) to update the 2D projections of the bounding box of an object. The regressor used to update the 2D projections may include a different regressor than the regressor 404. The 2D projections can be updated by comparing an input image to a rendering of the object for an initial pose estimate), as claimed.
With respect to claim 6, Grabner further discloses wherein before receiving the localization request the method further comprises: sending to the first electronic device a three-dimensional model corresponding to the target to track the target based on the pose, (see paragraph 0075, wherein …referring again to FIG. 3D, the 3D airplane model 312 is selected to represent the airplane object 302, and can replace the airplane object 302 in the image 300. A viewer can then manipulate the location, orientation, geometry, and/or other characteristic of the 3D airplane model 312. In other examples, the 3D mesh of the selected 3D model can be used for 3D scene understanding, object grasping, object tracking, scene navigation…), as claimed.
Claim 7 is rejected for the same reasons as set forth in the rejection for claim 1, because claim 7 is claiming subject matter of similar scope as claimed in claim 1. Furthermore, Grabner in paragraph 0137 discloses “…In some examples, a camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives the captured video data…” in this the computing device is read as “server” as claimed in claim 7.
With respect to claim 8 as best understood, Grabner further discloses wherein after receiving the method further comprises: obtaining a current image frame, wherein the current image frame is an image obtained after the to-be-processed image is obtained; and determining, based on the current image frame, a three-dimensional model corresponding to the target, and the first pose, a second pose of the target in the current image frame, (see paragraph 0060, wherein …an object can be detected by extracting features of the object from the image, and using a learning algorithm to recognize an object category based on the extracted features…; see paragraph 0081, wherein …a CNN architecture which jointly predicts the 2D image locations of the projections of the eight 3D bounding box corners (16 values) can be used as well as the 3D bounding box spatial dimensions (3 values). The architecture shown in FIG. 2B supports joint optimization for 3D object pose estimation and 3D model retrieval…; and paragraph 0096, wherein …The similarity loss can then be used to select the retrieved 3D model for use in representing a target object. For example, the candidate 3D model (represented by a 3D mesh) that has the lease similarity loss, with respect to a target object's descriptor, can be selected as the 3D model that will be retrieved for the target object), as claimed.
With respect to claim 9 as best understood, Grabner further discloses wherein before determining the second pose the method comprises: receiving, from the server, an identifier of the target; and obtaining, based on the identifier the three-dimensional model that is stored in a storage and that is from the server. , (see paragraph 0060, wherein …an object can be detected by extracting features of the object from the image, and using a learning algorithm to recognize an object category based on the extracted features…; see paragraph 0081, wherein …a CNN architecture which jointly predicts the 2D image locations of the projections of the eight 3D bounding box corners (16 values) can be used as well as the 3D bounding box spatial dimensions (3 values). The architecture shown in FIG. 2B supports joint optimization for 3D object pose estimation and 3D model retrieval…; and paragraph 0096, wherein …The similarity loss can then be used to select the retrieved 3D model for use in representing a target object. For example, the candidate 3D model (represented by a 3D mesh) that has the lease similarity loss, with respect to a target object's descriptor, can be selected as the 3D model that will be retrieved for the target object), as claimed.
With respect to claim 10 as best understood, Grabner further discloses wherein determining the second pose comprises: performing N optimization processes, wherein N is a positive integer, and wherein each of the N optimization process comprises: calculating a pose correction amount based on an energy function, the three-dimensional model, and the second pose; updating the second pose based on the pose correction amount to obtain an updated second pose; and calculating an energy function value based on the energy function and the updated second pose; outputting the updated second pose as the second pose when the energy function value meets a preset condition; and performing the N optimization processes when the energy function value does not meet the preset condition, (see paragraph 0079-0081, for estimating the pose to the optimize value using the algorithm and lose functions), as claimed.
With respect to claim 12, Grabner further discloses wherein before sending the localization request the method comprises: displaying a user interface comprising a register control; and sending, to the server, a three-dimensional model corresponding to the target when detecting a first user operation for the register control, (see figure 2A, numerical 216 for display model for the user on the cell phone), as claimed.
With respect to claim 13, Grabner further discloses further comprising: obtaining, in response to receiving an external input, the three-dimensional model or generating, based on a second user operation the three-dimensional model, (see figure 2A, numerical 26 for display on the cell phone of the user the display model), as claimed.
Claim 14 is rejected for the same reasons as set forth in the rejection for claim 1, because claim 14 is claiming subject matter of similar scope as claimed in claim 1. Furthermore, Grabner in paragraph 0137 discloses “…In some examples, a camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives the captured video data…” in this the computing device is read as “server” as claimed in claim 14.
Claims 16-19 is rejected for the same reasons as set forth in the rejection for claims 3-6, because claims 16-19 is claiming subject matter of similar scope as claimed in claims 3-6 respectively.
Claim 20 is rejected for the same reasons as set forth in the rejection for claim 7, because claim 20 is claiming subject matter of similar scope as claimed in claim 7.
Claims 2 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Grabner in view of SyDPose: Object Detection and Pose Estimation in Cluttered Real-World Depth Images Trained using only Synthetic Data, by Thalhammer et al (IDS document).
With respect to claim 2, Grabner discloses all the limitations as claimed and as rejected in claim 1 above. Furthermore, Grabner disclose wherein before searching the pose estimation model library for the target pose estimation model the method further comprises: receiving, from a second electronic device, a three-dimensional model corresponding to the target; rendering the three-dimensional model to generate training images, (see figure 2A numerical s232 and 216 for receiving and rendering the three dimensional model), as claimed.
However, Grabner fails to explicitly disclose training, based on the training images, an initial pose estimation model to obtain the target pose estimation model, as claimed.
Thalhammer teaches training, based on the training images, an initial pose estimation model to obtain the target pose estimation model, (see page 108, section 4 right hand column, wherein …The estimated control points are reprojected into 3D space and the object’s pose is simultaneously estimated using PnP. For our purpose we use the iterative RANSAC based algorithm…; see page 109-110 section 5 for training data creation for training the model), as claimed.
It would be obvious to one ordinary skilled in the art at the effective date of invention to combine the two references as they are analogous because they are solving similar problem of object detection and pose estimation using image analysis. Teaching of Thalhammer to train the model to estimate the pose can be incorporated into Grabner system as suggested in paragraph 0051, “…a neural network based approach…””, for suggestion, and modifying the system yields a trained model for pose estimation (see Thalhammer Abstract), for motivation.
Claim 15 is rejected for the same reasons as set forth in the rejection for claim 2, because claim 15 is claiming subject matter of similar scope as claimed in claim 2.
Allowable Subject Matter
Claim 11 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIKKRAM BALI whose telephone number is (571)272-7415. The examiner can normally be reached Monday-Friday 7:00AM-3:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at 571-272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VIKKRAM BALI/Primary Examiner, Art Unit 2663
Prosecution Insights