NON-FINAL REJECTION, FIRST DETAILED ACTION
Status of Prosecution
The present application 18/692,372, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The application was filed in the Office on March 15, 2024 which is a national stage §371 of PCT/SG2021/050569 with a filing date of Sept. 17, 2021.
Claims 1-18 and 20 are pending and are all rejected in this rejection. Claim 19 is cancelled by amendment. Claims 1 is the independent claim.
Status of Claims
Claim 19 is cancelled by amendment.
Claims 1-18 and 20 are rejected under 35 USC § 103 as being unpatentable over non-patent literature Puang et al., “KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to Real Transfer for Robotics Manipulation,” (“KOVIS”), published on July 28, 2020, in view of non-patent literature Kim et al. (“Kim”), “Traversability Classification for UGV Navigation:A Comparison of Patch and Superpixel Representations,” published 2007 in further view of Chakavarty et al. (“Chakavarty”), United States Patent Application Publication 2020/0041276, published on Feb. 6, 2020.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-18 and 20 are rejected under 35 USC § 103 as being unpatentable over non-patent literature Puang et al., “KOVIS: Keypoint-based Visual Servoing with Zero-Shot Sim-to Real Transfer for Robotics Manipulation,” (“KOVIS”), published on July 28, 2020, in view of non-patent literature Kim et al. (“Kim”), “Traversability Classification for UGV Navigation:A Comparison of Patch and Superpixel Representations,” published 2007 in further view of Chakavarty et al. (“Chakavarty”), United States Patent Application Publication 2020/0041276, published on Feb. 6, 2020.
As to Claim 1, KOVIS teaches: A method for training a robot device controller comprising:
training a neural network comprising an encoder network, a decoder network and a policy model, such that, for each of a plurality of digital training input images (KOVIS: Fig. 2, Sec. III, the framework for KOVIS consists of two major modules: an autoencoder (which has an encoder and a decoder network); and feed-forward network for learning visual servoing (VS) (a policy model) that controls the motion based on the visual image input from the camera);
the encoder network encodes the digital training input image to a feature in a latent space (KOVIS: Sec. III, the latent keypoint representation of objects in a VS scene is encoded with a keypointer and optimized in equation 1; Sec. I, traditionally VS requires extracting and tracking of visual features);
the decoder network determines for each of a plurality of areas shown in the digital training input image information about the distance between the viewpoint of the digital training input image and the area (KOVIS: Sec. III.C, Fig. 2(a), depth buffer (distance) is determined per the decoder),;
and the policy model determines, from the feature, control information for controlling movement of a robot device (KOVIS: Fig. 2, motion commands are predicted in terms of direction and speed per the keypoint-based VS, which uses the features extracted from the keypoint extraction);
wherein at least the policy model is trained in a supervised manner using control information ground truth data of the digital training input images (KOVIS: Sec. 3, “Both networks are trained end-to-end using ground-truth data gathered from simulations).
PNG
media_image1.png
288
718
media_image1.png
Greyscale
KOVIS may not explicitly teach: the decoder network determines, from the feature, for each of a plurality of areas shown in the digital training input image, whether the area is traversable and information about the distance between the viewpoint of the digital training input image.
Kim teaches in general concepts related to robot navigation in complex outdoor terrain using accurate transversality classification (Kim: Abstract). Specifically, Kim teaches that visual features are used for traversability classification (Kim: Sec. IV, Table 1).A traversability classification algorithm which incrementally updatesand utilizes superpixels as the visual primitives (Kim: Secs. I, V).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the KOVIS disclosures and teachings by determining traversability based on classification per features as taught and suggested by Kim. Such a person would have been motivated to do so with a reasonable expectation of success to allow for the computing at a sufficient frame rate (Kim: Abstract).
KOVIS and Kim may not explicitly teach: the policy model determines, from the feature, for each of a plurality of areas shown in the digital training input image, whether the area is traversable and information about the distance between the viewpoint of the digital training input image and the area in the form of relative depth of the areas.
KOVIS does discuss a depth buffer, but does not discuss in detail how it is related to the traversability considerations. Chakavarty teaches in general concepts related to simultaneous localization and mapping of a robot in an environment utilizing a variational autoencoder generative adversarial network (Chakavarty: Abstract). Simultaneous Localization and Mapping (SLAM) is used to recover depth (Chakavarty: par. 0015, “SLAM may also be implemented using monocular vision, but the depth recovered through triangulation of landmarks from a moving camera over time is up to scale only, such that relative depths of objects in the scene are recovered without absolute depth measurements.”).
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have modified the KOVIS-Kim combination by determining traversability with relative depth considered as taught and suggested by Chakavarty. Such a person would have been motivated to do so with a reasonable expectation of success to allow for depth recovery without absolute depth measurement, reducing computing costs (Chakavarty: par. 0015).
As to Claim 2, KOVIS, Kim and Chakabarty teach the elements of claim 1.
KOVIS further teaches: wherein training the encoder network and the decoder network comprises training an autoencoder comprising the encoder network and the decoder network (KOVIS: Fig. 2, Sec. III, the framework for KOVIS includes an autoencoder).
As to Claim 3, KOVIS, Kim and Chakabarty teach the elements of claim 1.
KOVIS further teaches: comprising training the encoder network jointly with the decoder network (KOVIS: Fig. 2, Sec. III, both networks are trained end-to-end together).
As to Claim 4, KOVIS, Kim and Chakabarty teach the elements of claim 1.
KOVIS further teaches: comprising training the encoder network jointly with the decoder network and the policy model (KOVIS: Fig. 2, Sec. III, both networks are trained end-to-end together).
As to Claim 5, KOVIS, Kim and Chakabarty teach the elements of claim 1.
KOVIS further teaches: wherein the decoder network comprises a semantic decoder and a depth decoder and wherein the neural network is trained such that, for each digital training input image (KOVIS: Sec. III, “In staying agnostic towards object texture, the input image x is transformed into grayscale domain, and the decoder’s outputs x’ := (xd, xs) reconstructs the
depth buffer xd and semantic segmentation xd only using the keypoint”),
Kim further teaches: the semantic decoder determines, from the feature, for each of a plurality of areas shown in the digital training input image, whether the area is traversable (Kim: Sec. VI.A.(1) “the pixels in the images were manually labeled as either traversable or nontraversable.”); and
Chakavarty further teaches: the depth decoder determines, from the one or more features, for each of a plurality of areas shown in the digital training input image, information about the distance between the viewpoint of the digital training input image and the area Chakavarty: par. 0015, “SLAM may also be implemented using monocular vision, but the depth recovered through triangulation of landmarks from a moving camera over time is up to scale only, such that relative depths of objects in the scene are recovered without absolute depth measurements.”).
As to Claim 6, KOVIS, Kim and Chakabarty teach the elements of claim 5.
KOVIS further teaches: wherein the semantic decoder is trained in a supervised manner (KOVIS: II. C, self-supervised learning).
As to Claim 7, KOVIS, Kim and Chakabarty teach the elements of claim 5.
KOVIS further teaches: wherein the depth decoder is trained in a supervised manner (KOVIS: II. C, self-supervised learning) or wherein the depth decoder is trained in an unsupervised manner.
As to Claim 8, KOVIS, Kim and Chakabarty teach the elements of claim 1.
KOVIS further teaches: wherein one or more of the encoder network, the decoder network and the policy model are convolutional neural networks (KOVIS: Sec. III.A, “We first learn the latent keypoint representation of objects in a VS scene with an CNN-based autoencoder”).
As to Claim 9, KOVIS, Kim and Chakabarty teach the elements of claim 1.
KOVIS further teaches: wherein the control information comprises control information for each of a plurality of robot device movement commands (KOVIS: Fig. 2, motion commands are predicted in terms of direction and speed per the keypoint-based VS).
As to Claim 10, KOVIS, Kim and Chakabarty teach the elements of claim 1.
KOVIS further teaches: wherein the neural network is trained such that the policy model determines the control information from features to which the encoder has encoded a plurality of training input images (KOVIS: Fig. 2, motion commands are predicted in terms of direction and speed per the keypoint-based VS, which uses the features extracted from the keypoint extraction).
As to Claim 11, KOVIS, Kim and Chakabarty teach: A method for controlling a robot device comprising:
training a robot device controller according to claim 1 (KOVIS: Sec. III.C training end-to-end);
obtaining one or more digital images showing surroundings of the robot device (KOVIS: Sec. III, autoencoder for learning a keypoint representation from input images);
encoding the one or more digital images to one or more features using the encoder network (KOVIS: Fig. 2, encoder);
supplying the one or more features to the policy model(KOVIS: Fig. 2, encoder, features to keypoint-based VS); and
controlling the robot according to control information output of the policy model in response to the one or more features (KOVIS: Fig. 2, motion commands are predicted).
As to Claim 12, KOVIS, Kim and Chakabarty teach the elements of claim 11.
KOVIS further teaches: comprising receiving the one or more digital images from one or more cameras of the robotic device (KOVIS: Abstract, an eye-in-hand stereo camera system).
As to Claim 13, KOVIS, Kim and Chakabarty teach the elements of claim 11.
KOVIS further teaches: wherein the control information comprises control information for each of a plurality of robot device movement commands and wherein the method comprises receiving an indication of a robot device movement command and controlling the robot according to the control information for the indicated robot device movement command.
As to Claim 14, KOVIS, Kim and Chakabarty teach the elements of claim 11.
KOVIS further teaches: wherein the neural network is trained such that the policy model determines the control information from features to which the encoder has encoded a plurality of training input images and wherein the method comprises obtaining a plurality of digital images showing surroundings of the robot device (KOVIS: Sec. III.C training end-to-end);
encoding the plurality of digital images to a plurality of features using the encoder network (KOVIS: equation 1, optimized self-supervision autoencoder and decoder);
supplying the plurality of features to the policy model (KOVIS: Sec. III.B); and
controlling the robot according to control information output of the policy model in response to the plurality of features (KOVIS: Sec. III.B, the predicted motion consists of direction and as well normalized speed).
As to Claim 15, KOVIS, Kim and Chakabarty teach the elements of claim 14.
KOVIS further teaches: wherein the plurality of digital images comprises images received from different cameras (KOVIS: Abstract, stereo camera).
As to Claim 16, KOVIS, Kim and Chakabarty teach the elements of claim 14.
KOVIS further teaches: wherein the plurality of digital images comprises images taken from different viewpoints (KOVIS: Abstract, stereo camera).
As to Claim 17, KOVIS, Kim and Chakabarty teach the elements of claim 14.
Kim further teaches: wherein the plurality of digital images comprises images taken at different times (Kim: the image frames are taken at different times, per the “sufficient frame rate” discussion).
As to Claim 18, is rejected for similar reasons as claim 1.
As to Claim 20, is rejected for similar reasons as claim 1. Chakavarty discloses computer readable medium (Chakavarty: par. 0094).
Conclusion
Additional prior art made of the record deemed relevant though not relied upon:
Lin et al., US PG Pub 2020/0241574 (7/30/2020) (describing generalized robot approach control techniques.
Mishra et al. ,SG 10201605032T A (1/30/2018 (robot policies navigation).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAMES T TSAI whose telephone number is (571)270-3916. The examiner can normally be reached M-F 8-5 Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached on 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000./JAMES T TSAI
/JAMES T TSAI/ Primary Examiner, Art Unit 2147
