DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Status of Claims
Claims 1-4 were originally filed on 10/18/2024 and claimed priority on CN2024100602565, which was filed on 01/16/2024.
Claim Objections
Claims 1-4 are objected to because of the following informalities:
For claim 1, “a mixed reality device” in line 3 of claim 1 should be “the mixed reality device” since the mixed reality device has already been introduced in line 2.
“the operator’s behavior” in the last line should be “an operator’s behavior” since this is the first time the operator is being introduced.
“at slave side” and “at master side” should be “at a slave side” and “at a master side”.
For claims 2-4, there should be a “wherein” after the preamble (e.g. according to claim 1, wherein the virtual fixture; according to claim 1, wherein the point cloud; according to claim 3, wherein a coordinate system).
For claim 3, “a real environment collected by a depth camera” should be “the real environment collected by the depth camera” since the real environment and the depth camera have already been introduced.
For claim 4, “of master robot” should be “of the master robot”, “of slave robot” should be “of the slave robot”, and “mounted on a slave robot” should be “mounted on the slave robot”.
“an OST-HMD” should be “an Optical See-Through Head-Mounted Display (OST-HMD)” since this is the first time the OST-HMD acronym is being introduced.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al (CN115229789A) in view of Lee et al “Implementation of Augmented Teleoperation System based on Robot Operating System (ROS)” (Hereinafter referred to as Zhu and Lee respectively)
Regarding Claim 1, Zhu teaches a master-slave teleoperation robot system (See at least Zhu Page 1 Paragraph 3, and Page 2 Paragraph 8, the slave robot is teleoperated using the master robot)…comprising a master robot and a slave robot (See at least Zhu Page 2 Paragraph 8),… an end of the slave robot is equipped with a depth camera (See at least Zhu Page 4 second to last Paragraph-Page 5 Paragraph 4, the end effector of the slave robot has a depth camera), the depth camera is used to collect point cloud information of a real environment (See at least Zhu Page 5 Paragraphs 1-4, the depth camera is used to generate a point cloud of a real environment)…the point cloud information is used to produce a virtual fixture (See at least Zhu Page 5 Paragraph 5, the point cloud information is used to generate a virtual fixture), the virtual fixture is used to produce a virtual force which is applied to the slave robot (See at least Zhu Page 5 Paragraph 5 and Page 6 Paragraph 5), the virtual force comprises a guiding force and a prohibiting force (See at least Zhu Page 5 Paragraph 5, and Page 7 Paragraphs 3 and 6, the guiding fixture generates a guiding force and the forbidden area fixture generates a prohibiting/repulsive force), the virtual force interacts with an actual environment interaction force at slave side to guide and norm the operator’s behavior at master side (See at least Zhu Page 6 Paragraph 5, Page 7 Paragraphs 3 and 6-7, and Page 8 Step 104-Paragraph 5, the virtual force interacts with the actual force at the slave side to help guide the operator at the master side).
Even though Zhu teaches a master-slave teleoperation robot system with a master robot, Zhu fails to disclose the system is…based on force mixed reality, the master robot is connected with a mixed reality device, which is used to receive information and produce a virtual environment in a mixed reality device; and send the point cloud information to the mixed reality device in real time.
However, Lee teaches the system is…based on force mixed reality (See at least Lee Page 5498 Section III, Page 5500 Section VI Paragraph 2, and Page 5500-5501 Section VII Paragraph 2 and Paragraph 5, the master-slave system uses force mixed/augmented reality), the master device is connected with a mixed reality device, which is used to receive information and produce a virtual environment in a mixed reality device (See at least Lee Page 5498 Section III, the master device is connected to an AR system/mixed reality device, which receives the 3D camera information from the slave side and produces a virtual environment); and send the point cloud information to the mixed reality device in real time (See at least Lee Page 5498 Section III, and Pages 5498-5499 section IV, the 3D camera information used to generate a point cloud is sent to the mixed reality device/AR device).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in Zhu with Lee to have the master robot be connected with a mixed reality device, which is used to receive information and produce a virtual environment in a mixed reality device; and send the point cloud information to the mixed reality device in real time. This modification, as taught by Lee, would enhance a teleoperator interface by incorporating multi-modal augmented reality, so as to draw more performance and intuition from the human operator (See at least Lee Abstract).
Regarding Claim 2, modified Zhu teaches the virtual fixture comprises a forbidden-region virtual fixture (FRVF) and a guidance virtual fixture (GVF) (See at least Zhu Page 5 Paragraph 5), through an artificial potential field, the point cloud information of a prohibited area of the real environment is collected to produce the forbidden-region virtual fixture (FRVF) (See at least Zhu Page 6 last Paragraph-Page 7 first paragraph and Figures 4-5, the point cloud information is used to generate reference points and an artificial field around the reference points as the forbidden region virtual fixture); the guidance virtual fixture (GVF) is a human-robot interaction motion path that the operator wants to achieve (See at least Zhu Page 6 Paragraph 4, Page 7 Paragraph 6, Page 8 Paragraph 5, and Figure 3, the guidance virtual fixture provides an attraction force for the desired motion path of the operator).
Claims 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu in view of Lee, and in further view of Azuma et al (US 20240169550 A1) (Hereinafter referred to as Azuma)
Regarding Claim 3, modified Zhu teaches the point cloud information is a three-dimensional information of a real environment collected by a depth camera (See at least Zhu Page 3 last paragraph and Page 5 Paragraphs 1-4, the depth camera collects 3d information of the real environment to generate the three-dimensional point cloud).
Even though modified Zhu teaches the master robot being connected to the mixed reality device, which is used to receive the point cloud information and produce a virtual environment (See claim 1 rejection), modified Zhu fails to disclose after the point cloud information is transformed to a robot base coordinate system, it is sent to the master robot in real time, the master robot and the slave robot are in an isomorphic master-slave configuration, therefore, in the mixed reality device, the point cloud information can be directly projected to the corresponding position in the base coordinate of the master robot.
However, Azuma teaches after the point cloud information is transformed to a robot base coordinate system, it is sent to the master robot in real time (See at least Azuma Paragraphs 0055, 0111-0114, and Figure 1, the point cloud data is transformed into the robot coordinate system, and the data is sent to the operation controller of the master robot), the master robot and the slave robot are in an isomorphic master-slave configuration (See at least Azuma Paragraphs 0036, 0051, and Figure 1, the master robot/operator and slave robot have the same shape and corresponding coordinate systems, which is interpreted as an isomorphic master-slave configuration), therefore,…the point cloud information can be directly projected to the corresponding position in the base coordinate of the master robot (See at least Azuma Paragraphs 0051, and 0114, the point cloud data is transformed into the robot coordinate system, which corresponds to the coordinate system of the master robot/operator).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Zhu with Azuma to send the point cloud information to the master robot after it is transformed to a robot base coordinate system, and so that the point cloud information can be directly projected to the corresponding position in the base coordinate of the master robot. Azuma teaches that the master robot and slave robot have corresponding coordinate systems (See at least Azuma Paragraphs 0036, 0051, and Figure 1), and that point cloud data is transformed to the slave coordinate system and sent to the master robot (See at least Azuma Paragraphs 0055, 0111-0114 and Figure 1). This modification allows the system to project the point cloud information onto the coordinate system of the master device, thus, allowing the operator of the master device to visualize the environment of the slave robot, which would increase the awareness of the operator and improve the teleoperation.
Regarding Claim 4, modified Zhu fails to disclose a base coordinate system of slave robot, a coordinate system of the depth camera mounted on a slave robot, and a coordinate system of an OST-HMD.
However, Lee teaches a base coordinate system of slave robot (See at least Lee Pages 5498-5499 section IV, the slave robot includes a base coordinate system), a coordinate system of the depth camera mounted on a slave robot (See at least Lee Pages 5498 section III, and Pages 5498-5499 section IV, the RGB-Depth camera on the slave robot has a coordinate system), and a coordinate system of an OST-HMD (See at least Lee Pages 5498-5499 section IV, and Page 5499 section V first paragraph, each independent component, which includes the head-mounted display for augmented reality, has its own reference frame).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Zhu with Lee to have a base coordinate system of slave robot, a coordinate system of the depth camera mounted on a slave robot, and a coordinate system of an OST-HMD. This modification, as taught by Lee, would allow the system to easily calculate transformations between coordinate systems of different components (See at least Lee Pages 5498-5499 section IV), thus, allowing the system to determine the relative locations between different components.
Modified Zhu fails to disclose a base coordinate system of master robot.
However, Azuma teaches a base coordinate system of master robot (See at least Azuma Paragraphs 0051, the operation coordinate system of the operator is the coordinate system of the master robot).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings disclosed in modified Zhu with Azuma to have a base coordinate system of master robot. A defined orthogonal three-axis coordinate system for a master robot, as taught by Azuma, is well-known and routinely used for position control of a robot. This modification would allow for the system to keep track of the position of the master robot, which would improve the teleoperation.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hong et al (US 20220148458 A1) teaches adaptive force guidance by using virtual fixtures for telesurgery
Chizeck et al (US 20180232052 A1) teaches collecting point cloud data and generating virtual fixtures of for a teleoperation system
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ESVINDER SINGH whose telephone number is (571)272-7875. The examiner can normally be reached Monday-Friday: 9 am-5 pm est.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Lin can be reached at 571-270-3976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ESVINDER SINGH/Examiner, Art Unit 3657