DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Response to Amendment
Claims 1-5, 9-13, and 17-20 have been amended. No claims have been added or cancelled. The 35 U.S.C. 101 rejection, 35 U.S.C. 102(a)(1) rejection and the claim objections have been withdrawn as a result of amendment.
Response to Arguments
Applicant's arguments filed 01/09/2026 have been fully considered but they are not persuasive.
Examiner acknowledges Applicant’s arguments with respect to the 35 U.S.C. 101 rejection and finds them moot as the rejection has been withdrawn in view of amendment.
Examiner acknowledges Applicant’s arguments with respect to the 35 U.S.C. 102(a)(1) rejection and finds them moot with consideration to the updated 35 U.S.C. 103 rejection presented below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-5, 7-13, and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Chao et al. (US11893468, referred to as Chao) in view of Tappeiner (US20190070735), and further in view of Edsinger et al. (US9694495, referred to as Edsinger).
Regarding claim 1: Chao discloses: A method comprising: receiving a demonstration of a lab protocol for operating a type of machine from a user; recording a sequence of actions associated with operating the type of machine based on the demonstration; for each action in the sequence of actions, using computer vision to derive information associated with the action based on the recorded sequence of actions; extracting semantic meanings of the action based in part on the information derived from the computer vision; and generating a machine executable protocol for operating the type of machine by a robot based in part on the semantic meanings of the actions. ([col. 4, lines 18-21] the camera 108 captures video data of an area 130 in which a demonstration is performed. The demonstration is performed, in at least one embodiment, by a human demonstrator 110. [col. 4, lines 28-38] In the example 100, a box 112, can 114, and pot 116 are placed on work surface 124. The demonstrator 110 intends to cook the contents of the can 114 by placing the can's contents into pot 116 and moving the pot 116 to a burner of a stove 120. This goal is not communicated explicitly to the computing device 102. In at least one embodiment, the computing device 102 infers the goal based on video data of the observation that is captured by the camera 108. Subsequently, robotic device 106 can achieve the goal again, in a potentially different area 132 with a different stove 122 and work surface 126.) [and 2) information provided by a human in response to prompting the human for missing protocol information; and] performing, by the robot, the sequence of actions by executing the machine executable protocol. ([col. 20, lines 44-48] controller(s) 1136 may include one or more 45 onboard ( e.g., integrated) computing devices that process sensor signals, and output operation commands ( e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving vehicle 1100.)
Chao does not disclose: and 2) information provided by a human in response to prompting the human for missing protocol information; and
Chao does not disclose the following limitations, however Tappeiner, from an analogous field of endeavor, further teaches: and 2) information provided by a human in response to prompting the human for missing protocol information; and ([0077] a user can ask the robot to sing a song. In response to detecting likely voice voice audio and tonal qualities indicative of a question, the robot can open an audio stream with an online NLP service to understand the voice input. The robot can prompt the user for more information by playing an audio sound asking, “Which song?”)
Chao and Tappeiner are analogous art to the claimed invention since they are from the similar field of robotic systems for user engagement. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual training system of Chao to enable the prompt for missing user data as taught in Tappeiner.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to instances where additional data is needed prior to execution, as taught in Edsinger.
Regarding claim 2: The combination of Chao and Tappeiner teaches: The method of claim 1,
Chao does not explicitly disclose: wherein the demonstration comprises that the user physically moves a robotic arm or an end effector of the robotic arm of the robot to operate a machine of the type.
Chao does not disclose the following limitations, however Edsinger, in an analogous field of endeavor teaches: the demonstration comprising that the user physically moves the robotic arm or the end effector to operate a machine of the type ([col. 3, lines 55-66] A user may author a skill by defining one or more mechanism (i.e., subtasks), that when executed by a manipulator of the robotic device, the manipulator moves in such a way that the trajectory and forces imparted by the manipulator solve the skill, i.e., perform the task. The user may physically interact with the manipulator in a way that an input value (e.g., position or force) is inferred by the robotic device. For example, the user may push on the manipulator with a force, and the robotic device infers that this is the force the manipulator would apply to an object while performing a task.)
Chao, Tappeiner, and Edsinger are analogous art to the claimed invention since they are from the similar field of demonstratable robot teaching methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual training system of Chao to enable the physical movement demonstration taught in Edsinger.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to physical teaching taught in Edsinger.
Regarding claim 3: The combination of Chao, Tappeiner, and Edsinger teaches: The method of claim 2,
Chao does not explicitly disclose, however Edsinger further teaches: wherein the demonstration comprises moving the robotic arm and the end effector to cause the end effector to press a button of the machine of the type. ([col. 2, lines 27-40] The functions also include determining task information for a task to be performed by the robotic device. The task comprises a combination of Subtasks of the plurality of Subtasks. The task information includes a trajectory to be followed by the manipulator, where the trajectory comprises a combination of paths corresponding to the combination of Subtasks. The task information also includes forces to be exerted by the manipulator at points along the trajectory, where the forces are based on the respective forces experienced by the manipulator while performing subtasks of the combination of subtasks. The functions further include determining, based on the task information, torques to be applied over time to the manipulator [col. 5, lines 10-16] another example tool includes a “spring mechanism, where the end-effector behaves as if a viscous spring (spring and a damper) attach the end-effector to a an axis of motion Such that the end-effector applies a force or pushes on a Surface with a particular stiffness to simulate, for example, a push on a button or a door.)
Chao, Tappeiner, and Edsinger are analogous art to the claimed invention since they are from the similar field of demonstratable robot teaching methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual training system of Chao to enable the physical movement demonstration taught in Edsinger.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to physical teaching taught in Edsinger.
Regarding claim 4: The combination of Chao, Tappeiner, and Edsinger teaches: The method of claim 2,
Chao does not explicitly disclose, however Edsinger further teaches: wherein the demonstration comprises moving the robotic arm and the end effector to cause the end effector to grab an object from a location outside the machine of the type and place the object inside the machine of the type. ([col. 2, lines 27-40] The functions also include determining task information for a task to be performed by the robotic device. The task comprises a combination of Subtasks of the plurality of Subtasks. The task information includes a trajectory to be followed by the manipulator, where the trajectory comprises a combination of paths corresponding to the combination of Subtasks. The task information also includes forces to be exerted by the manipulator at points along the trajectory, where the forces are based on the respective forces experienced by the manipulator while performing subtasks of the combination of subtasks. The functions further include determining, based on the task information, torques to be applied over time to the manipulator)
As previously stated, Chao, Tappeiner, and Edsinger are analogous art to the claimed invention since they are from the similar field of demonstratable robot teaching methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual training system of Chao to enable the physical movement demonstration taught in Edsinger.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to physical teaching taught in Edsinger.
Regarding claim 5: The combination of Chao, Tappeiner, and Edsinger teaches: The method of claim 2,
Chao does not explicitly disclose, however Edsinger further teaches: wherein the demonstration comprises moving the robotic arm and the end effector to cause the end effector to open a lid of the machine of the type. ([col. 2, lines 27-40] The functions also include determining task information for a task to be performed by the robotic device. The task comprises a combination of Subtasks of the plurality of Subtasks. The task information includes a trajectory to be followed by the manipulator, where the trajectory comprises a combination of paths corresponding to the combination of Subtasks. The task information also includes forces to be exerted by the manipulator at points along the trajectory, where the forces are based on the respective forces experienced by the manipulator while performing subtasks of the combination of subtasks. The functions further include determining, based on the task information, torques to be applied over time to the manipulator)
Chao, Tappeiner, and Edsinger are analogous art to the claimed invention since they are from the similar field of demonstratable robot teaching methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual training system of Chao to enable the physical movement demonstration taught in Edsinger.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to physical teaching taught in Edsinger.
Regarding claim 7: The combination of Chao, Tappeiner, and Edsinger teaches: The method of claim 2,
Chao further discloses: wherein recording the sequence of actions comprises: recording a sequence of motions of the robotic arm; and recording a sequence of images surrounding an end effector of the robotic arm. ([col. 4, lines 18-21] the camera 108 captures video data of an area 130 in which a demonstration is performed. The demonstration is performed, in at least one embodiment, by a human demonstrator 110. [col. 4, lines 28-38] In the example 100, a box 112, can 114, and pot 116 are placed on work surface 124. The demonstrator 110 intends to cook the contents of the can 114 by placing the can's contents into pot 116 and moving the pot 116 to a burner of a stove 120. This goal is not communicated explicitly to the computing device 102. In at least one embodiment, the computing device 102 infers the goal based on video data of the observation that is captured by the camera 108. Subsequently, robotic device 106 can achieve the goal again, in a potentially different area 132 with a different stove 122 and work surface 126.)
Regarding claim 8: The combination of Chao and Edsinger teaches: The method of claim 2,
Chao further discloses: wherein generating a machine-executable protocol comprises: accessing a library of atomic actions that are executable by the robotic arm; for each action in the sequence of actions, identifying one or more atomic actions from the library; linking the one or more atomic actions corresponding to each action in the sequence to generate the machine-executable protocol. ([col. 117, lines 4-19] access to DICOM, RIS, CIS, REST compliant, RPC, raw, and/or other data type libraries may be accumulated and pre-processed, including decoding, extracting, and/or performing any convolutions, color corrections, sharpness, gamma, and/or other augmentations to data. In at least one embodiment, DICOM, RIS, CIS, REST compliant, RPC, and/or raw data may be unordered and a pre-pass may be executed to organize or sort collected data. In at least one embodiment, because various applications may share common image operations, in some embodiments, a data augmentation library ( e.g., as one of services 3720) may be used to accelerate these operations. In at least one embodiment, to avoid bottlenecks of conventional processing approaches that rely on CPU processing, parallel computing platform 3830 may be used for GPU acceleration of these processing tasks. [col. 4, lines 28-38] In the example 100, a box 112, can 114, and pot 116 are placed on work surface 124. The demonstrator 110 intends to cook the contents of the can 114 by placing the can's contents into pot 116 and moving the pot 116 to a burner of a stove 120. This goal is not communicated explicitly to the computing device 102. In at least one embodiment, the computing device 102 infers the goal based on video data of the observation that is captured by the camera 108. Subsequently, robotic device 106 can achieve the goal again, in a potentially different area 132 with a different stove 122 and work surface 126.)
Regarding claim 9: Rejected using the same rationale as claim 1, however further directed to a “non-transitory computer readable storage medium”, which is not explicitly disclosed by Chao
Chao does not explicitly disclose: non-transitory
Chao does not disclose the following limitations, however Edsinger, in an analogous field of endeavor teaches: non-transitory ([col. 7, lines 18-24] The system 100 may further include any type of computer readable medium (non-transitory medium) or memory, for example, Such as a storage device including a disk or hard drive, to store the program code that when executed by one or more processors cause the system 100 to perform the functions described above)
As previously stated, Chao, Tappeiner, and Edsinger are analogous art to the claimed invention since they are from the similar field of demonstratable robot teaching methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual training system of Chao to enable the common storage taught in Edsinger.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to the non-transitory storage taught in Edsinger.
Regarding claim 10: Rejected using the same rationale as claim 2.
Regarding claim 11: Rejected using the same rationale as claim 3.
Regarding claim 12: Rejected using the same rationale as claim 4.
Regarding claim 13: Rejected using the same rationale as claim 5.
Regarding claim 15: Rejected using the same rationale as claim 7.
Regarding claim 16: Rejected using the same rationale as claim 8.
Regarding claim 17: Chao discloses: An automation system comprising: a robotic arm with an end effector; a processor; a [non-transitory] computer-readable storage medium having instructions encoded thereon that, when executed by the processor, cause the automation system to: ([col. 11, lines 55-61] The operations depicted by FIG. 7 may be performed by a system, such as robotic device 106 depicted in FIG. 1. The robotic device, in at least one embodiment, comprises a robotic manipulation device, such as a robotic arm, at least one processor, and a memory comprising instructions that are executable by the at least one processor, and which when executed cause the robotic device to perform the operations described in relation to FIG. 7.) receive a demonstration of a lab protocol for operating a type of machine by a user, [the demonstration comprising that the user physically moves the robotic arm or the end effector to operate a machine of the type;] record a sequence of actions associated with operating the type of machine based on the demonstration, recording the sequence of actions comprising recording, by the imaging sensor, a sequence of images surrounding an end effector of the robotic arm; for each action in the sequence of actions, ([col. 4, lines 18-21] the camera 108 captures video data of an area 130 in which a demonstration is performed. The demonstration is performed, in at least one embodiment, by a human demonstrator 110.) use computer vision to derive information associated with the action based on the sequence of images; extract semantic meanings of the action based in part on the information derived from the computer vision; and generate a machine executable protocol for operating the type of machine by a robot based in part on the semantic meanings of the actions. ([col. 4, lines 28-38] In the example 100, a box 112, can 114, and pot 116 are placed on work surface 124. The demonstrator 110 intends to cook the contents of the can 114 by placing the can's contents into pot 116 and moving the pot 116 to a burner of a stove 120. This goal is not communicated explicitly to the computing device 102. In at least one embodiment, the computing device 102 infers the goal based on video data of the observation that is captured by the camera 108. Subsequently, robotic device 106 can achieve the goal again, in a potentially different area 132 with a different stove 122 and work surface 126.) [and 2) information provided by a human in response to prompting the human for missing protocol information; and] performing, by the robot, the sequence of actions by executing the machine executable protocol. ([col. 20, lines 44-48] controller(s) 1136 may include one or more 45 onboard ( e.g., integrated) computing devices that process sensor signals, and output operation commands ( e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving vehicle 1100.)
Chao does not disclose: and 2) information provided by a human in response to prompting the human for missing protocol information; and
Chao does not disclose the following limitations, however Tappeiner, from an analogous field of endeavor, further teaches: and 2) information provided by a human in response to prompting the human for missing protocol information; and ([0077] a user can ask the robot to sing a song. In response to detecting likely voice voice audio and tonal qualities indicative of a question, the robot can open an audio stream with an online NLP service to understand the voice input. The robot can prompt the user for more information by playing an audio sound asking, “Which song?”)
Chao and Tappeiner are analogous art to the claimed invention since they are from the similar field of robotic systems for user engagement. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual training system of Chao to enable the prompt for missing user data as taught in Tappeiner.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to instances where additional data is needed prior to execution, as taught in Edsinger.
Chao does not explicitly disclose: non-transitory … the demonstration comprising that the user physically moves the robotic arm or the end effector to operate a machine of the type
Chao does not disclose the following limitations, however Edsinger, in an analogous field of endeavor teaches: non-transitory ([col. 7, lines 18-24] The system 100 may further include any type of computer readable medium (non-transitory medium) or memory, for example, Such as a storage device including a disk or hard drive, to store the program code that when executed by one or more processors cause the system 100 to perform the functions described above)… the demonstration comprising that the user physically moves the robotic arm or the end effector to operate a machine of the type ([col. 3, lines 55-66] A user may author a skill by defining one or more mechanism (i.e., subtasks), that when executed by a manipulator of the robotic device, the manipulator moves in such a way that the trajectory and forces imparted by the manipulator solve the skill, i.e., perform the task. The user may physically interact with the manipulator in a way that an input value (e.g., position or force) is inferred by the robotic device. For example, the user may push on the manipulator with a force, and the robotic device infers that this is the force the manipulator would apply to an object while performing a task.)
As previously stated, Chao, Tappeiner, and Edsinger are analogous art to the claimed invention since they are from the similar field of demonstratable robot teaching methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual training system of Chao to enable the physical movement demonstration taught in Edsinger.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to physical teaching taught in Edsinger.
Regarding claim 18: Rejected using the same rationale as claims 2 and 10.
Regarding claim 19: Rejected using the same rationale as claims 3 and 11.
Regarding claim 20: Rejected using the same rationale as claims 4 and 12.
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Chao et al. (US11893468, referred to as Chao) in view of Tappeiner (US20190070735), further in view of Edsinger et al. (US9694495, referred to as Edsinger) and further in view of Astle (US5470744, referred to as Astle).
Regarding claim 6: The combination of Chao, Tappeiner, and Edsinger teaches: The method of claim 2,
Chao does not explicitly disclose: wherein the machine is one or more of an incubator, a centrifuge, an imaging microscope, a thermocycler, a flow cytometer, a cell counter, an ultra- low-temperature (ULT) freezer, an automated tissue processor, or a spectrophotometer.
Chao does not disclose the following limitations, however Astle, from an analogous field of endeavor, teaches: wherein the machine is one or more of an incubator, a centrifuge, an imaging microscope, a thermocycler, a flow cytometer, a cell counter, an ultra- low-temperature (ULT) freezer, an automated tissue processor, or a spectrophotometer. ([col. 3, lines 27-34] Robotic access window 44 is provided so that a robotic hand (not shown) can make access through this window to reach a selected one of a plurality of storage shelves, as at 46, disposed within incubator 10 and is minimally dimensioned therefor to minimize the disturbance of the interior environment of incubator 10.)
Chao, Tappeiner, Edsinger, and Astle are analogous art to the claimed invention since they are from the similar field of robotic arm control methods. It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention, with a reasonable expectation for success, to modify the visual robotic arm training system of Chao to enable the specific use on an incubator as taught in Astle.
The motivation for modification would have been to provide the teaching method disclosed in Chao with the method applied to robotic control of an incubator taught in Astle.
Regarding claim 14: Rejected using the same rationale as claim 6.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ATTICUS A CAMERON whose telephone number is 703-756-4535. The examiner can normally be reached M-F 8:30 am - 4:30 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Thomas Worden can be reached on 571-272-4876. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ATTICUS A CAMERON/ /JASON HOLLOWAY/ Primary Examiner, Art Unit 3658 Examiner, Art Unit 3658A
Prosecution Insights