DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/23/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Examiner notes that US 2624238, “Selectively Reflecting Interference Mirrors,” does not appear to be relevant to the present application. Publication EP 2624238, listed in the Applicant-provided International Search Report dated 11/10/2022, has also been considered.
Drawings
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims. Therefore, the hand-mounted camera calculating position data for the hand by visual odometry and the hand-mounted camera providing the control signal to the industrial machine must be shown or the feature(s) canceled from the claim(s). No new matter should be entered.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference characters "400" and "700" have both been used to designate a sensor arrangement. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “700” has been used to designate both a sensor arrangement and a camera arrangement. In paragraph [00145], “sensor arrangement 700” should read “camera arrangement 700”.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference characters "230" and "270" have both been used to designate an orientation monitor module. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “230” has been used to designate both an orientation monitor module and a modality monitor module. In paragraph [00147], “orientation monitor module 230” should read “orientation monitor module 270”.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference characters "225" and "227" have both been used to designate a hand-mounted camera. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “225” has been used to designate both a fixed camera and a hand-mounted camera. In paragraph [00218], “hand-mounted camera 225” should read “hand-mounted camera 227”.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference characters "110" and "120" have both been used to designate an op-unit. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “110” has been used to designate both a location change unit and an op-unit. In paragraph [00218], “op-unit 110” should read “op-unit 120”.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “912” has been used to designate both a low-speed interface in [00223] and a low-speed controller in [00226].
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “914” has been used to designate both a low-speed bus in [00223] and a low-speed expansion port in [00226].
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “980” has been used to designate both a GPS receiver module in [00233] and a cellular telephone in [00235].
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “982” has been used to designate both an expansion interface in [00231] and a smart phone in [00235].
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 970, 972, and 974.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objections to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the presence of numerous informalities. The following is an incomplete list of examples:
Specific types of modules are often shortened to just “module” with or without a reference character. For example, see “Module 228 merges both relative position data that go (via module 240)” in [00218]. Similar shortened references are used for “unit” and “device” throughout the Specification. To avoid confusion, Examiner requests such elements be labeled consistently with their full names.
Paragraph [0060] recites “system 200… would control to stop”. What is controlled or stopped is not stated.
In [0061], there is a reference to “both options”, but only one option is explained.
In [00171]-[00172], “camera attachment 700” should read “camera arrangement 700” for consistency.
Paragraph [00218] recites “Module 228 merges both relative position data that go (via module 240) as merged relative position (data) to machine 100”. This sentence does not make sense. How does data “go” via processing module 240?
Numerous other grammatical errors are present in the Specification, such as run-on sentences (see [0016], [0023], and [0082]), missing or wrong articles and prepositions (see [0050] and [0071]), and incorrect verb conjugations (see [0050] and [0093]).
Appropriate correction is required.
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Objections
Claims 3-4, 8, and 12-14 are objected to because of the following informalities:
Claim 3 recites “wherein the computer system is adapted to receive the modality attribute with a first modality value when… the pair of fingers are touching each other, and provides the modality attribute with a second modality value when the pair of fingers are not touching each other.” It is not understood why the computing system would receive a modality attribute in one case, but provide the modality attribute in the opposite case. What does the computer system provide the modality attribute to? Note that claim 2 recites that the computer system is adapted to receive a modality attribute, but does not recite that the computer system is adapted to provide a modality attribute.
In claim 8, “implemented by” should be deleted for conciseness. Alternatively, the wording “wherein the further image capturing device and the hand-mounted camera comprise RGB cameras” can be used.
In claim 12, in each instance of “a human operator”, “an industrial machine”, “a computer system”, and “an operation”, “a” or “an” should be replaced by “the” as each object has been previously recited in claim 10. Additionally, “monitoring movement” should read “monitoring the movement” for the same reason. Optionally, consider deleting “that is adapted to control an operation of an industrial machine by monitoring movement of the human operator” for conciseness as this limitation has been previously recited in claim 10.
Claim 13 should start with “A ”. Additionally, “signal to an industrial machine” should read “signal to the industrial machine” as an industrial machine is previously recited in the preamble.
In claim 14, “a industrial machine” should read “an industrial machine”.
Appropriate correction is required.
Claim Interpretation
Examiner did not see a special definition for “industrial machine” in the Specification. The term has been interpreted under its broadest reasonable interpretation—a machine used in any industry—in the rejections below.
An “operative unit” has been interpreted as a physical part of an industrial machine and “related to the function of the [industrial] machine” [0077]. The Specification does not support an operative unit as being a unit of software.
In claim 2, “including receiving the modality attribute… touching” has been interpreted as a limitation of what the computer system is further adapted to do.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Song and Bae (KR 20180077974 A; hereafter “Song”) in view of Gonenc et al. (US 20210128260 A1; hereafter “Gonenc”).
Regarding claim 1, Song discloses
A computer system configured to obtain a control signal to control an operation of a… machine that has an operative unit (Computer system: a VR-robot linkage system comprising a linkage server 100 [0021]; machine: robot system 300; operative unit: robot arm 46 and/or interaction equipment. See “The linkage server (100)… obtains user body data in the second coordinate system, which is a real coordinate system, from the tracking sensor (400), and controls the robot system (300) by generating robot data matching the second object data using the interaction data” [0023]. See “The robot system (300) can receive a robot control signal from the linkage server (100) and move the robot arm and interaction equipment to [operation:] directly provide physical feedback to the user” [0024].),
the computer system monitoring a movement of a human… (The user is a human; see [0022] and Fig. 4. See “The sensor (400) can sense [monitor] the position and movement of a part of the user's body. According to one embodiment of the present invention, the sensor (400) may be a camera that can capture a part of a user's body and generate position data for the part of the body. The tracking sensor (400) can provide information about a sensed part of the user's body to the linkage server (100) so that the linkage server [part of computer system] (100) can generate body data” [0025]. See also [0042]-[0043].),
the computer system comprising: a hand-mounted camera mounted on a hand of the human… (Computer system: a VR-robot linkage system, which includes a sensor 400 [0021]. See “a sensor (400) mounted on a part of the user's body (e.g., a wearable device in the form of a glove)… the tracking sensor (400) may be a camera” [0042]. The camera 400 is therefore mounted on the user’s hand with a glove.) and
configured to capture images and calculate position data for the hand therefrom by visual odometry (Capture images: see “tracking sensor (400) may be a camera, and the user movement acquisition unit (113) may acquire the user's body data from an image captured by the camera” [0042]. Calculate position data: see “the sensor (400) may be a camera that can capture a part of a user's body and generate position data for the part of the body” [0025]. Finally, “the user movement acquisition unit (113) can acquire data tracking the user's hand from a camera, which is a tracking sensor (400). That is, the user movement acquisition unit (113) can acquire the movement of the user's hand based on coordinates in the real world” [0043]. Together, these quotations tell us that the camera 400 captures images of the user’s hand, generates position data of the hand from the images, and tracks the hand’s movement in real-world coordinates, which is visual odometry. The camera 400 then gives the tracking data to the user movement acquisition unit 113. See also [0012] and [0055].),
…being further adapted to provide the control signal to the… machine to cause the… machine to move according to the position data (Machine: robot system 300 comprising robot arm 46. The hand 43 position data is used to generate a corresponding VR hand object 31, which interacts with another VR object, like a VR ball 32 [0049]. The VR ball 32 has a corresponding real-world ball 45 attached to a robot arm 46 [0049]. See “the robot system (44) controls the robot arm (46) to move according to the robot control signal generated by the VR-robot linkage system [computer system] so that the robot ball (45) comes into contact with the user's hand (43)” [0049]. Therefore, the robot arm moves according to the position data. See also [0044], [0051], and [0057].).
However, Song does not explicitly teach “an industrial machine”, “a human operator”, and “the hand-mounted camera being further adapted to provide the control signal to the industrial machine to cause the industrial machine to move according to the position data.”
Gonenc, in the same field of endeavor (robotic control systems), teaches
A computer system configured to obtain a control signal to control an operation of an industrial machine that has an operative unit (Computer system: user console 2 comprising console computer system 16; industrial machine: surgical robotic system 1 (used in the healthcare industry); operative unit: robotic arm(s) 4 and/or the attached surgical tools 7. See Fig. 1 and Fig. 2. Obtain a control signal: “In one embodiment, the remote operator 9 holds and moves the UID [user input device] 14 to provide an input command [control signal] to drive one or more robotic arm actuators 17 in the robotic system 1 for teleoperation. The UID 14 may be communicatively coupled to the rest of the robotic system 1, e.g., via a console computer system 16” [0036]. In this case, an operation is a surgical operation [0034]. See also [0082].),
the computer system monitoring a movement of a human operator (See “the remote operator 9 holds and moves the UID 14 to provide an input command to drive one or more robotic arm actuators 17 in the robotic system 1 for teleoperation” [0036]. The remote operator 9 is a human; see Fig. 1.),
…being further adapted to provide the control signal to the industrial machine to cause the industrial machine to move according to the position data (See “The UID 14 can generate spatial state signals corresponding to movement of the UID 14, e.g. position and orientation of the handheld housing of the UID, and the spatial state signals may be input signals [control signal] to control motions of the robotic arm actuators 17… the movement of a corresponding surgical tool that is attached to the arm may mimic the movement of the UID 14. Similarly, interaction between the remote operator 9 and the UID 14 can generate for example a grip control signal that causes a jaw of a grasper of the surgical tool 7 to close and grip the tissue of patient 6” [0036]. See also [0037] and [0082].).
Song’s camera 400 is a device to input user commands by visual odometry. Gonenc’s user input device (UID) 14 generates position data and corresponding control signals to control a robotic system to move and mimic a surgeon operator’s movements. Combining Song’s camera 400 with Gonenc’s UID 14 to control an industrial robot arm teaches “the hand-mounted camera being further adapted to provide the control signal to the industrial machine to cause the industrial machine to move according to the position data.” Thus, the combination of Song and Gonenc as a whole teaches the claim.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the robot control system of Song with the input system of Gonenc. One of ordinary skill in the art would have been motivated to make this modification so that the user can operate multiple tools; for example, “The surgical robotic system 1 may include several UIDs 14, where respective control signals are generated for each UID that control the actuators and the surgical tool (end effector) of a respective arm 4” (Gonenc, [0037]).
Regarding claim 9, Song/Gonenc disclose the limitations of claim 1 as addressed above, and Gonenc additionally discloses
wherein the computer system is further adapted to monitor an orientation of the hand and to derive roll, pitch and yaw data further used to obtain the control signal (In Gonenc, see “The UID 14 can generate spatial state signals corresponding to movement of the UID 14, e.g. position and orientation of the handheld housing of the UID, and the spatial state signals may be input signals [control signal] to control motions of the robotic arm actuators 17” [0036]. From the orientation sensed by the UID, roll, pitch, and yaw data is derived to control the robot arm 4 in a “planned trajectory T can be provided with respect to a 3-axis coordinate system, such as a system of mutually-perpendicular X-, Y-, and Z-axes, and can include translational movement along one or more of the X-, Y-, and Z-axes, as well as rotational orientation about one or more of the X-, Y-, and Z-axes, e.g., roll, pitch, and yaw” [0054]. See also [0040].).
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Song and Gonenc and further in view of Adams et al. (US 9104271 B1; hereafter “Adams”) and Simão et al. (“Natural control of an industrial robot using hand gesture recognition with neural networks”, 2016; hereafter “Simão”).
Regarding claim 2, Song/Gonenc disclose the limitations of claim 1 as addressed above, including being adapted to provide the control signal to the industrial machine to cause the industrial machine to move according to the position data. Song additionally discloses
wherein the computer system is adapted to provide the control signal and thereby cause the… machine to move… the operative unit… (Machine: robot system 300 comprising robot arm 46. The hand 43 position data is used to generate a corresponding VR hand object 31, which interacts with another VR object, like a VR ball 32 [0049]. The VR ball 32 has a corresponding real-world ball 45 attached to a robot arm 46 [0049]. See “the robot system (44) controls the robot arm (46) to move according to the robot control signal generated by the VR-robot linkage system [computer system] so that the robot ball (45) comes into contact with the user's hand (43)” [0049]. Therefore, the operative unit (robot arm 46 and/or attached robot ball 45) moves according to the position data. See also [0044], [0051], and [0057].).
Gonenc additionally discloses that the computer system, through the user input device 14, receives finger movement data (“The system 1 translates the surgeon's hand, wrist, and finger movements through the UID 14 and the foot-operated controls 13 into precise real-time movements of the surgical tools” [0082]).
However, Song/Gonenc does not explicitly teach “wherein the position data is determined at a first repetition interval; wherein the computer system is further adapted to receive a modality attribute of the hand of the human operator at a second repetition interval that is shorter than the first repetition interval by a pre-defined factor, including receiving the modality attribute from a modality converter that processes modality data indicating for a pair of fingers of the hand whether the pair of fingers are touching each other or not touching; wherein the computer system is adapted to provide the control signal and thereby cause the industrial machine to move or to stop the operative unit depending on the modality attribute.”
Adams, in the same field of endeavor (human-machine interfaces), teaches
wherein the position data is determined at a first repetition interval (See “The hand position tracking process 37 can utilize the processor to calculate the pixel location of each light emitting diode in a continuous stream of digital image frames generated by the camera 13 at a predetermined rate [first repetition interval], which can be approximately 20 to approximately 120 frames per second, e.g., approximately 100 frames per second. Via the hand position tracking process 37, the processor can use the light emitting diode pixel locations to calculate the position [position data] and/or orientation of the outer glove 12 with respect to the camera 13” [col. 9, 1-17]. The position data is determined at a first repetition interval of 20-120 frames per second, which is equivalent to determining the position every 8-50 milliseconds);
wherein the computer system is further adapted to receive a modality attribute of the hand of the human operator at a second repetition interval that is shorter than the first repetition interval by a pre-defined factor (See “The recognition of a finger tapping motion in the mouse gesture recognition process 46 can comprise: (1) estimation of individual finger speed as the change in finger position over a predetermined clock interval [second repetition interval] on the processor which can be in a range between approximately 0.1 and approximately 20 (e.g., approximately 1.0) milliseconds” [col. 10, lines 13-27]. This finger movement data can be analyzed for mouse button event data (modality attribute) 47 like “MOUSE_CLICK, MOUSE_DOUBLE_CLICK, MOUSE_DOWN, MOUSE_UP, and/or WHEEL_SCROLL for any or all buttons including LEFT, RIGHT, and/or MIDDLE, etc.” received by computing system 23 [col. 10, lines 3-13]. The suggested second repetition interval (1.0 millisecond) is shorter than the suggested first repetition interval (100 frames per second = every 10 milliseconds) by a factor of 10. See also Fig. 11.),
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the industrial machine control system of Song/Gonenc with the finger tapping detection of Adams. One of ordinary skill in the art would have been motivated to make this modification for the benefit of tracking the position of all of a user’s fingers and “prevent[ing] natural finger motion synergies (for example between the ring and pinky fingers) from generating erroneous events” (Adams; col. 9, line 60 to col. 10, line 2 and col. 10, lines 14-28).
However, Adams does not explicitly teach “including receiving the modality attribute from a modality converter that processes modality data indicating for a pair of fingers of the hand whether the pair of fingers are touching each other or not touching; wherein the computer system is adapted to provide the control signal and thereby cause the industrial machine to move or to stop the operative unit depending on the modality attribute.”
Simão, in the same field of endeavor (gestural robotic control), teaches
including receiving the modality attribute from a modality converter that processes modality data indicating for a pair of fingers of the hand whether the pair of fingers are touching each other or not touching (See Fig. 1: the neural network (modality converter) can distinguish 25 static gestures (SGs) and 10 dynamic gestures (DGs), including if the thumb is touching any other finger (SG8-SG11), based on a data glove and magnetic tracker [III.A. Data Acquisition and III.B. Gesture Data Set]. These gestures can each correspond to a robotic command received by a robot controller [I. Introduction; page 1, col. 2 to page 2, col. 1]. See also section III.E. Robot Control.);
…to provide the control signal and thereby cause the industrial machine to move or to stop the operative unit depending on the modality attribute (Industrial machine: “industrial robot” arm; operative unit: end-effector/gripper [III.E. Robot Control]. See via direct teleoperation, “The virtual joystick is activated by closing the fist (S7) and moving it in the direction the subject wants the robot to translate to… a command [control signal] is sent to [the] robot to incrementally re-position [move] the end-effector” [III.E. Robot Control]. See “Since the closed fist gesture is necessary to use the virtual joystick, anytime the subject [human operator] is not performing it, the system is paused… Other gestures are used to forcefully stop the robot - see figure 4 (c) -, open or close the gripper and select the rotation axis” [III.E. Robot Control]. In this case, the specified modality attribute values include “go” (closed fist; virtual joystick is activated) and “forceful stop” (hand spread out as in Fig. 4(c)). See also Fig. 4 and its caption.).
Adams teaches a position of a hand is determined less often than a gesture/movement of the hand. The sensors of Simão send modality data to a neural network (in a computer) to classify a gesture. The classified gesture is a modality attribute used to control a robot arm and gripper. As previously addressed, Song/Gonenc’s computing system provides the control signal to the industrial machine. Thus, the combination as a whole teaches the claim.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the industrial machine control system of Song/Gonenc/Adams with the gestural detection and control of Simão. One of ordinary skill in the art would have been motivated to make this modification for the benefit of allowing the operator “to re-position and reorient himself” and perform “safe collaboration with the robot” (Simão, III.E. Robot Control and caption of Fig. 4).
Regarding claim 3, Song/Gonenc/Adams/Simão disclose the limitations of claim 2 as addressed above. Adams additionally discloses
wherein the computer system is adapted to receive the modality attribute with a first modality value… (Finger movement data can be analyzed for mouse button event data (modality attribute) 47 like “MOUSE_CLICK, MOUSE_DOUBLE_CLICK, MOUSE_DOWN, MOUSE_UP, and/or WHEEL_SCROLL for any or all buttons including LEFT, RIGHT, and/or MIDDLE, etc.” (modality values) received by computing system 23 [col. 10, lines 3-13].)
Simão additionally discloses
…receive the modality attribute with a first modality value when the modality data indicates that for the pair of fingers, the pair of fingers are touching each other (As stated in the rejection of claim 2, the specified modality attributes include “go” (closed fist; virtual joystick is activated) and “forceful stop” (hand spread out as in Fig. 4(c)). See “The virtual joystick is activated by closing the fist (S7) and moving it in the direction the subject wants the robot to translate to… a command [control signal] is sent to [the] robot” [III.E. Robot Control]. In the closed fist gesture, each finger touches the finger(s) next to it; we can pick a specific finger pair where the fingers in the pair are touching each other. The modality attribute is provided to the robot controller with a “go” value. See also Fig. 4 and its caption.), and
provides the modality attribute with a second modality value when the pair of fingers are not touching each other (See “Other gestures are used to forcefully stop the robot - see figure 4 (c)” [III.E. Robot Control]. The forceful stop gesture (see Fig. 4(c) and SG20 in Fig. 1) has all fingers spread apart. Therefore, in no pair of fingers are the fingers touching each other. The modality attribute is provided to the robot controller with a “stop” value. Note that the neural network reliably distinguished all gestures in this experiment, so any of the gestures in Fig. 1 could be assigned to the first or second modality values, including the thumb touching each other finger in SG8-SG11.).
In the combination of Song/Gonenc/Adams/Simão, the computer system can obtain/receive and provide the modality attribute to internal (i.e., software) or external (i.e., additional computing hardware) units as a part of the control signal. For example, in Adams, the computer system 23 receives the modality attribute from the mouse gesture recognition process 46 (software unit) as input for use in other software [col. 10, lines 13-27]. A person of ordinary skill in the art would be able to configure the systems to receive or provide the modality attribute as best applied to a particular industrial machine.
Regarding claim 4, Song/Gonenc/Adams/Simão disclose the limitations of claim 2 as addressed above. Adams additionally discloses
wherein the computer system is communicatively coupled to a sensor arrangement that is attached to the hand of the human operator and is adapted to receive signals from the sensor arrangement at the second repetition interval, wherein the signals either communicate modality data or the modality attribute (Flexion sensors 20 (sensor arrangement) are embedded in a data glove and therefore attached to the hand of the human operator [col. 5, line 28 to col. 6, line 9]. Fig. 10 shows that the flexion sensors 20 are communicatively coupled to portable electronics 14. Then see “flexible cable 17 can… carry event messages [e.g., mouse button event data 47], diagnostic messages, sensor data messages [e.g., flexion sensor data], and/or any other information from the portable electronics 14 to the computing system 23” [col. 5, line 28 to col. 6, line 9]. Therefore, the computer system is communicatively coupled to the flexion sensor arrangement. The flexion sensor (finger movement) data signals are received at the second repetition interval; see “The recognition of a finger tapping motion in the mouse gesture recognition process 46 can comprise: (1) estimation of individual finger speed as the change in finger position over a predetermined clock interval [second repetition interval] on the processor which can be in a range between approximately 0.1 and approximately 20 (e.g., approximately 1.0) milliseconds” [col. 10, lines 13-27].).
Claims 5-7 are rejected under 35 U.S.C. 103 as being unpatentable over Song and Gonenc and further in view of Oleynik (US 20160059412 A1).
Regarding claim 5, Song/Gonenc disclose the limitations of claim 1 as addressed above. However, Song/Gonenc does not explicitly teach “further comprising a further image capturing device, and wherein the computer system is adapted to obtain the position data by image processing using the hand-mounted camera and the further image capturing device.”
Oleynik, in the same field of endeavor (robot control systems), teaches
further comprising a further image capturing device (See “Sensor-data from a multitude of sensors, such as (but not limited to) smell 126, video cameras 128, infrared scanners and rangefinders 130, stereo (or even trinocular) cameras 132, haptic gloves 134, articulated laser-scanners 136, virtual-world goggles 138, microphones 140 or an exoskeleton motion suit 142, human voice 144, touch-sensors 146, and even other forms of user input 148, are used to collect data through a sensor interface module 150” [0330]. The possible image capturing devices include at least the video cameras 128 and stereo cameras 132. See also three-dimensional vision sensors 66 in Fig. 7A.), and
wherein the computer system is adapted to obtain the position data by image processing using the hand-mounted camera and the further image capturing device (First, the list of sensors above, including the video cameras 128 and stereo cameras 132, “may not be limited to capturing human position and/or motion,” implying that the cameras do capture human position and motion data [0330]. This includes position of the human chef’s hands [0337]. Image processing: see “The data is acquired and filtered 152, … after which a multitude of (parallel) software processes [running on a computer system] utilize the temporal and spatial data to generate the data that is used to populate the machine-specific recipe-creation process” [0330]. More details on image processing are given in [0337]-[0340]. See also Fig. 5A and Fig. 5C.).
Since Song teaches a camera 400 on a glove [0042], the hand-mounted camera and Oleynik’s haptic gloves 134 are easily combined. Oleynik further teaches (image) processing “all available data measurable in the chef studio 44” including “two-dimensional and three-dimensional data collected by multi-spectrum sensory equipment (including cameras…)” [0340]. Thus, the combination teaches obtaining position data by image processing using multiple cameras, including the hand-mounted camera and the further image capturing device.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the industrial machine control system of Song/Gonenc with a camera and the central sensor processing of Oleynik. One of ordinary skill in the art would have been motivated to make this modification for the benefit of avoiding occlusions of an “eye-in-hand” camera and providing “a higher-level view of the robotic hand's 72 gripping of [an] object, and whether the robotic hand has gripped or relinquished/released the object” by placing another camera a distance away (Oleynik; [0397] and [0392]; though these specific paragraphs are about a robot, the motivation applies to the cameras imaging the human operator).
Regarding claim 6, Song/Gonenc/Oleynik discloses the limitations of claim 5 as addressed above. The combination additionally teaches “wherein the computer system is further configured to… use second position data obtained by the image processing.” As addressed in the rejection of claim 1, the hand-mounted camera of Song/Gonenc calculates (first) position data obtained by visual odometry, which is a type of image processing. As addressed in the rejection of claim 5, the computer system calculates (second) position data from the images of the further image capturing device by image processing.
However, Song/Gonenc/Oleynik does not explicitly teach “wherein the computer system is further configured to selectively use first position data obtained by visual odometry when an orientation of the hand is within pre-defined ranges”.
Oleynik additionally teaches
wherein the computer system is further configured to selectively use first position data obtained by visual odometry when an orientation of [a] hand is within pre-defined ranges, or use second position data obtained by the image processing (Use first position data: “The palm of the robotic hand 72 includes an RGB-D sensor 500… [that] uses structured light to capture the shape of the object, three-dimensional mapping and localization, path planning, navigation, object recognition and people tracking [0392]. Localization based on image features (e.g., objects in the kitchen) from a camera 500 on the moving robotic hand is visual odometry. “However, the robotic hand 72 with the eye-in-hand 518 [comprising RGB-D camera 500] may encounter occlusions when grasping an object” [0397]. In [0395]-[0397], Oleynik indicates that while one camera may be occluded in one situation, a different camera may not be occluded. Use second position data: the computer system can determine “whether the robotic hand has gripped or relinquished/released the object” from data from video camera 66 because “video camera 66 is positioned at an angle and some distance away from the robotic hand 72, and therefore provides a higher-level view of the robotic hand's 72 gripping of the object” [0392]. See also [0393], [0409]-[0410], and [0416].).
Though Oleynik mentions the problem of occlusions only with respect to the camera of the robotic hand, the same solution of using a camera with a different field of view is applicable to the human operator’s hand-mounted camera. Video camera 66 is a further image capturing device that “provides a way to capture, follow, or direct the movement of the kitchen tool as used by the [human] chef 49,” which is replicated by the robotic kitchen (Oleynik, [0392]). Where the hand-mounted camera is occluded, the further image capturing device tracks movement of the hand.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to selectively use position data from a hand-mounted camera 500 when an orientation of the hand is in pre-defined ranges (not oriented such that an object occludes its background) or use position data from image processing of images from the further image capturing device 66 when the hand-mounted camera is occluded. The modification would have been obvious because a person of ordinary skill in the art would have been motivated to avoid occlusions of a hand-mounted camera and provide “a higher-level view of the [human] hand's 72 gripping of [an] object, and whether the [human] hand has gripped or relinquished/released the object” by using data from another camera with a different field of view (Oleynik; [0392]).
Regarding claim 7, Song/Gonenc/Oleynik discloses the limitations of claim 5 as addressed above. Oleynik additionally discloses
wherein the computer system is further configured to consolidate preliminary position data from the hand-mounted camera and from the further image capturing device (See “The [sensor] data is acquired and filtered 152, … after which a multitude of (parallel) software processes [running on a computer system] utilize the temporal and spatial data to generate the data that is used to populate the machine-specific recipe-creation process” [0330]. The filtering process is shown in Fig. 5C. Hand position data is consolidated in data process mapping algorithm 220. Where both the hand-mounted camera and the further image capturing device are 2D cameras, the data is consolidated in data extraction and mapping engine 224. Where both cameras are 3D cameras, the data is consolidated in data-reduction and abstraction engine 226. Regardless of the source, all data is consolidated again in the data melding step of the recipe-script generation engine process 222. Data process mapping algorithm 220, data extraction and mapping engine 224, data-reduction and abstraction engine 226, and recipe-script generation engine process 222 all run on the central computer system [0337]. See also [0338]-[0341].).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Song, Gonenc, and Oleynik in view of Fang and Zhang (“Experimental Evaluation of RGB-D Visual Odometry Methods”, 2015; hereafter “Fang”).
Regarding claim 8, Song/Gonenc/Oleynik disclose the limitations of claim 5 as addressed above, including that the hand-mounted camera calculates position data by visual odometry as in claim 1.
However, Song/Gonenc/Oleynik does not explicitly teach “wherein the further image capturing device and the hand-mounted camera are implemented by RGB-cameras.”
Fang, also tracking position via visual odometry, teaches
wherein the further image capturing device and the hand-mounted camera are implemented by RGB-cameras (Fang presents a comparison of several visual odometry methods that use RGB-D (a type of RGB) cameras: “several RGB-D visual odometry estimation methods have been proposed by using visual data and/or depth data” [page 2, col. 2].).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have specified that the further image capturing device and the hand-mounted camera of Song/Gonenc/Oleynik were RGB cameras as taught by Fang. One of ordinary skill in the art would have been motivated to make this modification for the benefit of using “both RGB and depth information in different ways to improve the robustness and accuracy of motion estimation” (Fang, page 2, col. 2).
Claims 10-11 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Song and Adams.
Regarding claim 10, Song discloses
A sensor arrangement comprising: a camera (See “the tracking sensor (400) may be a camera” [0042].); and
wherein the sensor arrangement is adapted for attachment to a hand of a human… (See “a sensor (400) mounted on a part of the user's body (e.g., a wearable device in the form of a glove)… the tracking sensor (400) may be a camera” [0042]. The camera 400 is therefore mounted on the user’s hand with a glove. The user is a human; see [0022] and Fig. 4.), and
configured to communicate position data to a computer system that controls an operation of the… machine by monitoring movement of the human… using the camera… (Computer system: a VR-robot linkage system comprising a linkage server 100 [0021]; machine: robot system 300; operative unit: robot arm and/or interaction equipment. Communicate position data and monitoring movement: see “The sensor (400) can sense [monitor] the position and movement of a part of the user's body. According to one embodiment of the present invention, the sensor (400) may be a camera that can capture a part of a user's body and generate position data for the part of the body. The tracking sensor (400) can provide information about a sensed part of the user's body to the linkage server (100) so that the linkage server [part of computer system] (100) can generate body data” [0025]. See “The linkage server (100)… obtains user body data in the second coordinate system, which is a real coordinate system, from the tracking sensor (400), and controls the robot system (300) by generating robot data matching the second object data using the interaction data” [0023]. See “The robot system (300) can receive a robot control signal from the linkage server (100) and move the robot arm and interaction equipment to [operation:] directly provide physical feedback to the user” [0024]. See also [0042]-[0043].).
However, Song does not explicitly teach “a visual odometry processor”, “a human operator of an industrial machine”, and “a computer system that controls an operation of the industrial machine… by monitoring movement of the human operator using the… visual odometry processor.”
Adams, in the same field of endeavor (human-machine interfaces), teaches
A sensor arrangement comprising: a camera (camera 13; see Fig. 1.); and
a visual odometry processor (Computing system 23. See “Three light emitting diodes can create a triad that can be tracked by a camera 13 as a rigid body. In this embodiment, the computing system 23 can calculate up to three-dimensional position coordinates (x, y, and/or z) for either and/or each triad as well as its orientation (roll, pitch, and/or yaw)” [col. 16, line 65 to col. 17, line 11].),
wherein the sensor arrangement is… configured to communicate position data to a computer system that controls an operation of the industrial machine… (Sensor arrangement: camera 13 and computing system 23. The movement and position (3D coordinates) calculated by computing system 23 are then used for “providing control inputs to a robotic manipulator,” for example [col. 16, line 65 to col. 17, line 11] or for input to a computer or other information device [col. 3, lines 9-22]. See “Computing system 23 in turn can be connected to, or encompassed within, additional computing, communications, and/or other user interface elements” [col. 5, lines 3-27]. Such other computing elements or information devices may be a server, which provides services like “monitoring, management, and/or control of… industrial equipment [and] machine tools,” according to a server’s definition in col. 32, line 51 to col. 33, line 10. Controlling machine tools in industrial equipment or a robotic manipulator is controlling an operation of an industrial machine.)
…by monitoring movement of the human operator using the camera and the visual odometry processor (See claim 14: “a hand movement tracking circuit operatively configured to track movement of a gloved hand of a human relative to a predetermined reference point.” See also col. 9, lines 1-17; and col. 20, line 54 to col. 21, line 7. The human is an operator by virtue of controlling one of the industrial machines above with their hand movements. See “The hand position tracking process 37 can observe hand position input 36 through the position and/or motion of any and/or each light emitting diodes 11 on the back of one or more outer gloves 12. The hand position tracking process 37 can utilize the processor to calculate the pixel location of each light emitting diode in a continuous stream of digital image frames generated by the camera 13” [col. 9, lines 1-17]. The camera 13 captures images of the user’s hand, and the hand position tracking process running on computing system 23 calculates position coordinates of the hand from the images and tracks the hand’s position and movement in real-world coordinates, which is visual odometry. See also col. 16, line 65 to col. 17, line 11 and col. 17, lines 24-30.).
Song discloses a camera 400 attached to a human hand by a glove. Adams discloses a small camera 13 connected to a processor in computing system 23 by a camera cable 18. By shrinking the camera cable 18 or directly connecting Song’s camera 400 to Adams’s computing system 23, we have a sensor arrangement adapted for attachment to a hand of a human operator of an industrial machine. Thus, the combination teaches the claim as a whole.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the robot control system of Song with the industrial machine control method of Adams. One of ordinary skill in the art would have been motivated to make this modification for the benefit of using gesture recognition and tactile feedback to control input to computer systems in hazardous or sensitive environments, such as contamination-free manufacturing and outer space (Adams, col. 2, line 62 to col. 3, line 51).
Regarding claim 11, Song/Adams discloses the limitations of claim 10 as addressed above, and Song additionally discloses
The sensor arrangement according to claim 10, that is adapted for attachment to the hand by being mounted on a glove, being mounted on a wrist wrap, being mounted by adhesive tape, or being mounted on a finger ring (See “a mounted camera or a sensor (400) mounted on a part of the user's body (e.g., a wearable device in the form of a glove)” [0042]. The combined sensor arrangement of Song’s camera 400 and Adams’s computing system 23 is adapted for attachment to the hand by being mounted on a glove.)
Regarding claim 13, Song discloses
Computer-implemented method to operate a… machine, comprising: monitoring a position for at least one hand of a human… with an image capturing device that is mounted on the at least one hand of the human… and that captures images of the at least one hand (Machine: robotic system 300 comprising robot arm 46; image capturing device: camera 400. Monitoring for a body part: “The sensor (400) can sense [monitor] the position and movement of a part of the user's body… the sensor (400) may be a camera that can capture [an image of] a part of a user's body and generate position data for the part of the body” [0025]. The body part being a hand: “the user movement acquisition unit (113) can acquire data tracking the user's hand from a camera, which is a tracking sensor (400)” [0043]. Mounted on the hand: see “a sensor (400) mounted on a part of the user's body (e.g., a wearable device in the form of a glove)… the tracking sensor (400) may be a camera, and the user movement acquisition unit 113 may acquire the user’s body data from an image captured by the camera” [0042]. The camera 400 is therefore mounted on the user’s hand with a glove and