DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to under 37 CFR 1.83(a) because they fail to show “ET illuminator target 254” as described in the specification. Any structural detail that is essential for a proper understanding of the disclosed invention should be shown in the drawing. MPEP § 608.02(d). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims.
Therefore, the “odometry system” must be shown or the feature(s) canceled from the claim(s). No new matter should be entered.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The abstract of the disclosure is objected to because of the implied language “A system of the subject technology”. A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Objections
Claim 18 objected to because of the following informalities: In claim 18, “ET” should be spelled out the first time it is used in an independent claim. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 3-5,18-20 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Claims 3 and 18 recite an “odometry system”. The examiner is not sure what this term means and the specification does not describe adequately this term; see drawing objection above. The applicant’s specification mentions “odometry system” four times in the specification paragraphs 0007, 0021, 0049 and 0064. In none of these paragraphs does the applicant further explore anything to do with an odometer or odometry system.
Using GOOGLE define the examiner found the following definition for “visual odometry systems”. “Visual odometry (VO) systems determine a robot's or vehicle’s position and orientation (pose) by analyzing consecutive camera images to track motion, often used in robotics and autonomous navigation.” The applicant’s specification discusses none of this.
Following the Wands factors for this rejection:
(A) The breadth of the claims; [AltContent: rect]
The breadth for these claims is concerning calibration in the robotic arts containing standard method and apparatus claims. The breadth is not “undue”.
(B) The nature of the invention;
The nature of these claims involve robotic arts and camera technology. [AltContent: rect]
(C) The state of the prior art;
The state of the prior art is well developed with both national and international art. [AltContent: rect]
(D) The level of one of ordinary skill;
Ordinary skill level in this technology would be someone with at least a Bachelor’s degree in this technology.[AltContent: rect]
(E) The level of predictability in the art; [AltContent: rect]
The level of predictability is still growing and progressing.
(F) The amount of direction provided by the inventor; [AltContent: rect]
At least for the particular item mentioned above there is very little direction provided.
(G) The existence of working examples; and
The examiner could find no working examples. [AltContent: rect]
(H) The quantity of experimentation needed to make or use the invention based on the content of the disclosure. [AltContent: rect]
A great deal of experimentation would be required to make the “visual odometry system” claimed.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-10, 18-20 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
In claim 1, two components are used in a process without having first been positively recited. In line three “camera” and in lines 9-10 “inertial sensors”. See MPEP 2115 “ A claim is only limited by positively recited elements. Thus, "[i]nclusion of the material or article worked upon by a structure being claimed does not impart patentability to the claims.”
Claims 3 and 18 are not clear. There is no recitation of the camera orientation so that a perspective is described indicating what direction would be inward and what direction is outward. In other words, inward from what and outward from what? The examiner will interpret this claim as follows, inward from the device and outward from the device.
Claims 3 and 18 are also not clear in what does “odometry system” apply too? Both a “visual” AND “a visual-inertial system” or just to the visual inertial system? The examiner will interpret this passage to mean that odometry system only applies to the visual-inertial system.
Claim Rejections - 35 USC § 103
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 8 (as best understood) are rejected under 35 U.S.C. 103 as being unpatentable over Ramirez Luna et al (US2022/0303435) in view of Bravo Orellana (US2023/0045287).
A system (fig. 5), comprising:
a robotic arm (506) including an end-effector (fig. 33 at 3304 & figures 34-40) configured to rigidly attach to a device (“For example, camera 300 rigidly attached to arm”); a
plurality of camera (300) calibration targets (¶0089, ¶0097 “camera 300 includes a target site 503”, ¶0463, ¶0504) positioned around the robotic arm (fig. 5 at 503, ¶0127 “target site 700 (also referred to as a scene or field-of-view (“FOV”) or target surgical site)”, ¶0234); and
a multifaceted calibration target (¶0085 next to last sentence, ¶0127 “target site 700 may also include…calibration slides/templates”) including an eye-tracking (ET) target (¶0426 “the camera 300 along the line of sight towards or away from the target point”) and an ET illuminator target (fig. 8 camera 300, 708a/b/c, 700),
wherein:
the robotic arm is configured to be operable to rotate the device during a calibration process (¶0506, claim 11), and
the calibration process comprises a simultaneous calibration of cameras, inertial sensors and illuminators of the device.
Ramirez Lunas does not disclose that the calibration process comprises a simultaneous calibration of cameras, inertial sensors and illuminators of the device.
Bravo Orellana discloses (¶0035 “calibration process”) a calibration process (“the calibration process is a Simultaneous Localization and Mapping (SLAM) method”) comprising a simultaneous calibration of cameras (“camera 103”), inertial sensors (“sensor”) and illuminators of the device (¶0050 “sensor 102 is for example a light detection and ranging (LiDAR)”). At the time the invention was made it would have been well within the skills of one of ordinary skill in this art to upgrade the device of Ramirez Lunas with the SLAM method of Bravo Orellana. A motivation for this would be to have more accurate mapping which would provide better operation by the robot. This combination follows the KSR case law rationale A; combining prior art elements according to known methods to yield predictable results.
Both Ramirez Lunas and Bravo Orellana do not disclose an inertial sensor. The applicant in the last sentence of the Background section mentions the use of “inertial measurement units (IMUs)”. At the time the invention was made it would have been well known to one of ordinary skill to use a well-known inertial sensor as disclosed by the applicant (see MPEP section 2129) with the device of Ramirez Lunas in view of Bravo Orellana. A motivation for this would be to more accurately measure movement of the robot.
Claim 2 Ramirez Lunas discloses the system of claim 1, wherein the device comprises a mixed reality device or a smart eyeglass (fig. 17 at 1712, ¶0082 “smartglasses, virtual reality devices”, ¶0101 “smart-eyewear”).
Claim 3 Ramirez Lunas discloses the system of claim 1, wherein the cameras comprise inward-facing (figures 45-46) cameras configured to support ET or face-tracking and outward-facing cameras (300 in fig. 47) configured to support a visual or a visual-inertial odometry system.
Claim 4 Ramirez Lunas discloses the system of claim 3, wherein the robotic arm is configured to be operable to rotate the device to allow the outward-facing cameras to observe the camera calibration targets in most camera frames (Abstract, ¶0090, ¶0389, ¶0506).
Claim 5 Ramirez Lunas discloses the system of claim 3, wherein the robotic arm is configured to be operable to rotate the device to allow the inward-facing cameras to observe the multifaceted calibration target in most frames alternating between the ET target and the ET illuminator target (¶0014, ¶0437, ¶0593-594).
Claim 8 Ramirez Lunas discloses the system of claim 1, wherein the ET targets comprise one or more faces with a calibration target having calibration patterns with known dimensions (¶0234 “known pattern”, ¶0352 different patterns disclosed).
Claim 6 (as best understood) is rejected under 35 U.S.C. 103 as being unpatentable over Ramirez Luna et al (US2022/0303435) in view of Bravo Orellana (US2023/0045287) as applied to claim 1 above, and further in view of Phan et al (US2016/0284075A1).
Ramirez Lunas discloses (figures 7 & 8) a camera calibration target (¶0503) covering a uniform illuminator (708c) but not a camera calibration target comprising a dark sheet including patterned holes.
Phan discloses (fig. 19) a camera calibration target (10) comprising a dark sheet (“black sheet of felt 18 “) including patterned holes (16) covering a light. At the time the invention was made it would have been well known to one of ordinary skill to design one of the targets of Ramirez Luna in view of Bravo Orellana to be a target comprising a dark sheet (“black sheet of felt 18 “) including patterned holes as taught by Phan. A motivation for this modification to Ramirez Lunas would be to use a tried and true target that is well known for its efficacy. This combination follows the KSR case law rationale B; simple substitution of one known element for another (target for target) to obtain predictable results.
Claim 7 (as best understood) is rejected under 35 U.S.C. 103 as being unpatentable over Ramirez Luna et al (US2022/0303435) in view of Bravo Orellana (US2023/0045287).
The plurality of camera calibration targets (Bravo Orellana ¶0074 & ¶0099 “target markers”) comprise simultaneous localization and mapping (SLAM) targets (Bravo Orellana ¶0035 & 0097) positioned around the robotic arm in a half-cube layout.
Although Ramirez Luna describes various configurations for positioning targets around the arm (e.g., ¶0234, ¶0501 & 502, 503) neither directly discloses the formation of a half cube layout.
At the time the invention was made it would have been well known to one of ordinary skill to use the process of Ramirez Luna in view of Bravo Orellana with different configurations of targets including a half cube layout. Well known case law reveals that claimed variations in relative dimensions, which do not specify a device, which performs or operates any differently from the prior art, do not patentably distinguish applicant's invention. Gardner v. TEC Systems, Inc., 725 F.2d 1338 (Ct. App. Fed. Cir. 1984).
Claim 9 (as best understood) is rejected under 35 U.S.C. 103 as being unpatentable over Ramirez Luna et al (US2022/0303435) in view of Bravo Orellana (US2023/0045287).
Ramirez Luna discloses eye targets that are either reflective (¶0096, ¶0135) or spherical (¶0081, ¶0501 ¶0583) but not both characteristics at the same time and with a known radii.
At the time the invention was made it would have been well known to one of ordinary skill in this art to combine the properties of spherical and reflective and a known radii for calibration targets. All the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention. A motivation for this expansion of the attributes of Ramirez Luna would be to change the eye target for different applications.
Claims 11, 13 are rejected under 35 U.S.C. 103 as being unpatentable over Ramirez Luna et al (US2022/0303435) in view of Bravo Orellana (US2023/0045287).
A method, comprising:
turning on illuminators (“B. Example Lighting Sources”, ¶0135 “To illuminate the target site 700”, “light sources 708a and 708b” & “NUV light source 708c”) of a device rigidly attached to an end-effector of a robotic arm ( 506, figures 2 & 33-40), the device including cameras (300, 212) and inertial sensors (Abstract 2nd half, 744, 746, 748, 3306, ¶0014 “a joint sensor configured to transmit a position”);
causing the robotic arm to rotate the device around to allow scanning a multifaceted calibration target (“target surgical site”, “lock-to-target feature of FIG. 64”, “target site 700”, “target template 2802 “) surrounding the robotic arm (¶0437, ¶0509, ¶0593-0594);
collecting data from the cameras and the inertial sensors (¶0014 “output data”, “data structures”; ¶0118, 0274, 0279, 0414 “output force/motion data”, 0439 “rotational and/or translational data”, 0502 “robotic arm controller 4106 also use output data from joint sensors”);
preprocessing the collected data (¶0568); and
running an algorithm to implement a simultaneous calibration of the cameras, the inertial sensors and the illuminators of the device using the preprocessed collected data (¶0502 “During the calibration, the processor 4102 and/or the robotic arm controller 4106 receive view vector data 4807”, “the calibration target simultaneously”).
Ramirez Lunas does not disclose that the calibration process comprises a simultaneous calibration of cameras, inertial sensors and illuminators of the device.
Bravo Orellana discloses (¶0035 “calibration process”) a calibration process (“the calibration process is a Simultaneous Localization and Mapping (SLAM) method”) comprising a simultaneous calibration of cameras (“camera 103”), inertial sensors (“sensor”) and illuminators of the device (¶0050 “sensor 102 is for example a light detection and ranging (LiDAR)”). At the time the invention was made it would have been well within the skills of one of ordinary skill in this art to upgrade the device of Ramirez Lunas with the SLAM method of Bravo Orellana. A motivation for this would be to have more accurate mapping which would provide better operation by the robot. This combination follows the KSR case law rationale A; combining prior art elements according to known methods to yield predictable results.
Both Ramirez Lunas and Bravo Orellana do not disclose an inertial sensor. The applicant in the last sentence of the Background section mentions the use of “inertial measurement units (IMUs)”. At the time the invention was made it would have been well known to one of ordinary skill to use a well-known inertial sensor as disclosed by the applicant (see MPEP section 2129) with the device of Ramirez Lunas in view of Bravo Orellana. A motivation for this would be to more accurately measure movement of the robot.
Claim 13 Ramirez Luna discloses the method of claim 11, wherein the cameras include outward-facing cameras (300 in fig. 47) and inward-facing cameras (figures 45-46), wherein scanning the multifaceted calibration target (figures 6 & 50) allows the outward-facing cameras to observe the multifaceted calibration target.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Ramirez Luna et al (US2022/0303435) in view of Bravo Orellana (US2023/0045287) as applied to claim 11 above and further in view of Raab et al (US2017/0054965).
Neither Ramirez Luna nor Bravo Orellana disclose the quantifying concept of sampling rates.
Raab discloses an imaging device that uses sampling rates to help quantify data (¶0148 & 0149). At the time the invention was made it would have been well known to one of ordinary skill to use the technique of sampling rates in data collection as taught by Raab to improve the device of Ramirez Luna in view of Bravo Orellana. A motivation for this addition to Ramirez Luna in view of Bravo Orellana would be to more accurately convert analog to digital data and to determine the fidelity and reliability of the data. This combination follows the KSR case law rationale C; use of known technique to improve similar devices (methods or products) in the same way.
Claims 18 (as best understood) are rejected under 35 U.S.C. 103 as being unpatentable over Ramirez Luna et al (US2022/0303435) in view of Bravo Orellana (US2023/0045287).
A calibration system, comprising:
a robotic arm ( 506, figures 2 & 33-40) operable to rotate a device during a calibration process (Abstract, ¶0506, claim 20);
a plurality of camera calibration targets positioned around the robotic arm (¶0433 “the robotic arm 506…calibration, target-lock…inputting a new orientation of the camera 300”, ¶0499 last sentence, ¶0502 next to last sentence); and
a multifaceted calibration target including an ET target and an ET illuminator target (¶0504, ¶0558),
wherein:
the device is attached to the robotic arm and includes cameras, inertial sensors and illuminators to be simultaneously calibrated during the calibration process, and
the cameras comprise inward-facing cameras (figures 45-46) configured to support ET or face-tracking and outward-facing (300 in fig. 47) cameras configured to support a visual or a visual-inertial odometry system.
Ramirez Lunas does not disclose that the calibration process comprises a simultaneous calibration of cameras, inertial sensors and illuminators of the device.
Bravo Orellana discloses (¶0035 “calibration process”) a calibration process (“the calibration process is a Simultaneous Localization and Mapping (SLAM) method”) comprising a simultaneous calibration of cameras (“camera 103”), inertial sensors (“sensor”) and illuminators of the device (¶0050 “sensor 102 is for example a light detection and ranging (LiDAR)”). At the time the invention was made it would have been well within the skills of one of ordinary skill in this art to upgrade the device of Ramirez Lunas with the SLAM method of Bravo Orellana. A motivation for this would be to have more accurate mapping which would provide better operation by the robot. This combination follows the KSR case law rationale A; combining prior art elements according to known methods to yield predictable results.
Both Ramirez Lunas and Bravo Orellana do not disclose an inertial sensor. The applicant in the last sentence of the Background section mentions the use of “inertial measurement units (IMUs)”. At the time the invention was made it would have been well known to one of ordinary skill to use a well-known inertial sensor as disclosed by the applicant (see MPEP section 2129) with the device of Ramirez Lunas in view of Bravo Orellana. A motivation for this would be to more accurately measure movement of the robot.
Allowable Subject Matter
Claims 10 & 19-20 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Claims 14-17 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael C Zarroli whose telephone number is (571)272-2101. The examiner can normally be reached Monday-Friday 9-5 ET IFP.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at 5712705744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MICHAEL C. ZARROLI
Primary Examiner
Art Unit 3658B
/MICHAEL C ZARROLI/Primary Examiner, Art Unit 3658 /M.C.Z/Primary Examiner, Art Unit 3658 .
Prosecution Insights