DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Group I (claims 1-12) in the reply filed on 09/03/2025 is acknowledged.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “video processing unit” in claim 1.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
A review of the specification clarifies that this structure is never explicitly defined; however, the figures depict and the specification at [0040] clarifies that the invention comprises at least one general purpose computer which executes the algorithms and instructions of the invention. As such the term appears to be implicitly defined and will be interpreted to cover any general processor/computer configured to perform the same functions.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 and 7 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by WO2020205829A1 by Feather et al. (hereafter Feather).
Regarding claim 1, Feather teaches: 1. A surgical robotic system (as this is a preamble limitation it does not confer any examinable structure and the examiner also notes that this is a “system” defined by the structures iterated in the body of the claim hereinafter taught by Feather; however and for compact prosecution purposes, the examiner notes that while surgical robotic structures do not appear required by the claim at the current juncture they are still taught by Feather as Feather is integrated into a surgical robotic system including e.g. robotic arms as per [0076]-[0082] and Fig. 12 noting e.g. robotic arms 1212) comprising:
an endoscopic (see Feather’s [0032], 0036], [0040] and/or [0046] etc. for the camera being mounted on an endoscope) camera configured to output a stereoscopic video stream (see Feather’s [0036], [0040]-[0041], and/or [0080] etc. each of which describe gathering a stereoscopic video stream);
a video processing unit coupled to the endoscopic camera (in summary see Feather’s [0003]-[0008] for the entire system being processor controlled, and/or see [0021]-[0023] for the processing system being used in general for the depth mapping and surgical assistance, with the specific processing steps/functions addressed below), the video processing unit configured to:
process the stereoscopic video stream using a first algorithm to obtain a first depth map; process the stereoscopic video stream using a second algorithm to obtain a second depth map (Feather does not use the terms “first depth map” and “second depth map”, but nonetheless has two depth maps. One of Feather’s depth maps is a classical depth map directly generated using the stereoscopic video stream, called a depth map or an initial depth map, and described in various places such as [0040]-[0041] or [0050]-[0051] which will hereafter be referred to as the first depth map. Another one of Feather’s depth maps is referred to as a model. This is at various points called an object model, expanded model, aligned model, or simply the model; but in any event, it is a second data set generated from the stereo images that maps depth information to the image and is therefore a depth map by ordinary meaning, see e.g. [0037] describing the model in general and showing that it has depth information, see [0035] or [0042]-[0044] for the term object model having depth information, see [0058] for the term expanded model having depth information, see [0061]-[0062] for the aligned model having depth information where these more specific terms therefore also have data inclusive of the tissue surrounding the object as well. The examiner iterated each of these hereinabove in order to be able to later refer to any of Feather’s model terms as a depth map; however, the examiner does not need to rely on ordinary meaning at least for the general case/prime embodiment of Feather as Feather also explicitly states that the model is / is used to generate a depth map per se in [0021] which iterates in salient part: “The processing system accesses a model of the object (e.g., model data representative of a computer-assisted design (CAD) model). The processing system identifies a pose of the object (e.g., a position and an orientation of the object) in the surgical space and aligns the model with the pose of the object to define depth data for the model in the surgical space. The processing system generates a depth map in the surgical space for at least a portion of the object based on the depth data for the model.” Which will hereafter be referred to as the second depth map);
compare the first depth map to the second depth map; determine accuracy of the first depth map based on a comparison of the first depth map to the second depth map (this is most easily shown by viewing Figs. 5 and 7-8 wherein the first depth map has its accuracy analyzed and areas of low accuracy are shown in cross-hatching, then if the second depth map has accurate information for such areas, the second depth map is used to enhance the first depth map as shown in Fig. 9 with this sort of accuracy based comparison also described in [0063] and especially [0046] which iterates that “The modification of depth map 306 may include replacing unreliable, unknown, and/or difficult to determine depth data with depth data that is determined based on the aligned object model. The model-based depth data may represent reliable and/or known depth values that effectively transform depth map 306 into enhanced depth map 314 that may be more reliable, accurate, and/or complete than depth map 306.” Where 306 is the first depth map and the model is again the second depth map); and
display the stereoscopic video stream enhanced by the first depth map depending on the accuracy of the first depth map (displaying the video stream enhanced by the first depth map (and also by the second depth map for such areas as the comparison indicated low accuracy, low reliability, or incompleteness in the first depth map) is covered in many sections of Feather such as [0022], [0064]m and [0076] with additional information about why this is done also being potentially relevant as per [0023] and [0074] and/or is depicted in Fig. 10).
Regarding claim 7, Feather teaches: 7. A method for processing video data of a surgical scene (Feather is integrated into a surgical robotic system including e.g. robotic arms as per [0076]-[0082] and Fig. 12 noting e.g. robotic arms 1212 such that the video discussed below is video of the surgical scene), the method comprising:
outputting a stereoscopic video stream from an endoscopic camera to a video processing unit (see Feather’s [0032], 0036], [0040] and/or [0046] etc. for the camera being mounted on an endoscope; see Feather’s [0036], [0040]-[0041], and/or [0080] etc. each of which describe gathering a stereoscopic video stream; then see in summary see Feather’s [0003]-[0008] for the entire system being processor controlled, and/or see [0021]-[0023] for the processing system being used in general for the depth mapping and surgical assistance, with the specific processing steps/functions addressed below);
processing the stereoscopic video stream using a first algorithm to obtain a first depth map; processing the stereoscopic video stream using a second algorithm to obtain a second depth map (Feather does not use the terms “first depth map” and “second depth map”, but nonetheless has two depth maps. One of Feather’s depth maps is a classical depth map directly generated using the stereoscopic video stream, called a depth map or an initial depth map, and described in various places such as [0040]-[0041] or [0050]-[0051] which will hereafter be referred to as the first depth map. Another one of Feather’s depth maps is referred to as a model. This is at various points called an object model, expanded model, aligned model, or simply the model; but in any event, it is a second data set generated from the stereo images that maps depth information to the image and is therefore a depth map by ordinary meaning, see e.g. [0037] describing the model in general and showing that it has depth information, see [0035] or [0042]-[0044] for the term object model having depth information, see [0058] for the term expanded model having depth information, see [0061]-[0062] for the aligned model having depth information where these more specific terms therefore also have data inclusive of the tissue surrounding the object as well. The examiner iterated each of these hereinabove in order to be able to later refer to any of Feather’s model terms as a depth map; however, the examiner does not need to rely on ordinary meaning at least for the general case/prime embodiment of Feather as Feather also explicitly states that the model is / is used to generate a depth map per se in [0021] which iterates in salient part: “The processing system accesses a model of the object (e.g., model data representative of a computer-assisted design (CAD) model). The processing system identifies a pose of the object (e.g., a position and an orientation of the object) in the surgical space and aligns the model with the pose of the object to define depth data for the model in the surgical space. The processing system generates a depth map in the surgical space for at least a portion of the object based on the depth data for the model.” Which will hereafter be referred to as the second depth map);
and comparing the first depth map to the second depth map; determining accuracy of the first depth map based on a comparison of the first depth map to the second depth map (this is most easily shown by viewing Figs. 5 and 7-8 wherein the first depth map has its accuracy analyzed and areas of low accuracy are shown in cross-hatching, then if the second depth map has accurate information for such areas, the second depth map is used to enhance the first depth map as shown in Fig. 9 with this sort of accuracy based comparison also described in [0063] and especially [0046] which iterates that “The modification of depth map 306 may include replacing unreliable, unknown, and/or difficult to determine depth data with depth data that is determined based on the aligned object model. The model-based depth data may represent reliable and/or known depth values that effectively transform depth map 306 into enhanced depth map 314 that may be more reliable, accurate, and/or complete than depth map 306.” Where 306 is the first depth map and the model is again the second depth map); and
displaying the stereoscopic video stream enhanced by the first depth map depending on the accuracy of the first depth map (displaying the video stream enhanced by the first depth map (and also by the second depth map for such areas as the comparison indicated low accuracy, low reliability, or incompleteness in the first depth map) is covered in many sections of Feather such as [0022], [0064]m and [0076] with additional information about why this is done also being potentially relevant as per [0023] and [0074] and/or is depicted in Fig. 10).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 2-4 and 9-11 are rejected under 35 U.S.C. 103 as being unpatentable over Feather as applied to claims 1 and 7 above, and further in view of Fast Depth Map Super-Resolution using Deep Neural Network by Korinevskaya et al. (hereafter Korinevskaya).
Regarding claims 2-4 and 9-11, Feather teaches the basic invention as given above in regards to claims 1 and 7 and additionally, both of Feather’s depth maps are analytical reconstruction algorithms (noting the breadth of the terminology, any analysis leading to a reconstruction (i.e. a depth map image being formed) is an ‘analytical reconstruction’ so as to render this inherent in any generation of a depth map given the current claim breadth. For compact prosecution purposes the examiner notes that the first depth map is a classical depth map formed by image analysis of raw stereo video frames, see e.g. [0036], and the second depth map again uses analysis of the raw stereo video, along with other information, to form a depth map as per e.g. [0021] so as to also fit within how the applicant’s specification uses the term) so as to teach the subject matter of the body of claim 3/10. However, as Feather’s depth maps are based on analytical reconstruction Feather fails to teach “wherein the first algorithm is a deep learning image processing algorithm” as required by claim 2/9 or that the “wherein the deep learning image processing algorithm is adjusted based on the second depth map” as required by claim 4/11. Therefore, Feather alone fails to fully teach the subject matter of claims 2-4 and 9-11.
However Korinevskaya in the related field of depth map formation and solving the same problem of how to efficiently form depth maps from video streams, teaches the use of deep learning neural networks such as CNNs and GANs to create depth maps in real time (this can actually be seen from the mere Abstract of Korinevskaya; however and for compact prosecution purposes the examiner notes that section 3, on pages 117-118 address in all due detail how these deep learning algorithms can be used to form depth maps applicable to any image data set) and Korinevskaya goes on to teach that these sorts of depth maps are advantageous over convention depth maps of the sort utilized by the Feather (Korinevskaya’s section 1 on page 117 sets forth some disadvantages to and limitations of traditional depth mapping, an then in sections 1 and 3 on pages 117-118 it is established that the deep learning based depth maps have improved sharpness and quality while still being able to run in real time (e.g. 30 Hz)).
Therefore it would have been obvious to one of ordinary skill in the art prior to the date of invention to improve Feather by replacing or supplementing the first depth map with the deep learning based depth map generation taught by Korinevskaya in order to advantageously improve the sharpness and quality of the depth map.
Claim(s) 5-6 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Feather IVO Korinevskaya as applied to claims 2 and 11 above, and further in view of Miniature 6-axis force/torque sensor for force feedback in robot-assisted minimally invasive surgery by Li et al. (Hereafter Li).
Regarding claims 5-6 and 12 Feather IVO Korinevskaya teaches the basic invention as given above in regards to claims 2 and 11, and Feather further teaches: 5. The surgical robotic system according to claim 2, further comprising: a robotic arm including an instrument and at least one … sensor. 6. The surgical robotic system according to claim 5, wherein the second algorithm receives sensor feedback corresponding to physical contact by the instrument from the at least one … sensor. 12 The method according to claim 11, wherein processing the stereoscopic video stream using the second algorithm, further includes receiving sensor feedback from at least one … sensor corresponding to physical contact by a robotic instrument (see Feather’s [0076]-[0082] or Fig. 12 for the robotic arms 1212, additionally, [0033]-[0034] and [0078] state that displacement, orientation, position, force, or kinematic sensors can be mounted on the arms or the instruments and iterates that these are used by the processing system in determining the pose of the object, thus in the second map determination).
In the foregoing the examiner omitted the term “torque” as indicated by ellipsis as Feather uses more generic term force sensors or other kinematic sensors without specifying that they are torque sensors per se. While force sensors are technically a genus with torque sensors being a species thereunder, the examiner notes that the near ubiquity of torque sensors for determining force in surgical robots has caused the terms to be used interchangeably and would at the very least be the case that torque sensors are at once envisaged from the generic force sensor, see MPEP 2131.02(III). For compact prosecution purposes, at least in the alternative, the examiner notes that advantageous torque sensor setups designed for use in surgical robots could also be brought in to serve as the force/kinematic sensor of Feather as follows.
Li in the related field of surgical robotic systems teaches a force-torque sensor of the sort that could serve as Feather’s force sensor but is explicitly a torque sensor which is designed to be used in this same exact application per se (se Li’s Section 1 Introduction and Section 2 sensor overview which establish as much) and Li goes on to teach that this specific sensor arrangment is also advantageous to use (see Li’s Section 7 Discussion noting e.g. “Advantages and novelties of the sensor presented in this work are pointed out as follows. Based on the resistive sensing method, a flexural-hinged Stewart platform is employed and designed, which makes it an effective 6-axis force/torque sensor. Based on isotropy of measured forces and sensor sensitivity, a straightforward parameterized optimization method is proposed to balance the sensitivity and stiffness of the flexible structure. Therefore, the sensing ranges are large enough and sensor resolutions are very high.”).
Therefore and in the alternative, it would be obvious to one of ordinary skill in the art prior to the date of invention to utilize a torque sensor as taught by Li as the force sensor of Feather in order to advantageously ensure that the sensor being used is one that has high resolution and large sensing range.
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Feather as applied to claim 7 above, and further in view of Virtual wall–based haptic-guided teleoperated surgical robotic system for single-port brain tumor removal surgery by Seung et al. (hereafter Seung).
Regarding claim 8, Feather teaches the basic invention as given above in regards to claim 7 and even teaches that the depth map processing and enhanced image generated using the first depth map is designed to help guide the surgeons actions during surgery (simply stated in Feather’s [0023] or [0086]); however, Feather does not describe any limitations on the operation of the robotic surgical system and therefore fails to fully teach “generating a virtual wall based on the first depth map; and limiting movement of a robotic arm based on the virtual wall.”
However Seung in the same or eminently related field of image guided robotic surgical procedures, teaches how and why to use of virtual walls and iterates numerous reasons and advantages to employ them (while nearly any section of Seung could be cited to teach as much, the Introduction section teaches this succinctly by iterating that: “Virtual wall–based haptic guidance, or so-called active constraint, is a well-known technology to improve accuracy and safety during human–robot collaborative work, including surgery. Since it was introduced first in Rosenberg,20 it was implemented for various robotic systems, as described in Bowyer et al.21 The Active Constraint ROBOT (ACROBOT), the robotic system for bone joint replacement surgery, is one of the representative surgical robotic systems that uses virtual wall–based active haptic guidance.22,23 This “hands-on” robotic system restricts its tool position within the predefined virtual wall, resulting that a safer operation becomes possible. Robotic surgery with the virtual wall–based haptic guidance has been shown to be very effective in hard tissue removal as described in the literature.23–25 The virtual wall–based haptic guidance is also developed to deal with soft tissues. In Park et al.,26 the virtual fixtures was developed for the internal mammary artery harvest of an artery bypass graft procedure, which made the execution to be performed more quickly and precisely. In order to deal with a moving soft tissue, such as a beating heart, a dynamic 3D virtual fixture was also proposed and implemented in Ren et al.27 In addition, for several decades there have been lots of efforts to evaluate the performance and to analyze the stabilities in the literature.21,22,28–30”. In other words, since around 1993 it has been common practice to implement virtual walls in robotic surgery in order to increase safety, prevent damage to healthy tissues, and effectively remove specific tissues quickly and precisely. For compact prosecution purposes the examiner also notes that the Materials and methods section as a whole and in particular the Virtual wall generation and collision detection algorithm subsection therein contain all relevant details including how the tool tip depth is used to establish and avoid the virtual walls).
Therefore, it would have been obvious to one of ordinary skill in the art prior to the date of invention to improve the robotic surgery method of Feather with the use of virtual walls limiting the robotic arm’s movement in order to advantageously improve the safety of the system, prevent damage to healthy tissues, and more effectively and precisely remove specific tissues as taught by Seung.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure is as follows:
Automatic Corrections of Human Body Depth Maps using Deep Neural Networks by Gorana Gojić et al. is similar reference to Korinevskaya which utilizes deep learning to generate depth maps and which has other relevant teachings related to combining/enhancing depth maps.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael S Kellogg whose telephone number is (571)270-7278. The examiner can normally be reached M-F 9am-1pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui Pho can be reached at (571)272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL S KELLOGG/Examiner, Art Unit 3798
/PASCAL M BUI PHO/Supervisory Patent Examiner, Art Unit 3798