Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Falco et al., US 2008/0219405 A1, “Falco”, in view of Fan et al., US 2014/0369584 A1, “Fan”.
Regarding claim 2, FALCO discloses a camera system (FALCO, ¶ [[0009] To circumvent these issues, it has been proposed to use a camera system installed in the treatment room to obtain external surface information from the patient, and, based on images obtained from the cameras. While this approach may be able to compensate for changes in the patient's external surface, changes in internal anatomy (which can occur on a daily basis) are not considered. For example, the lung/chest wall interface position relative to the patient surface can change daily, especially if the patient's arm position is not reproducible. This interface is important since the whole breast, including the chest wall, must be treated uniformly while maintaining a minimal amount of radiation dose to the lung and/or heart) for use (FALCO, abstract,) with a radiotherapy treatment system (FALCO, FIG. 6, abstract) having a coordinate system (FALCO, ¶ 23 In another aspect, a system for determining an adjustment to a radiation treatment plan includes a receiver for receiving a radiation treatment plan, a visual representation of a patient's external feature and a visual representation of a patient's internal anatomical feature, and a treatment positioning module. The radiation treatment plan includes various treatment parameters that describe the location of a patient with respect to the external features and internal anatomical features. The visual representation of the patient's external feature is referenced to a first reference coordinate system, and the visual representation of the patient's internal feature is referenced to a second reference coordinate system. Based on the radiation treatment plan and the received visual representations, the treatment positioning module adjusts one or more of the treatment parameters to compensate for changes in the position of the patient with respect to their internal anatomy.), said camera system comprising: a camera configured to image a physical calibration phantom (FALCO ¶ [0063] In one embodiment, the system provides a signal to the user indicating that a threshold difference between the present anatomical data and stored treatment plan data has been exceeded. This signal may include, but is not limited to, any appropriate visual and/or acoustical signal. Alternatively, exceeding a threshold value may result in the treatment system automatically recalculating the treatment plan and adjusting one or more system parameters in accordance with the new plan. In a further alternative embodiment, a plurality of threshold values may be set, with different system responses depending upon the specific threshold exceeded/) positioned in a camera coordinate system (¶ [0062] Here, it can first be determined whether the differences in the external and/or internal data are lower than a predetermined threshold amount. If the differences are below these thresholds, the external data may be used to determine an appropriate adjustment of one or more beam parameters, while the internal data may be used to determine an appropriate adjustment of the patient position, as described above. However, if the difference between the present measurements and the stored treatment plan data, for either the external or internal data, exceeds the set threshold, a more involved adjustment and/or recalculation may be required. This may involve adjusting the beam parameter(s) and/or patient position. Alternatively, if all threshold values are exceeded, a partial or complete recalculation of the treatment plan may be required.); and
at least one processor configured to: receive image data (FALCO, ¶ [0068] In an alternative embodiment, an automated computer planning system capable of calculating dosages and other treatment parameters generates a new treatment plan prior to each treatment session, taking dose calculations and the newly determined patient anatomy positioning into account. Based on patient surface and lung information, an optimization routine finds the best beam shapes and dosages to deliver a uniform dose to the breast while minimizing lung dose, or, in some cases, to minimize the difference in doses between the treatment plan and the dose calculated on the current treatment anatomy. of the physical calibration phantom from the camera assembly (FALCO, Fig.6 , ¶ [0067] Using such techniques, or a combination thereof, any adjustments made to the radiotherapy beams prior to each treatment session can be based on both surface information and ultrasound-based internal anatomy, where the images are referenced in the same or related coordinate systems. As a result, the required treatment may be accurately delivered to the correct location, and at the correct angle, regardless of the time between treatments and even the location of the treatment.);
It is noted that FALCO is silent about receive coordinate data indicative of the physical calibration phantom's position and orientation with respect to the radiotherapy treatment system; determine, based on the image data and the coordinate data, an offset between the camera coordinate system and the radiotherapy treatment system's coordinate system; and compensate, based on the offset, the camera coordinate system for provision of an updated camera coordinate system as claimed.
However, FAN discloses receive coordinate data (FALCO, ¶ [0014] In an embodiment, method for determining a 3D model of a surface includes calibrating 3D reconstruction parameters for at least one reference setting of an optical system; calibrating image warping parameters for at least one secondary calibration setting, the image warping parameters adapted to control an image warping routine to warp images taken at that secondary calibration setting into warped images corresponding to images taken at the reference setting; taking an stereo image through the optical system with the optical system at a current setting; determining warping parameters from the image warping parameters for at least one secondary calibration setting of the at least one secondary calibration settings, the warping parameters for warping the stereo image taken at the current setting into a warped stereo image corresponding to a stereo image taken at the reference setting; warping the stereo image into the warped stereo image; determining three-dimensional (3D) warping parameters for warping a first image of the warped stereo image into a second image of the stereo image; and using the 3D warping parameters for determining the 3D model of the surface.) indicative of the physical calibration phantom's position and orientation (FAN, ¶ 15 … ng a 3D model of a surface includes an optical system having a plurality of settings, each setting providing a specific focal length and magnification, the optical system comprising an encoder for observing a current setting of the optical system; a memory configured to contain calibrated 3D reconstruction parameters for at least one reference setting of the optical system; the memory further configured with image warping parameters for at least one secondary calibration setting, the image warping parameters adapted to control an image warping routine to warp images taken at that secondary calibration setting into warped images corresponding to images taken at a reference setting of the at least one reference setting; a camera coupled to capture stereo images through the optical system; a processor configured with machine readable instructions in the memory, the machine readable instructions comprising instructions for determining warping parameters from the image warping parameters for at least one secondary calibration point, the warping parameters for warping the stereo image into a warped stereo image corresponding to a stereo image taken at the reference point; the memory further configured with machine readable instructions for warping the stereo image into the warped stereo image; the memory further configured with machine readable instructions for determining three-dimensional (3D) warping parameters for warping a first image of the warped stereo image into a second image of the stereo image; the memory further configured with machine readable instructions for using the 3D warping parameters for determining the 3D model of the surface.) with respect to the radiotherapy treatment system (FAN, ¶ 0115 Techniques for stereo image calibration and reconstruction based on a pinhole camera model and radial lens distortion correction are outlined here for completeness, and are used in some embodiments. A 3D point in world space (X, Y, Z) is transformed into the camera image coordinates (x, y) using a perspective projection matrix:, ( x y 1 ) = ( .alpha. x 0 C x 0 0 .alpha. y C y 0 0 0 1 0 ) .times. T .times. ( X Y Z 1 ) ( 1 ) ##EQU00001## where .alpha..sub.x and .alpha..sub.y incorporate the perspective projection from camera to sensor coordinates and the transformation from sensor to image coordinates, (C.sub.x, C.sub.y) is the image center, and T is a rigid body transformation describing the geometrical relationship of the effective optical centers between the views of the two cameras, 120, 122.); determine, based on the image data and the coordinate data, an offset between the camera coordinate system and the radiotherapy treatment system's coordinate system; and compensate, based on the offset, the camera coordinate system for provision of an updated camera coordinate system (FAN, ¶ 119 establish epipolar constraints that limit the search for correspondence points along "epipolar lines" (defined as the projection of the optical ray of one camera via the center of the other camera following a pinhole model). In addition, images are rotated so that pairs of epipolar lines are collinear and parallel to image raster lines in order to facilitate stereo matching. In an embodiment, an intensity-based correlation metric and a smoothness constraint aware used to find the correspondence points in both images of the pair. Each pair of correspondence points was is then transformed into their respective 3D camera space using the intrinsic parameters, and transformed into a common 3D space using the extrinsic parameters. Together with their respective camera centers in the common space, two optical rays were constructed with their intersection defining the 3D location of each of the correspondence point pair.2.).
Both FALCO and FAN teach systems with imaging registration, and those systems are comparable to that of the instant application. Because the two cited references are analogous to the instant application, it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains, to include in the FALCO disclosure, to determine intraoperative locations of a lesion in tissue from lesion locations determined in preoperative imaging, as taught by FAN. Such inclusion would have increased the usefulness of the system by providing more accurate predicted intraoperative brain tumor locations, and would have been consistent with the rationale of combining prior art elements according to known methods to yield predictable results to show a prima facie case of obviousness (MPEP 2143(I)(A)) under KSR International Co. v. Teleflex Inc., 127 S. Ct. 1727, 82 USPQ2d 1385, 1395-97 (2007).
Regarding claim 3, FALCO/FAN, for the same motivation of combination, further discloses the camera system of claim 2, wherein the image data is optical image data (FAN, ¶ 96, …An optical-flow method is used to determine 706 a warping of a first image of the pair to match a second image of the pair, the warping is used to derive 708 a 3-dimensional initial surface model ISM. The ISM is a 3-D model of an image, with intensities at each pixel derived from the ISM images. The ISM is coordinate remapped 710 to a common coordinate system, such as the coordinate system of presurgical MRI images (pMR). A post-resection (PIR images) stereo image pair is captured 712. Optical system settings are read from the encoders on the optical system, and these settings are interpolated into a table as previously described to determine image-warping parameters for warping the pre-resection stereo image pair to a calibrated reference setting, and the images are warped 714. An optical-flow method is used to determine 716 a warping of a first image of the pair to match a second image of the pair, the warping is used to derive 718 a 3-dimensional post-resection model PIR. The PIR is a 3-D model of an image, with a height at each pixel and intensities at each pixel derived from the PIR images. The PIR is coordinate remapped 720 to the common coordinate system. Both the ISM model and the PIR model are projected 711, 721 onto a common plane as ISM and PIR projections, respectively. An optical-flow process is then used to determine 724 a 2-dimensional image warping that will map the ISM to the PIR projection; in an alternative embodiment a 2-dimeensional warping that maps the PIR projection to the ISM projection is determined. The spatial mapping of the ISM to the plane, and the PIR model to the plane is then inverted and subtracted to determine 726 a three-dimensional displacement vector of difference between the ISM and PIR models. The 3-D displacement vector in turn is used to derive 728 a 3D difference model (delta-M), a volume of the difference model being an initial estimate of the volume of resected tissue. In an alternative embodiment, the spatial mappings of the ISM to the plane and PIR to the plane are determined at planar grid points, inverted, and subtracted to determine 728 the three-dimensional difference model.).
Regarding claim 4, FALCO/FAN, for the same motivation of combination, further discloses the camera system of claim 2, wherein the at least one processor is configured to receive the coordinate data from the radiotherapy treatment system (see FALCO, Fig. 6 ¶ 67)
Regarding claim 5, FALCO/FAN, for the same motivation of combination, further discloses the camera system of claim 2, wherein the at least one processor is configured to generate a 3D model based on the image data (FAN, as cited above, i.e. ¶ 96).
Regarding claim 6, FALCO/FAN, for the same motivation of combination, further discloses the camera system of claim 2, wherein the at least one processor is configured to output updated data (FAN, ¶ 95) indicative of the updated camera coordinate system (FALCO, ¶ 65).
Regarding claim 7, FALCO/FAN, for the same motivation of combination, further discloses the camera system according to claim 2, wherein to determine the offset comprises to compare the received image data to reference data (FAN, ¶ 95).
Regarding claim 8, FALCO/FAN, for the same motivation of combination, further discloses the camera system according to claim 2, wherein the at least one processor is configured to calculate, based on the offset, a transformation between the camera coordinate system and the second coordinate system (FAN, ¶ 119).
Regarding claim 9, FALCO/FAN, for the same motivation of combination, further discloses the camera system according to claim 8, wherein to compensate the camera coordinate system comprises to apply the transformation to the camera coordinate system (FAN,¶ 115).
Regarding claim 10, FALCO/FAN, for the same motivation of combination, further discloses the camera system according to claim 8, wherein the transformation aligns the camera coordinate system (FAN, ¶ 119) with the second coordinate system (FAN,¶ 115).
Regarding claim 11, FALCO/FAN, for the same motivation of combination, further discloses the camera system according to claim 2, wherein the at least one processor is configured to calculate, based on the offset, a transformation between the camera coordinate system and the second coordinate system (see rejection of claim 10).
Regarding claim 12, FALCO/FAN, for the same motivation of combination, further discloses the camera system of claim 2, wherein the coordinate data is derived from radiographic images taken by the radiotherapy treatment system (See FALCO, Fig. 6 ¶ 67)).
Regarding claim 13, FALCO/FAN, for the same motivation of combination, further discloses the camera system of claim 2, wherein camera is a stereoscopic camera (FAN, ¶ 14).
Regarding claim 14, FALCO/FAN, for the same motivation of combination, discloses a radiotherapy system comprising: a radiotherapy treatment system having a treatment coordinate system (see rejection of claim 2); and a camera system comprising: a camera configured to image a physical calibration phantom positioned in a camera coordinate system; and at least one processor configured to: receive image data of the physical calibration phantom from the camera assembly (see rejection of claim 2); receive coordinate data indicative of the physical calibration phantom's position and orientation with respect to the treatment coordinate system; determine, based on the image data and the coordinate data, an offset between the camera coordinate system and the treatment coordinate system (see rejection of claim 2); and compensate, based on the offset, the camera coordinate system for provision of an updated camera coordinate system (see rejection of claim 2).
Regarding claim 15, FALCO/FAN, for the same motivation of combination, further discloses the radiotherapy system of claim 14, wherein camera is a stereoscopic camera (FAN, ¶ 14)
Regarding claim 16, FALCO/FAN, for the same motivation of combination, further discloses the radiotherapy system of claim 14, wherein the image data is optical image data FAN, ¶ 96, …An optical-flow method is used to determine 706 a warping of a first image of the pair to match a second image of the pair, the warping is used to derive 708 a 3-dimensional initial surface model ISM. The ISM is a 3-D model of an image, with intensities at each pixel derived from the ISM images. The ISM is coordinate remapped 710 to a common coordinate system, such as the coordinate system of presurgical MRI images (pMR). A post-resection (PIR images) stereo image pair is captured 712. Optical system settings are read from the encoders on the optical system, and these settings are interpolated into a table as previously described to determine image-warping parameters for warping the pre-resection stereo image pair to a calibrated reference setting, and the images are warped 714. An optical-flow method is used to determine 716 a warping of a first image of the pair to match a second image of the pair, the warping is used to derive 718 a 3-dimensional post-resection model PIR. The PIR is a 3-D model of an image, with a height at each pixel and intensities at each pixel derived from the PIR images. The PIR is coordinate remapped 720 to the common coordinate system. Both the ISM model and the PIR model are projected 711, 721 onto a common plane as ISM and PIR projections, respectively. An optical-flow process is then used to determine 724 a 2-dimensional image warping that will map the ISM to the PIR projection; in an alternative embodiment a 2-dimeensional warping that maps the PIR projection to the ISM projection is determined. The spatial mapping of the ISM to the plane, and the PIR model to the plane is then inverted and subtracted to determine 726 a three-dimensional displacement vector of difference between the ISM and PIR models. The 3-D displacement vector in turn is used to derive 728 a 3D difference model (delta-M), a volume of the difference model being an initial estimate of the volume of resected tissue. In an alternative embodiment, the spatial mappings of the ISM to the plane and PIR to the plane are determined at planar grid points, inverted, and subtracted to determine 728 the three-dimensional difference model.).
Regarding claim 17, FALCO/FAN, for the same motivation of combination, further discloses the radiotherapy system of claim 14, wherein the at least one processor is configured to receive the coordinate data from the radiotherapy treatment system (see FALCO, Fig. 6 ¶ 67)
Regarding claim 18, FALCO/FAN, for the same motivation of combination, further discloses the radiotherapy system according to claim 14, wherein the at least one processor is configured to calculate, based on the offset, a transformation between the camera coordinate system and the second coordinate system (FAN, ¶ 119).
Regarding claim 19, FALCO/FAN, for the same motivation of combination, further discloses the radiotherapy system according to claim 18, wherein to compensate the camera coordinate system comprises to apply the transformation to the camera coordinate system (FAN, ¶ 115).
Regarding claim 20, FALCO/FAN, for the same motivation of combination, further discloses the radiotherapy system of claim 14, wherein the at least one processor is configured to generate a 3D model based on the image data (FAN, as cited above, i.e. ¶ 96).
Regarding claim 21, FALCO/FAN, for the same motivation of combination, further discloses the radiotherapy system of claim 14, wherein the coordinate data is derived from radiographic images taken by the radiotherapy treatment system (See FALCO, Fig. 6 ¶ 67)).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
US 11916928 B2 Communication protocols over internet protocol (IP) networks
US 11816323 B2 Automation system user interface
US 11811845 B2 Communication protocols over internet protocol (IP) networks
US 11750414 B2 Bidirectional security sensor communication for a premises security system
US 11071596 B2 Systems and methods for sensory augmentation in medical procedures
Any inquiry concerning this communication or earlier communications from the examiner should be directed to FRANK F HUANG whose telephone number is (571)272-0701. The examiner can normally be reached Monday-Friday, 8:30 am - 6:00 pm (Eastern Time), Federal Alternative First Friday Off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jay Patel can be reached at (571)272-2988. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/FRANK F HUANG/Primary Examiner, Art Unit 2485