DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Regarding the amendment filed 18 September 2025, line 10 of claim 6 includes a minor discrepancy: The term “pixel” in line 10 is both underlined and struck through and the comma following the term “pixle” was not carried over. Examiner assumes that “” was not meant to be included in the strikethrough of line 10 and that the comma was meant to be included in the strikethrough. In a subsequent response, Applicant should treat the claims received 18 September, 2025 as not including “” and including the carried-over comma in the strikethrough of line 10.
On page 10 of Remarks, filed 18 September 2025, Applicant indicates a further request from the USPTO to retrieve the priority document was made on 30 July, 2025. Examiner notes that the priority document retrieval was unsuccessful, as indicated in the report mailed 4 August, 2025.
Applicant’s arguments, see Remarks at page 10, filed 18 September 2025, with respect to the objection to the abstract have been fully considered and are persuasive. The objection has been withdrawn.
Applicant’s arguments, see Remarks at page 10, filed 18 September 2025, with respect to the objections of claims 2-6 have been fully considered and are persuasive. The objections have been withdrawn.
Applicant’s arguments, see Remarks at page 11, filed 18 September 2025, with respect to the rejections of claims 1-16 under 35 U.S.C. 112(b) have been fully considered and are persuasive in part. Applicant argues that the amendment provided 18 September 2025 necessitates withdrawal of each rejection. Examiner respectfully disagrees. Claim 7 remains rejected under 35 U.S.C. 112(b) because it recites “the area of focus” in line 18, however there is no previously-recited area of focus. Regarding claim 15, the amendment did not clarify the antecedent basis of “the movement of the tracked target in the 3D space” and the addition of “location information of the marker” necessitated a new ground of rejection. It is unclear if “the movement” corresponds to “location information of the marker”, “continuously obtained position information”, a combination thereof, or something else. Claim 16 remains rejected under 35 U.S.C. 112(b) because when claim 16 is read as a whole with claim 15, the antecedent basis of “the marker” in claim 16 is unclear because claims 15 and 16 each recite “a flexible marker” making it unclear which marker corresponds to “the marker” as recited in claim 16. The remaining rejections under 35 U.S.C. 112(b) from the previous Office action are withdrawn, as necessitated by amendment.
Applicant’s arguments, see Remarks at pages 11-12, filed 18 September 2025, with respect to the rejection of claims 1-4 and 8-14 under 35 U.S.C. 103 have been fully considered but are not persuasive.
On pages 11-12, Applicant argues that Fisher’s disclosure of “rigid body motion” and the examiner’s characterization of Fisher’s marker does not describe “a flexible marker” as recited in claims 1 and 15. Fisher is not relied upon to teach a “flexible marker”, as Applicant’s amendment necessitated a new ground of rejection that does not rely on Fisher to teach a flexible marker. Accordingly, Applicant’s arguments pertaining to Fisher’s alleged deficiencies regarding the disclosed marker being flexible or not are moot. Regarding the concept of rigid body motion, rigid body motion tracking means the subject being tracked is treated/assumed as a rigid body (i.e., does not deform) for the purpose of motion tracking to ensure accurate analysis of the subject’s motion over time, but that does not imply that the subject is incapable of flexing or is not flexible. In rigid body motion, the subject is merely assumed to not flex during the movement being tracked so that the subject’s relative changes in translation and rotation can be determined.
Regarding the scope of the pending claims, the claims do not require a marker to flex or deform during motion tracking. Rather, the marker is merely required to be flexible (i.e., capable of flexing) and directly attached to a patient’s body. A flexible marker, such as an ArUco marker printed on a piece of paper or a flexible sheet of another material like plastic, can be attached to a patient and still not substantially flex during movement of the patient. And if it does flex during tracking, as long as it is visible to the camera, feature points of the marker can still be extracted from the two-dimensional images of it acquired by the camera. Furthermore, every physical object can flex or deform to at least some degree. Still further, through plane fitting, a well-known practice in the area of augmented reality and camera-based target tracking, an initially planar flexible marker (e.g., a printed Aruco marker) that deforms/flexes or is bent around something supporting the marker (e.g., a patient’s body, a motorized testing rig) can still be localized by using plane fitting to approximate its pose because it is known a priori that the pattern of the marker is two-dimensional but will appear distorted/transformed in the acquired image thereof because the surface of the marker will rarely, if ever, be perfectly orthogonal to the central axis of the camera’s view of the marker as it moves about. Even if the surface of the marker was perfectly orthogonal, factors like lens distortion will still affect how the surface appears to the camera image to some degree. In practice, this is routinely handled through coordinate transformations.
On page 12, Applicant states, “The marker used in the Fisher method also contains three colors, including red, blue and green. See for example the legend for Figure 2, which states ‘The marker's coordinate system is marked in perpendicular green, blue, and red lines at the center of the marker.’ The markers of the presently claimed method require only two contrasting colors.” Examiner respectfully disagrees. The marker is a sheet of binary-colored codes (i.e., black and white ArUco Markers) as shown in Fig. 1. The colors in Fig. 2 of Fisher referenced by Applicant are visual aids for the purpose of explaining how the marker’s coordinate system relates to the camera coordinate system (i.e., pose) as understood by the tracking procedure and are not actually present on the physical marker. In other words, they represent what the camera system ‘sees’, not what is actually on the marker in reality. Additionally, the claims only require two contrasting colors at a minimum because the transitional phrase “comprises” in the preambles means that prior art disclosing more than two contrasting colors is not excluded.
Applicant’s arguments on page 12 regarding Lee and Jin have been fully considered but are moot because Applicant merely points out differences between the geometric shapes of the specific markers used by Lee and Jin without explaining how those differences have any bearing on the combination of references in a rejection under 35 U.S.C. 103. Lee and Jin are not relied upon to teach the “flexible marker”. Still, the use of two-dimensional binary-colored codes for camera-based object tracking is a well-developed field. There are numerous variations of different marker shapes and sizes that rely on the same fundamental principles of feature detection and coordinate system transformation. One of ordinary skill in the art would not find it difficult to combine such principles as applied to different markers.
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in China on May 30,2022. However, Applicant has not filed a certified copy of Chinese Pat. Appl. No. 202210597366.6 as required by 37 CFR 1.55.
Claim Objections
Claim 4 is objected to because of the following informality: “an corner point” in line 5 should be changed to “a corner point” for clarity. Appropriate correction is required.
Claim 6 is objected to because of the following informalities: “key region of interest” in line 9 should be changed to “key area of interest” to be consistent with “key area of interest” in the preamble of claim 6 and “the key area of interest” in line 2 should be changed to “the key area of interest of each square-shaped area” to clarify which area is being referenced from claim 5 (i.e., “key areas of interest of each square-shaped area” in line 9); and “calculating the 3D coordinate of corner points” in line 14 should be changed to “calculating the 3D coordinates of corner points” for clarity. Appropriate correction is required.
Claim 7 is objected to because of the following informality: “the key area of interest” in line 2 should be changed to “the key area of interest of each square-shaped area” to clarify which area is being referenced from claim 5 (i.e., “key areas of interest of each square-shaped area” in line 9). Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“visible light image acquisition module for acquiring a visible light image of a flexible marker” in claims 15-16;
“depth image acquisition module for acquiring a depth image of the marker” in claims 15-16;
“image processing module for detecting the two-dimensional code on the visible light image” in claims 15-16; and
“an execution module for generating a motion instruction to the robot” in claims 15-16.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed functions, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed functions); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed functions so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7, 15, and 16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 7 recites “the area of focus” in line 18, however there is no previously-recited area of focus. Therefore, “the area of focus” lacks antecedent basis. For purposes of applying prior art, the examiner interprets “the area of focus” as “[[the]] an area of focus”.
Claim 15 recites, in part, “the movement of the tracked target in the 3D space”, which lacks antecedent basis because there is no otherwise-recited “movement”. It is unclear if “the movement” corresponds to “location information of the marker”, “continuously obtained position information”, a combination thereof, or something else. For purposes of applying prior art, the examiner interprets claim 15 to correspond to claim 1 (i.e., position/location information of the marker is tracked) with the inclusion of the structural components that invoke 35 U.S.C. 112(f), such that “position information” of the marker is its three-dimensional coordinates and “location information” is the “position information” over time. Dependent claim 16 is rejected for inheriting and not curing the deficiencies of claim 15.
Claim 16 recites “a flexible marker” and then “the marker”. However, claim 15 also recites “a flexible marker”. Thus, the antecedent basis of “the marker” in claim 16 is unclear. For purposes of applying prior art, the examiner assumes each “marker” is the same marker and that claim 16 further limits claim 15 by reciting a system including the robot and flexible marker of claim 15.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed inventions absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4 and 8-16 are rejected under 35 U.S.C. 103 as being unpatentable over Optimizing Marker Design to Improve Precision of Optical Camera-Based Rigid-Body motion Tracking and Correction in Medical Imaging to Fisher et al. (hereinafter “Fisher”) in view of Joint Scene and Object Tracking for Cost-Effective Augmented Reality Guided Patient Positioning in Radiation Therapy to Sarmadi et al. (hereinafter “Sarmadi”), in view of Sensor Fusion for Fiducial Tags: Highly Robust Pose Estimation from Single Frame RGBD to Jin et al. (hereinafter “Jin”), and in further view of U.S. Pat. Appl. Pub. No. 2014/0275760 to Lee et al. (hereinafter “Lee”).
Regarding claim 1, Fisher teaches a target tracking method for a surgical robot, wherein the method comprises:
obtaining a visible light image (Fisher, pg. 2, section C, “The main challenge of a 2D marker is that the full marker must be visible to the camera to estimate its pose. Therefore, to handle this limitation, we incorporated ArUco markers [11] in the checkerboard, as shown in Fig 2B.”) of a marker (Fisher, Fig. 3 shows the marker’s three-dimensional motion tracking data derived from the two-dimensional images of the marker, where Z corresponds to depth.) attached on a surface of a tracked target (Fisher, Fig. 1, “The marker is mounted to [a] 4 [degree] of freedom stage”; section II.A: patient’s head), the marker being a tracked target and having an upper surface (the surface where the codes are visible to a camera), wherein the marker is provided with a checkerboard pattern formed by adjacent square-shaped areas having a first or second contrasting color on the upper surface of the marker (Fisher, Fig. 2B shows an upper surface of the marker. The full pattern includes contrasting black and white squares. Black and white are both considered colors. Each white square containing a unique code is an adjacent square-shaped area.), wherein each of the square-shaped areas (Fisher, Fig. 2B, The white squares include individual two-dimensional codes.) includes a two-dimensional code having a pattern that serves as an identifier (ID) of the two-dimensional code (each ArUco marker is a unique two-dimensional code), and wherein each of the two-dimensional codes is arranged inside one of the square-shaped areas of the checkerboard (Fisher, pg. 2, section C, “These ArUco Markers serve as unique barcodes with known relative position.”; As shown in Fig. 2B, each white square contains a two-dimensional code inside the white square .); and
carrying out two-dimensional code detection on the visible light image (Fisher, pg. 2, Fig. 2B, “Each ArUco Marker has its unique ID marked in green”), and obtaining two-dimensional (2D) coordinates of four corner points of each square-shaped area of the marker (Fisher, Fig. 2B and Fig. 3; The marker is tracked in three-dimensional space as shown in Fig. 3 and relative positions of detected key points and codes from the two-dimensional camera images are used to estimate the marker’s pose as described in section I.C. For a plurality of the white squares of the 3D-encoded marker, each square has four corners detected as key points. The marker’s pose is determined from the detected key points, which are two-dimensional (x,y) corner points in the camera plane.), wherein the ID of each square-shaped area is within the four corner points of each square-shaped area of the marker (Fisher, pg. 2, Fig. 2B; Each white square is bounded by 4 detected corner points, where the black-and-white code is within the bounds of the four corner points.).
Fisher does not teach that which is explicitly taught by Sarmadi.
Sarmadi teaches obtaining a visible light image and a depth image (Sarmadi, pg. 8, section 3.4, “depth map and … RGB image”; pg. 10, section 4.2: RGB-D video sequence) of a flexible marker (Sarmadi, pg. 2, “a set of printed markers”) directly attached on a surface of a patient’s body (Sarmadi, Figure 1, “ArUco markers are attached to the body for tracking, while another set of ArUco markers are placed on the environment to track the camera pose in the environment.”).
Fisher discloses a fiducial marker tracking system for medical imaging applications where the marker is designed to be attached to a patient’s body (e.g., head). Thus, Fisher shows that it was known in the art before the effective filing date of the claimed invention to use fiducial tags attached to a patient’s body for medical imaging applications, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, accurately tracking non-intrusive markers attached to a patient’s body. Sarmadi discloses a fiducial marker tracking system for medical imaging applications where the markers are flexible and directly attached to a patient’s body. Thus, Sarmadi shows that it was known in the art before the effective filing date of the claimed invention to use flexible fiducial tags for medical imaging applications that are directly attached to the patient’s body, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, accurately tracking non-intrusive markers attached to a patient’s body.
A person of ordinary skill in the art would have been motivated to modify the 3D-encoded marker and camera system disclosed by Fisher to construct the marker as a flexible marker directly attached to a three-dimensional surface of a patient and track the attached marker using an RGBD video camera system as disclosed by Sarmadi to thereby track the patient’s movement via the marker using visible and depth image data to localize the marker. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of placing the marker as close as possible to the patient’s body to thereby more precisely track the pose of the patient’s body.
Fisher in view of Sarmadi does not teach that which is explicitly taught by Jin.
Jin teaches according to the depth image (Jin, pg. 5772, section IV.A “depth image”; Jin, pg. 5773, section IV.A, “our depth sensor (time of flight sensor from Kinect V2)”), 2D coordinates of the four corner points of each square-shaped area of the marker (Jin, pg. 5772, section IV.A, “The rectangular patch of points in the depth image bounded by the approximated corner pixels y = [y1, y2, y3,y4] contains the range information of all the points on the tag”) and the IDs of each square-shaped are obtained (Jin, pg. 5772, section IV, “decoding the tag”, “First, we find the plane in SO(3) containing the tag using depth data and detected corners. Secondly, an approximate initial pose is computed using the depth plane. Finally, the method refines the initial pose using the RGB data by minimizing the reprojection error within a constrained space”), thereby obtaining three-dimensional (3D) coordinates of the four corner points of each of the square-shaped areas on the marker (Jin, pg. 5772, section IV, “The process of detecting and decoding the tag is identical to previous fiducial tag systems. After the tag corners are detected, they are treated as approximated locations of the true corners. Using the corners, the method implicitly evaluates the depth data and RGB data as two separate observations and fuse them to minimize the error in 2D and 3D space.”); and
according to the 3D coordinates of the four corner points of each of the square-shaped areas, obtaining position information of the tracked target in 3D space (Jin, pg. 5773, section IV.B, “An alternative is to use all 4 detected corners as 4 pairs of point correspondences for the optimization. We projected the detected corners onto [the depth plane] to get the [four corner coordinates] in the robot sensory frame. The [four] corner coordinates in the tag frame can be easily calculated since the tag is a square plane. We define the center of the tag as the origin, and the coordinates are simply the location of the corners on a Cartesian plane. Given these two sets of 3D point correspondences, the pose can be computed as a rigid body transformation estimation.”).
Fisher in view of Sarmadi is analogous to the claimed invention for the same reasons provided above. Jin discloses plane fitting on depth image data for tasks where nonintrusive markers are tracked using a commercially-available RGBD device. Thus, Jin shows that it was known in the art before the effective filing date of the claimed invention to use plane fitting with nonintrusive markers to determine their pose relative to an RGBD camera, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, accurately tracking non-intrusive markers attached to a patient’s body.
A person of ordinary skill in the art would have been motivated to combine the RGBD camera system and use of plane-fitting as disclosed by Jin with the tracking algorithm of Fisher in view of Sarmadi to estimate the marker’s position according to the depth image of the RGBD camera system to thereby use the depth image from a low-cost, off-the-shelf RGBD camera system as disclosed by Jin to fit a plane to the marker area attached to the patient and recover the pose of the marker relative to the camera system. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of accurately tracking the marker in the presence of variations in its three-dimensional surface.
Fisher in view of Sarmadi and in further view of Jin does not teach that which is explicitly taught by Lee.
Lee teaches providing the position information to a surgical robot (Lee, par. [0104], “the disclosure herein has provided example embodiments of a surgical robot and control method to control the surgical robot”), wherein the position information is used to track the tracked target (marker) during a surgical procedure (Lee, par. [0091], “the augmented reality image generator 430 receives a real image captured by the camera 410, detects a plurality of markers, estimates that the camera 410 faces the abdomen of the patient P at the center of the patient P in a state of being spaced apart from the patient P using position information of each marker in the real image, and overlays a virtual image of the corresponding region received from the imaging system 300 over the real image.”; par. [0063], “Examples of the surgical tool 230 may include a skin holder, a suction line, a scalpel, scissors, a grasper, a surgical needle, a needle holder, a stapler, a cutting blade, and the like, without being limited thereto.”).
Fisher in view of Sarmadi and in further view of Jin is analogous to the claimed invention for the same reasons provided above. Lee discloses a surgical robot that uses markers attached to a patient’s body to determine positional information for guiding the surgical robot. Thus, Lee shows that it was known in the art before the effective filing date of the claimed invention to use fiducial tags for medical applications including surgical procedures, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, accurately tracking non-intrusive markers attached to a patient’s body.
A person of ordinary skill in the art would have been motivated to use the flexible marker disclosed by Fisher in view of Sarmadi and in further view of Jin to guide a surgical robot as disclosed by Lee during a surgical procedure using positional information obtained from the marker directly attached to the patient’s body. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of using the marker in surgeries in a nonintrusive manner, thereby minimizing any discomfort felt by the patient.
Regarding claim 2, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, wherein the two-dimensional code detection comprises:
for the pattern of each two-dimensional code inside a square-shaped area of the marker, matching the two-dimensional code to the visible light image (Fisher, pg. 2, section II.C, “identifying the barcodes”; Before a code from a white square can be identified, it must first be detected, which matches the location of a two-dimensional code to its position in the image.), wherein a similarity between the two-dimensional code of the marker and a detected pattern of a two-dimensional code inside a square-shaped area in the visible light image is obtained (Fisher, FIG. 2B; Identifying an individual ArUco marker means the two-dimensional code of the marker and the corresponding pattern depicted in the image are the same, and therefore, similar to one another.); and
determining whether a two-dimensional code of a selected square-shaped area of the marker matching a two-dimensional code in the visible light image is detected according to the similarity (Fisher, Fig. 3; section I.B, “For each frame estimate the marker’s pose relative to the camera”, “Estimate motion between frames at times 0 and t”, and “using two sets of poses … estimate the transformation between them”; Successfully tracking the pose of the marker from one pose to the next means that a two-dimensional code pattern of a selected square of the marker containing an ArUco marker was determined to be similar to the corresponding image features acquired by the camera.), wherein if the pattern of a two-dimensional code of the marker matching a corresponding two-dimensional code in the visible light image is detected according to the similarity (Fisher - Successfully tracking the pose of the marker.), the 2D coordinates of the four corner points of the selected square-shaped area and the ID of the detected two-dimensional code are obtained (Fisher, section II.C, “When the algorithm detects the key points and barcodes, it knows each key point's relative position by identifying the barcodes adjacent to it, and thus can estimate the marker’s pose using only a subset of the key points.”; Pose is determined after a prerequisite step of detecting corner/key points and ID is successful. Tracking the pose of the marker means the prerequisite step has occurred, which means the 2D coordinates and ID have been obtained.).
Regarding claim 3, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, wherein before obtaining the 3D coordinates of the corner points on the marker according to the depth image, the 2D coordinates of the corner points of the square-shaped areas containing the two-dimensional codes and the IDs of the two-dimensional codes, the method further comprises:
obtaining an actual position distribution of the two-dimensional codes on the marker according to the 2D coordinates of the corner points of the square-shaped areas containing the two-dimensional codes and the IDs of the two-dimensional codes (Fisher, Fig. 2B shows annotated labels of all the detected IDs and key point positions, thereby exhibiting a distribution of positions. Multiple camera poses/acquisitions take place. Thus, before 3D or 2D corner points or IDs are obtained in a subsequent camera pose/acquisition, 3D or 2D corner points or IDs are obtained in a prior camera pose/acquisition);
comparing a standard position distribution and the actual position distribution of the two-dimensional codes on the marker, and verifying the 2D coordinates of the corner points of the square-shaped areas containing the two-dimensional codes and the IDs of the two-dimensional codes (Fisher, pg. 2, section II.C, “When the algorithm detects the key points and barcodes, it knows each key point's relative position by identifying the barcodes adjacent to it, and thus can estimate the marker’s pose using only a subset of the key points.”; The pose (position and orientation) of the marker is determined by the relative positions of the actual positions of the two-dimensional codes, thereby determining the pose of the marker according to a standard/expected relationship between the two-dimensional codes.); and
discarding or adjusting the 2D coordinates of the corner points of the square-shaped areas having corner points with 2D coordinates that deviate by more than a predetermined amount (Fisher, pg. 2, section II.C, “When the algorithm detects the key points and barcodes, it knows each key point's relative position by identifying the barcodes adjacent to it, and thus can estimate the marker’s pose using only a subset of the key points.”; If the algorithm is aware of every detected key point and uses a subset of them to determine marker pose, then the subset is verified and the use of the subset is pre-programmed or predetermined.).
Regarding claim 4, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, wherein:
according to the 2D coordinates of the corner points of the square-shaped areas containing the two-dimensional codes and the IDs of the two-dimensional codes, key areas of interest in the visible light image are obtained, wherein each key area of interest corresponds to a corner point of a square-shaped area (Fisher, pg. 2, section II.C, “key points.” Key points delineate the square-shaped areas of the checkerboard pattern containing ArUco markers, thereby providing key areas of interest corresponding to the locations of the ArUco markers.); and
for each key area of interest, the 3D coordinates of the corresponding corner point are obtained according to the key area of interest and the depth image (Jin, pg. 5772, section IV, “The process of detecting and decoding the tag is identical to previous fiducial tag systems. After the tag corners are detected, they are treated as approximated locations of the true corners. Using the corners, the method implicitly evaluates the depth data and RGB data as two separate observations and fuse them to minimize the error in 2D and 3D space.”).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 8, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, wherein the marker comprises a flexible planar substrate (Sarmadi, pg. 2, “a set of printed markers”; Figure 1, “ArUco markers are attached to the body for tracking, while another set of ArUco markers are placed on the environment to track the camera pose in the environment.”; Figure 1 shows the printed markers, which are printed onto flexible planar substrates. The substrates are flexible because they conform to the surface they are disposed on.).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 9, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, wherein the contrasting colors are black and white (Fisher, Fig. 2B shows an upper surface of the marker. The full patterns includes contrasting black and white squares. Black and white are both considered colors.).
Regarding claim 10, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, further comprising the step of moving a robotic arm of the surgical robot (Lee, par. [0104], “the robot and augmented reality image display system may be utilized to perform operations in any confined space or enclosure in which an operator may need to perform controlled movements using an instrument attached to a robot arm”).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 11, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, further comprising the step of moving a surgical tool of the surgical robot (Lee, par. [0063], “Examples of the surgical tool 230 may include a skin holder, a suction line, a scalpel, scissors, a grasper, a surgical needle, a needle holder, a stapler, a cutting blade, and the like, without being limited thereto.”).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 12, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 11, wherein the surgical tool is a drill guide, a drill, a puncture needle (Lee, par. [0063], “a surgical needle”), scissors (Lee, par. [0063], “scissors”), a grasper (Lee, par. [0063], “a grasper”), or a needle holder (Lee, par. [0063], “a needle holder”).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 13, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, further comprising the step of performing a surgical operation with the surgical robot (Lee, par. [0104], “the robot and augmented reality image display system may be utilized to perform operations in any confined space or enclosure in which an operator may need to perform controlled movements using an instrument attached to a robot arm, so as to avoid or to prevent injuries to bodies or objects, that may be located or disposed within the space or enclosure, due to imprecise movements of the robot”).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 14, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 1, wherein the surgical operation is performing a drilling operation, performing a cutting operation (Lee, par. [0063], “Examples of the surgical tool 230 may include a skin holder, a suction line, a scalpel, scissors”), or performing a grasping operation (Lee, par. [0063], “Examples of the surgical tool 230 may include … a grasper”).
The rationale for obviousness is the same as provided for claim 1.
Claim 15 corresponds to claim 1 by reciting a robot that implements functions corresponding to the method of claim 1, primarily differing by comprising a visible light image acquisition module, a depth image acquisition module, an image processing module, and an execution module.
Fisher in view of Sarmadi, in view of Jin, and in further view of Lee, as applied in claim 1, teaches a visible light image acquisition module (Fisher, pg. 2, section C, “camera”), a depth image acquisition module (Jin, pg. 5773, section IV.A, “our depth sensor (time of flight sensor from Kinect V2)”), an image processing module (Fisher, pg. 1, Fig. 1, “computer”), and an execution module (Fisher, pg. 1, Fig. 1, “computer”; Computers include a processor to execute program instructions.).
The rationale for obviousness is the same as provided for claim 1
Regarding claim 16, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches a target tracking system, wherein the system comprises:
a flexible marker for attachment to a surface of an object to be tracked (Sarmadi, pg. 2, “a set of printed markers”), wherein the marker is provided with a black and white checkerboard pattern (Fisher, Fig. 2B shows an upper surface of the marker. The full pattern includes contrasting black and white squares. Black and white are both considered colors.), and two-dimensional codes are arranged inside square-shaped areas of the checkerboard pattern (Fisher, Fig. 2B); and
the robot of claim 15.
The rationale for obviousness is the same as provided for claim 1.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Fisher in view of Sarmadi, in view of Jin, in view of Lee, and in further view of U.S. Pat. Appl. Pub. No. 20180071032 to de Almeida Barreto (hereinafter “Barreto”).
Regarding claim 5, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee teaches the target tracking method according to claim 4, further comprising:
performing detection of the four corner points of each square-shaped area according to the two-dimensional code IDs (Fisher, pg. 2, section II.C, “key points.”; Key points are detected according to their positions relative to the ArUco markers.),
using the 2D coordinates of eight corner points of two adjacent square-shaped areas (Fisher, Fig. 2B, The checkerboard pattern is comprised of a 10 x 10 alternating pattern of black and white square shapes with the two-dimensional codes located on the white shapes. Each two-dimensional code has four key points at its corners. The center of the checkerboard pattern includes a 3 x 3 area with four adjacent two-dimensional codes adjacent to the center two-dimensional code. Thus, there are 2 pairs of two-dimensional codes, each having eight corners.), and obtaining key areas of interest of each square-shaped area on the visible light image according to preset areas of the square-shaped areas on a standard arrangement of the marker (Fisher, section II.C; The layout of the individual ArUco markers is known by the algorithm so that the relative positions of the key points can be determined.), wherein each preset area is a square area with a corner of a square-shaped area as the center and corners of adjacent square-shaped areas comprising two-dimensional codes as the diagonal vertices (Fisher, Fig. 2B, The center square area has four corners. At each diagonal corner is a two-dimensional code having four corners.), but does not teach that which is explicitly taught by Barreto.
Barreto teaches for each corner point detected, calculating a homography transformation matrix from a standard image of the marker (Barreto, par. [0063], “HC can be estimated from image information”; pars. [0090]-[0091], “the visual marker can have different topological configurations but, for the sake of simplicity and without compromising generality, it will be assumed that the visual marker is a planar surface with a known pattern. This planar pattern should be such that it has a local system of coordinates, it is amenable to be detected and uniquely identified from its image projection”; The image of the marker before projection to the camera coordinate system is a standard or typical camera image.) to the visible light image of the marker (Barreto, par. [0063], “If the marker is planar, then its projection is described by an homography HC, that maps plane points into image points, and encodes the relative rotation RC and translation tC between marker and camera reference frames. Thus, and since HC can be estimated from image information, it is possible to use this homography relation to determine at every frame time instant the 4×4 matrix C that transforms world coordinates into camera coordinates (FIG. 1A).”).
Fisher in view of Sarmadi, in view of Jin, and in further view of Lee discloses a surgical robot that tracks a checkerboard pattern. Thus, Fisher in view of Sarmadi, in view of Jin, and in further view of Lee shows that it was known in the art before the effective filing date of the claimed invention to use a checkerboard pattern with a surgical robot, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, accurately tracking non-intrusive markers attached to a patient’s body. Barreto discloses methods of computer-aided surgery that use a checkerboard pattern attached to a patient’s body and calculating a homography matrix to determine positional information. Thus, Barreto shows that it was known in the art before the effective filing date of the claimed invention to calculate a homography matrix to determine the correspondence between a camera coordinate system and a world coordinate system, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, accurately tracking non-intrusive markers attached to a patient’s body.
A person of ordinary skill in the art would have been motivated to calculate a homography matrix between world and camera coordinate systems as disclosed by Barreto using the camera and marker images obtained by the surgical robot disclosed by Fisher in view of Sarmadi, in view of Jin, and in further view of Lee, to thereby determine the position of the marker attached to the patient using key points of the marker and each two-dimensional code detected. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of accurately determining position of the surgical robot relative to the patient.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Fisher in view of Sarmadi, in view of Jin, in view of Lee, in view of Barreto, and in further view of the online forum discussion Find center between 4 points from the website discussions.unity.com (hereinafter “Unity”).
Regarding claim 6, Fisher in view of Sarmadi, in view of Jin, in view of Lee, and in further view of Barreto teaches the target tracking method according to claim 5, wherein according to the key area of interest and the depth image, the method comprises calculating the corresponding square-shaped area’s 3D coordinates as follows:
calculating the 3D coordinates (x
i
3
d
, y
i
3
d
, z
i
3
d
) of each pixel in a focus area with the 2D coordinate of an i-th pixel in the depth image and the 2D coordinate of an i-th pixel in the visible light image, where i ∈ (1, … , N) indicates an i-th pixel among N pixel