DETAILED ACTION
Response to Amendment
Original claims 1-16 filed December 26, 2023, are pending.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on December 26, 2023, and August 21, 2024, are being considered by the examiner.
Claim Objections
Claim(s) 9 and 12 is/are objected to because of the following informalities:
In claim 9, “and” should be inserted at the end of line 24 on page 3
In claim 12, “and” should be inserted at the end of line 28 on page 4
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
the actuating device of claim 1, and
the actuating device of claim 9.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-7 and 9-15 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites generating 3D images of “a reference surface with known spatial information” and “calculating a plurality of surface equations” (first two steps).
The claim is drawn to a genus because there are many different types of reference surfaces. Some reference surfaces are planar – i.e., flat. Other references surfaces are curved – i.e., non-flat. Examples of curved reference surfaces include the surface of a sphere or the surface of a circular cylinder. Other reference surfaces may be irregular (i.e., not defined by a simple geometric primitive), yet their spatial information may be known in advance by (e.g., using a 3D scanner, a CAD model, etc.).
Examiner has reviewed the specification, but finds description of only one particular way of performing the method by using a planar reference surface. For example, par. [0024] (all citations are to the published specification, unless otherwise noted) states that “the following explanation is based on the embodiment of plane equations.” In another example, par. [0028] states that “when the reference surface 11 is a plane, if all lenses 32 and 34 are coplanar, the point clouds reconstructed from the plurality of 3D images should have the same height.” It goes on to describe that discrepancies in height (i.e.,
Δ
T
y
) are determined to “indicat[e] that lenses 32 and 34 are not coplanar, and it is necessary to adjust one of them (such as the lens 34) through the actuator 7 to align with the other (such as the lens 32).” This indicates that the described method relies on the specific properties of a planar surface to determine adjustments to a lens. Similarly, par. [0029] describes only plane equations. No other, more-general surface equations are described.
MPEP 2163, Subsection II.3.a.ii includes the following guidance for determining whether an original claim drawn to a genus satisfies the written description requirement of 35 U.S.C. 112(a):
“The Federal Circuit has explained that a specification cannot always support expansive claim language and satisfy the requirements of 35 U.S.C. 112 "merely by clearly describing one embodiment of the thing claimed." LizardTech v. Earth Resource Mapping, Inc., 424 F.3d 1336, 1346, 76 USPQ2d 1731, 1733 (Fed. Cir. 2005). The issue is whether a person skilled in the art would understand inventor to have invented, and been in possession of, the invention as broadly claimed. In LizardTech, claims to a generic method of making a seamless discrete wavelet transformation (DWT) were held invalid under 35 U.S.C. 112, first paragraph, because the specification taught only one particular method for making a seamless DWT and there was no evidence that the specification contemplated a more generic method. Id.; see also Tronzo v. Biomet, 156 F.3d at 1159, 47 USPQ2d at 1833 (Fed. Cir. 1998)(holding that the disclosure of a species in a parent application did not provide adequate written description support for claims to a genus in a child application where the specification taught against other species).”
Similar to the LizardTech example in the MPEP, the specification here describes only one particular method of multi-lens coplanar calibration using a planar surface, yet the claims cover a generic method using any type of surface. Par. [0015] of the specification does state that “In another embodiment, the reference surface 11 is a surface that can be defined by a mathematical model, such as a curved surface, but the disclosure is not limited thereto.” However, there is no further description of any method for performing calibration using a curved surface, any description of a mathematical model defining a curved surface, etc. As discussed above, the remainder of the description focuses specifically on a planar surface embodiment and relies upon assumptions (e.g., the equal-height assumption in [0028]) that are specific to a planar surface embodiment. Description of a single method that relies upon a specific planar reference surface does not adequately describe, and would not have demonstrated invention and possession of, a general method of performing calibration using any reference surface as presented in claim 1. Therefore, claim 1 is rejected for lack of adequate written description under 35 U.S.C. 112(a).
Claim 9 recites similar limitations and is also rejected for substantially the same reasons as claim 1. Claims 2-7 and 10-15 depend from claims 1 and 9, respectively, and are also rejected for substantially the same reasons. Claims 8 and 16 are not rejected because they specifically require a planar reference surface.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 1-16 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the second coordinate system" in lines 15 and 18. There is insufficient antecedent basis for this limitation in the claim.
Claim 1 previously introduces “a plurality of coordinate systems” where “each of the plurality of coordinate systems other than the first coordinate system is a second coordinate system” (lines 10 and 13-14). Accordingly, claim 10 contemplates there being multiple second coordinate systems. It is unclear which of the second coordinate systems is being referred to by recitation of the singular “the second coordinate system”.
Claims 2-8 are also indefinite at least because they include the limitations of claim 1. Claims 2 and 3 further recite “the second coordinate system” and are further indefinite for substantially the same reasons as claim 1. Claims 9-16 recite similar limitations and are also indefinite for substantially the same reasons as claims 1-8, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 6, 8-12, 14, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over ‘Fletcher’ (US 2019/0073792 A1) in view of ‘Wang’ (US 2017/0371123 A1).
Examiner notes that claims 1-4, 6, and 8 are directed to methods that are substantially the same as the methods performed by the systems of claims 9-12, 14, and 16, respectively. Fletcher in view of Wang teaches the systems of claims 9-12, 14, and 16 (see below). Accordingly, claims 1-4, 6, and 8 are also rejected under 35 U.S.C. 103 as being unpatentable over Fletcher in view of Wang for substantially the same reasons as claims 9-12, 14, and 16, respectively.
Regarding claim 9, Fletcher teaches a multi-lens coplanar calibration system (e.g., Figures 1A, 2, and 7A-B) comprising:
an object comprising a reference surface with known spatial information (e.g., [0042], Fig. 1A, real world object 145; e.g., object 145 may be a wall known to be flat – i.e., planar – or an object with a known surface shape, such as a teapot);
a three-dimensional (3D) imaging device with a plurality of lenses (e.g., Fig. 1A, combination of cameras 110 and 115), wherein the 3D imaging device is configured to shoot the reference surface of object through the plurality of lenses to generate a plurality of 3D images (e.g., [0042], [0045], Fig. 1A, both cameras generate 3D images of the reference object);
a computing device electrically connected to the 3D imaging device to obtain the plurality of 3D images, wherein the computing device is configured to perform a plurality of instructions to trigger a plurality of operations, wherein the plurality of operations comprises (e.g., Figs. 7A-B and [0048] et seq.; [0072] in particular):
calculating a plurality of surface equations according to 3D information of the plurality of 3D images (Fletcher describes two arrangements for mapping between the cameras: one using a homography (see [0093] et seq.) and the other using displacement fields (see [0097] et seq.); Both read on the claims and are mapped separately below where necessary; Homography: e.g., [0093], determination of planar surface points, homography matrix, etc.; Displacement Fields: see various equations at [0097] seq.; e.g., [0099], difference equations are used to find displacement field of the reference object’s surface);
calculating a plurality of coordinate systems corresponding to the plurality of lenses according to the plurality of surface equations (e.g., [0075], Homography: e.g., [0093]-[0094], the coordinate systems of the first image plane and the second image plane; Displacement Fields: e.g., [0098], [0100], first and second cameras’ coordinate systems), wherein each of the plurality of coordinate systems has a specific point as an origin (This is a fundamental property of the coordinate systems being used in Fletcher; For example, consider equation 3 in par. [0098]: The “specific point” is the point at which
x
,
y
and
z
are all zero), one of the plurality of coordinate systems is a first coordinate system (e.g., [0096], Fletcher maps a “first” image onto a “second” image; So, the coordinate system of the first image in Fletcher corresponds to the claimed second coordinate system, and vice versa), and each of the plurality of coordinate systems other than the first coordinate system is a second coordinate system (see note above; Note that Fletcher contemplates using additional cameras – [0044]);
calculating a calibration matrix for the second coordinate system relative to the first coordinate system according to at least the plurality of coordinate systems (Homography: e.g., [0093], homography matrix; [0096], composed mapping; Displacement Fields: e.g., [0097]-[0098], transformation matrix; e.g., [0102], composed displacement field, where the array of values making up the field are a matrix); and
an actuating device (Note the interpretation under 35 U.S.C. 112(f) – see Claim Interpretation above; Corresponding structure is described in [0021] of the published specification) electrically connected to the computing device, wherein the actuating device is configured to adjust one of the plurality of lenses corresponding to the second coordinate system according to the calibration matrix (see Note Regarding Actuating Device below).
Note Regarding Actuating Device. Fletcher teaches techniques that calculate an extrinsic calibration matrix that maps a second coordinate system of a 3D imaging device onto a first coordinate system of another 3D imaging device (see mapping above and Fig. 3). Fletcher uses this calibration matrix to account for discrepancies in camera positions so that, for example, images can be stitched together with improved alignment (e.g., [0128]).
Fletcher does not teach an actuating device electrically connected to the computing device, wherein the actuating device is configured to adjust one of the plurality of lenses corresponding to the second coordinate system according to the calibration matrix.
However, Wang does teach an actuating device (e.g., Fig. 1, lens adjusting mechanism 115) electrically connected to a computing device (e.g., Fig. 1, image processing device 120), wherein the actuating device is configured to adjust one of the plurality of lenses corresponding to the second coordinate system (e.g., Fig. 2, S210; Fig. 3A, S312; Fig. 4A, S412) according to the calibration matrix (e.g., Fig. 2, S208; Fig. 3A, S308; Fig. 4A, S408; first and second calibration parameter sets – i.e., the mapping between first and second coordinate systems – is used for the actuation; also see, e.g., [0033]).
Fletcher teaches that its cameras are ideally “geometrically related by a translation in one axis” and “rigidly coupled to one another” ([0042], Fig. 1A), but does not describe any mechanism for ensuring this ideal alignment. For example, Fig. 1A illustrates cameras 110 and 115 spaced apart without any mechanism for ensuring their proper alignment. However, Wang recognizes that it is difficult when manufacturing such a device to achieve this ideal alignment because “the optical axes and the lens centers are all invisible” ([0004]). Wang solves this problem by using calibration parameters that map from one camera to another to guide actuation of one of the cameras into ideal alignment (e.g., Figs. 3-4). As discussed above, the calibration matrix of Fletcher maps from one camera to another. Accordingly, the actuation device of Wang could be used to align the cameras of Fletcher according to the calibration matrix to ensure that they have the ideal alignment to one another.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify the system of Fletcher with the actuation device of Wang in order to improve the system with the reasonable expectation that this would result in a system that could ensure its cameras were fixed in an ideal alignment such that they were related by a translation in one axis. This technique for improving the Fletcher was within the ordinary ability of one of ordinary skill in the art based on the teachings of Wang.
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Fletcher and Wang to obtain the invention as specified in claim 9.
Regarding claim 10, Fletcher in view of Wang teaches the system of claim 9, and Fletcher further teaches that, in the plurality of operations, calculating the calibration matrix for each second coordinate system relative to the first coordinate system at least according to the plurality of coordinate systems by the computing device comprises:
calculating a plurality of transformation matrices according to a global coordinate system and the plurality of coordinate systems, wherein each of the plurality of transformation matrices is configured for a transformation between corresponding one of the plurality of coordinate systems and the global coordinate system (e.g., Fig. 3, steps 330 and 335; The coordinate system of the orthographic view is a global coordinate system); and
calculating the calibration matrix according to the transformation matrix of the first coordinate system and the transformation matrix of the second coordinate system (e.g., Fig. 3, steps 340 and 350, the transformation matrices describing mappings from the first and second cameras to the orthographic view are composed into the calibration matrix mapping between the first and second views).
Regarding claim 11, Fletcher in view of Wang teaches the system of claim 10, and Fletcher further teaches that in the plurality of operations, calculating the calibration matrix according to the transformation matrix of the first coordinate system and the transformation matrix of the second coordinate system comprises:
calculating an inverse matrix according to the transformation matrix of the first coordinate system (e.g., Fig. 3, step 340); and
calculating the calibration matrix of the second coordinate system relative to the first coordinate system according to the inverse matrix and the transformation matrix of the second coordinate system (e.g., Fig. 3, step 350).
Regarding claim 12, Fletcher in view of Wang teaches the system of claim 9, and Fletcher further teaches that the 3D information is point cloud (e.g., [0099], the real-world positions of the pixels form a point cloud); and in the plurality of operations, calculating the plurality of surface equations according to the 3D information of the plurality of 3D images by the computing device comprises:
generating a plurality of virtual surfaces according to the point cloud of the plurality of 3D images (Homography: e.g., [0093], object plane surfaces in each image; Displacement Fields: e.g., [0099], set of points on object surface in each image);
calculating the plurality of surface equations corresponding to the plurality of virtual surfaces (Homography: e.g., [0093], determination of planar surface points, homography matrix, etc. are all based on the virtual planar surfaces in each image; Displacement Fields: e.g., [0099], surface displacement equations are calculated according to displacement of the virtual surfaces).
Regarding claim 14, Fletcher in view of Wang teaches the system of claim 10, and Fletcher further teaches that the plurality of operations further comprises: before calculating the plurality of surface equations according to the 3D information of the plurality of 3D images by the computing device, setting a region of interest for each of the plurality of 3D images by the computing device (e.g., [0093], testing to see if region covered by images – see Fig. 1A – is of sufficiently constant depth, and thus is of interest for mapping via a homography).
Regarding claim 16, Fletcher in view of Wang teaches the system of claim 9, and Fletcher further teaches that the reference surface is a plane (e.g., [0042], “The real-world object 145 may be substantially 2D (two-dimensional, i.e. flat, such as a wall”), and the plane includes a plurality of positions with same height (e.g., [0093] approximately constant depth information).
Allowable Subject Matter
Claims 1-16 are rejected under 35 U.S.C. 112(b). Claims 1-7 and 9-15 are rejected under 35 U.S.C. 112(a). Claims 1-4, 6, 8-12, 14, and 16 are rejected under 35 U.S.C. 103. Independent claims 1 and 9 could be placed into condition for allowance by amendments that:
Resolve the issues of indefiniteness noted above,
Incorporate all of the limitations of claims 8 and 16, and
Incorporate all of the limitations of claims 5 and 13 and/or claims 7 and 15 (and any intervening claims).
Conclusion
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
‘Herrera’ (“Joint Depth and Color Camera Calibration with Distortion Correction,” 2012)
Performs extrinsic calibration of depth sensor and color cameras using plane equation – Section 3.2
‘Fernández-Moral’ (“Extrinsic calibration of a set of range cameras in 5 seconds without pattern,” 2014)
Describes techniques for calibrating multiple depth cameras using a planar reference surface – e.g., Figs. 1 and 3, Sec. V.
‘Fu’ (US 2018/0308254 A1)
Teaches examples of calibrating image sensors using sphere and cube reference surfaces
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GEOFFREY E SUMMERS whose telephone number is (571)272-9915. The examiner can normally be reached Monday-Friday, 7:00 AM to 3:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GEOFFREY E SUMMERS/Examiner, Art Unit 2669