DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . The Amendment filed 9 December 2025 (hereinafter “the Amendment”) has been entered and considered. Claims 1, 6, and 10-11 have been amended. Claims 1-13 are all the claims pending in the application. Claims 1-9 are rejected. Claims 10-13 are objected to. All new grounds of rejection set forth in the application were necessitated by Applicant’s claim amendments; accordingly, this action is made final.
Response to Amendment
Claim Objections
In view of the amendment to claim 1, the objection to this claim is withdrawn.
Claim 10, dependent on claim 9, has been amended to reference claims 7 and 8. Applicant contends that this amendment clarifies the dependency chain without narrowing scope. The Examiner respectfully disagrees.
Initially, the Examiner notes that claims 7-8 are found in a different dependency chain (claim 1-> claim 6-> claim 7-> claim 8) from that of claim 10 (claim 1-> claim 9-> claim 10). Thus, the amendment has further confused the scope of claim 10, rather than clarifying it, as Applicant contends.
Furthermore, 35 USC § 112(e), 37 CFR § 1.75(c), and MPEP § 608.01(n)(I) all require a dependent claim which refers to more than one other claim to refer to such other claims in the alternative only:
PNG
media_image1.png
488
764
media_image1.png
Greyscale
PNG
media_image2.png
62
762
media_image2.png
Greyscale
PNG
media_image3.png
196
778
media_image3.png
Greyscale
PNG
media_image4.png
220
750
media_image4.png
Greyscale
Since claim 10 refers to each of claims 7, 8, and 9 and does not do so in the alternative, the claim is in improper form. Accordingly, claim 10 and its dependent claims 11-13 are not treated on their merits (see MPEP § 608.01(n)(I)).
Claim Interpretation – “whereby” clauses
Claims 6 and 11 have been amended to allegedly “eliminate any ambiguity” (page 6 of the Amendment) with respect to the claim interpretation set forth in the previous Action. However, each of these claims still recites a “thereby” clause which includes only language directed to the intended result of the now-positively-recited steps. Accordingly, this “intended result” language is not given patentable weight.
Prior Art Rejections
The Examiner notes that many of Applicant’s arguments appear to focus mainly on the subject invention, as disclosed in the specification, rather than the claimed subject matter. Accordingly, as an initial matter, the Examiner notes that, although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). While the Examiner appreciates Applicant’s discussion of the disclosed invention, the prior art rejections are made over the claimed invention.
If Applicant believes that these features of the disclosed invention distinguish over the applied art, the Examiner recommends amending the independent claims to recite these currently-unclaimed features.
Additionally, some arguments are directed to claims that don’t match the heading provided, and several arguments are directed to groups of claims, several of which are not in the same dependency chain. In addressing each of Applicant’s arguments, the Examiner has attempted to organize the arguments by claim, in the order of the claims, and humbly requests Applicant do the same in future responses:
Independent claim 1 (Sections A4A-D on pages 10-13)
On pages 10-13, Applicant contends that the use of Jagadeesan to support the light distribution model is improper, and provides four arguments (A-D) in support of this assertion. With respect to the feature in question, independent claim 1 recites “adjusting of image brightness based on a light distribution model and the video camera pose, wherein the light distribution model is that of a gradual decrease of lighting from a center of the video camera pose towards a periphery of the raw image”. The Examiner maintains that the proposed combination of Summan and Jagadeesan teaches this feature at least for the following reasons and for the reasons set forth in the rejection of claim 1.
On pages 10-11, in section A4A, Applicant asserts that Jagadeesan’s weighting is for a fundamentally different purpose than Applicant’s light distribution model and provides allegedly contrasting purposes for each.
Initially, the features (purposes) of the subject invention upon which the Applicant relies are not recited in the claim and therefore are not read into the claim.
Furthermore, the Applicant has failed to cite any section of the MPEP or relevant case law which requires a prior art feature serve the same purpose as a claimed feature in order to render the claimed feature obvious, particularly when the purpose of the claimed feature is not recited in the claims. Indeed, the purpose of a feature of the prior art is not required to serve the same purpose as the disclosed invention to establish obviousness, as Applicant appears to imply.
Moreover, the Applicant’s characterization of the reference is incorrect. Importantly, Jagadeesan discloses that “the light source is often equipped at the tip of the imaging modality, hence the image edge is often darker than the central area”, and proposes to update the RGB color of points in the captured image in order “to generate smooth texture of the model” by updating weights to “1.0 if the point is at the center area of the image, and decreas[ing it] as it approaches the image edge” ([0235]; emphasis added).
That is, contrary to Applicant’s assertions, Jagadeesan’s light model is indeed directed to at least two of the purposes of the Applicant’s light distribution model: “LED falloff” and “Radial illumination correction” (page 11 of the Amendment). Specifically, Jagadeesan’s weights are updated for the purpose of addressing radial illumination falloff that results from the placement of the light source and the resulting gradual brightness loss from the central area towards the periphery.
Accordingly, Applicant’s argument A4A is not found convincing.
On pages 11-12, in section A4B, Applicant argues that Jagadeesan does not describe a “light distribution model”. In support of this argument, Applicant asserts that the claimed light distribution model involves elements which are not found in the claims. Since these features are not claimed, they are not considered to be included in the scope of independent claim 1.
Further, as discussed above, Jagadeesan’s weighting is performed because “the light source is often equipped at the tip of the imaging modality, hence the image edge is often darker than the central area” ([0235]; emphasis added). Since Jagadeesan’s weight updating model is intended to address a falloff in illumination distribution, Jagadeesan’s model is reasonably interpreted as a “light distribution model”, contrary to Applicant’s arguments.
Accordingly, Applicant’s argument A4B is not found convincing.
On page 12 of the Amendment, in section A4C, Applicant asserts that the combination of Summan and Jagadeesan lacks motivation and reasonable expectation of success. In support of this assertion, Applicant contends that there is no technical connection between Summan’s context of industrial pipe inspection and Jagadeesan’s context of medical imaging, thus concluding that there is no credible rationale for when an ordinarily skilled artisan would combine the references or expect the resulting modification to render success. The Examiner respectfully disagrees.
Initially, the Examiner notes that, although Jagadeesan’s disclosure is focused on use in medical images, the reference contemplates applying its teachings to “industrial or other exploratory procedures”, such as the pipe modeling of Summan ([0003]).
Furthermore both Summan and Jagadeesan are directed to a “3D reconstruction method” in which a “3D point cloud” is obtained from “video frames” from an “endoscope”, “camera pose” is estimated, and an “extended surface model is merged” with the results to add new points thereto (quotes found in 16A-B, 18A, [0006-0007, 0108, 0234-0242] of Jagadeesan; see pp. 1336-1339 of Summan for analogous features). That is, Applicant’s characterization of the references having different contexts is entirely incorrect.
Also, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Summan to equip the light source at the tip of the probe and to weight the image brightness values that contribute to the 3D model in manner that reflects the gradual decrease in light from center to periphery, as taught by Jagadeesan, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results.
More specifically, Summan’s 3D reconstruction method from endoscopic images as modified by Jagadeesan’s light distribution model can yield a predictable result of “generat[ing] smooth texture of the model”, as taught by Jagadeesan since this effect is explicitly disclosed by Jagadeesan ([0235]). Thus, a person of ordinary skill would have appreciated including in Summan’s 3D reconstruction method from endoscopic images the ability to do Jagadeesan’s weighting for illumination correction since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable. It is predictable that the proposed modification would have illuminated a wider field of view while also “generat[ing] smooth texture of the model”, as taught by Jagadeesan ([0235]).
Accordingly, Applicant’s argument A4C is not found convincing.
On pages 12-13, in section A4D, Applicant argues that the Applicant’s system solves a pipe-specific problem that neither Summan nor Jagadeesan recognizes, and lists several problems that the subject invention addresses. Applicant alleges that the Applicant’s light distribution model is uniquely tailored to the challenges of small-bore videoscope inspection and represents a non-obvious advance.
However, Applicant has failed to cite any section of the MPEP or relevant case law which requires a prior art feature solve the same problem as a claimed feature in order to render the claimed feature obvious.
Moreover, contrary to Applicant’s assertions, Jagadeesan does indeed contemplate a solution to at least one problem cited by the Applicant – namely, “radial darkening at image periphery caused by…LED falloff”. As discussed above, Jagadeesan’s weighting is performed to address the problem that “the light source is often equipped at the tip of the imaging modality, hence the image edge is often darker than the central area” ([0235]; emphasis added).
Accordingly, Applicant’s argument A4D is not found convincing.
For all the foregoing reasons, the prior art rejection of independent claim 1 is maintained.
Dependent claim 2 (Sections A2 and B, on pages 8-9 and 13-14)
On pages 8-9 of the Amendment, in Section A2, Applicant argues that Summan requires specialized hardware and conditions incompatible with Applicant’s invention, apparently referring to the features of the claim 2 that “the videoscope is equipped with only the video camera and illuminating lights, and does not include any camera centering hardware, laser projection hardware, or any other sensor”.
However, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
The Examiner maintains that the proposed combination of Summan and El Kahi teaches the limitation in question, a discussed further below.
Accordingly, Applicant’s argument A2 is not found convincing.
On pages 13-14 of the Amendment, in Section B, Applicant addresses El Kahi with respect to claim 2.
Firstly, Applicant argues that El Kahi still employs prior camera auxiliary calibration and uses feature extraction. However, claim 2 does not preclude such features. Rather, the claim merely precludes hardware including “camera centering hardware, laser projection hardware, or any other sensor”. Neither auxiliary calibration nor feature extraction constitutes hardware.
Secondly, Applicant argues that El Kahi’s scope differs from Applicant’s since El Kahi’s system is deployed on a mobile robot, as opposed to Applicant’s system which operates with arbitrary handheld motion in low-texture, low-light environments. However, none of these features are claimed. Thus, these allegedly distinguishing features are not read into the scope of claim 2.
Thirdly, Applicant argues that El Kahi does not address small-bore constraints since El Kahi’s experiments focus on pipes greater than or equal to 20cm in diameter, whereas the invention specifically targets pipes less than or equal to 25mm. However, none of these features are recited in claim 2. While this limitation is claimed in claim 3, it is done so in a separate dependency chain; thus, these allegedly distinguishing features are not read into the scope of claim 2.
Lastly, Applicant asserts that El Kahi does not provide any motivation to combine it’s single-camera system with Summan’s specialized sensor suite. The Examiner respectfully disagrees. As noted in the prior Action, modifying the proposed combination of Summan and Jagadeesan to simplify the probe by including a single monocular camera as the only sensor, as taught by El Kahi, would have provided “advantages in terms of cost and safety” (Abstract of El Kahi). Indeed, presented with both Summan’s multi-sensor system and El Kahi’s simplified system, an ordinarily skilled artisan before the effective filing date of the claimed invention would have recognized a trade-off between accuracy and cost, as expressly noted by El Kahi. If said ordinarily skilled artisan would have preferred to reduce cost, as expressly proposed by El Kahi, then the proposed modification would have been quite obvious.
Accordingly, Applicant’s argument B is not found convincing.
For all the foregoing reasons, the prior art rejection of dependent claim 2 is maintained.
Dependent claim 3 (Section C, on pages 14-15)
Initially, the Examiner notes that Section C references dependent claims 3 and 6-7. However, these dependent claims are found in separate dependency chains. Since the arguments appear to be directed mainly to the features of claim 3, the Examiner will consider the arguments with respect to this claim only.
On pages 14-15, Applicant argues that scaling Summan’s probe (minimum 60mm) down to Kagami’s endoscope (8 mm capable) introduces new challenges including reduced optical field of view, diminished texture, increased sensitivity to motion artifacts, and compounded illumination vignetting, thus concluding that the substitution is not predictable. The Examiner respectfully submits that Applicant’s list of challenges does not refute the predictability of results of the proposed modification.
In particular, both Summan and Kagami are directed to “pipe inspection” based on “a 3D pipe reconstruction system using sequential images” using “Structure-from-Motion (SfM)” (Abstract). That is, despite the challenges noted by the Applicant, Kagami is still able to perform the analogous process disclosed in Summan, despite using an “endoscope” that can traverse a pipe network with “a narrower inner diameter (8.0 mm) than others (16.1mm)” (Section IV of Kagami). Thus, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to replace Summan’s probe with an endoscope small enough to traverse pipes as small as 8mm in diameter, as taught by Kagami, to arrive at the claimed invention discussed above. Such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, Summan’s probe and Kagami’s endoscope perform the same general and predictable function, the predictable function being traversing a network of pipes and capturing images thereof in order to create a 3D reconstruction of the pipe network. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of Summan’s probe by replacing it with Kagami’s endoscope. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious. It is predictable that the proposed modification would have allowed pipe reconstruction of a wider range of pipe networks including those with pipes having a small diameter.
Applicant additionally argues that Kagami’s problem domain of shape detection differs from Applicant’s problem domain. However, Applicant has failed to cite any section of the MPEP or relevant case law which requires a prior art feature solve the same problem as a claimed feature in order to render the claimed feature obvious.
Moreover, Kagami solves the same problem as both Summan and the subject invention – namely, pipe inspection based on a 3D pipe reconstruction system using sequential images using Structure-from-Motion (SfM). In view of such overlapping concepts found in each of Summan, Kagami, and the subject invention, Applicant’s argument is not found persuasive.
Finally, Applicant argues that Summan and Kagami still lack essential elements including features taught in claims 11-12 and 9-10 (page 15 of the Amendment). However, each of claims 9-12 are found in separate dependency chains from claim 3; thus, the features recited in claims 9-12 are not interpreted as being included in the scope of claim 3.
If Applicant believes that the features of claims 9-12 in combination with those recited in claim 3 would distinguish over the applied art, the Examiner recommends amending independent claim 1 to include the features of claims 3 and 9-12. In the meantime, no claim captures this combined scope.
For all the foregoing reasons, Applicant’s argument C is not found convincing. Thus, the prior art rejection of dependent claim 3 is maintained.
Dependent claim 6 (Section A1, on pages 7-8)
Initially, the Examiner notes that Section A references claims 1, 4-5 and 9-12. However, the arguments in Section A1 appear directed to claim 6. Thus, the Examiner will consider the arguments with respect to this claim only.
On pages 7-8 of the Amendment, Applicant asserts that Summan does not teach “building a second point cloud using characteristics of the first point cloud to replace it” since Summan’s “curve fitting procedure” in which the “position and orientation of each ring was set to the camera pose associated with each cross section” does not:
PNG
media_image5.png
272
778
media_image5.png
Greyscale
However, many of the details included in the list of things that Summan allegedly does not teach are not, in fact, recited in claim 6 or its dependent claims. These feature are therefore not read into the scope of claim 6.
Furthermore, as the Applicant acknowledges, Summan’s “curve fitting procedure” includes a process in which “the position and orientation of each ring was set to the camera pose associated with each cross section” (See page 7 of the Amendment and Fig. 6 of Summan, reproduced below).
PNG
media_image6.png
418
428
media_image6.png
Greyscale
Clearly, once the point cloud is fitted which such rings, it constitutes a new point cloud (one with fitted rings) different from the original point cloud (which did not include fitted rings). Thus, Applicant’s argument that Summan does not teach one point cloud that replaces another is incorrect.
This point is emphasized by claim 7 which recites that “the step of building the pipe point cloud further comprises a step of creating a virtual pipe using known pipe dimensions followed by a step of fitting pipe poses onto the virtual pipe to generate the second pipe point cloud”. According to claim 7, the step of building and replacing a first point cloud with a second one comprises a fitting process similar to that disclosed in Summan. That is, Summan appears to teach the limitation in question in a similar manner to the claimed invention.
Nonetheless, Kagami is relied upon to teach many of the features of claim 6 in question. On Kagami, Applicant asserts that the reference also fails to teach replacing an entire point cloud with a synthetic geometric model (page 8 of the Amendment). The Examiner respectfully disagrees.
Kagami discloses that “incremental SfM firstly initializes the 3D model for a selected image pair”, then “incrementally registers the input images to the model, while enriching the model by adding new 3D points correspond to 2D local feature tracks”, thus constructing a “temporary model” (Section III(A)). Kagami then discloses that “the system refines the temporary 3D model after each input image registration” (Section III(A)). Subsequently, the model can be refined by “fitting 3D points to the known pipe surface”, using “the known inner diameter of the pipe” (Sections II and III(C-D)). Thus, Kagami discloses: (a) building a first point cloud from the raw images; (b) creating a virtual pipe using known pipe dimensions; and (c) building a second pipe point cloud by fitting the virtual pipe to the first point cloud, as claimed in claim 6, contrary to Applicant’s assertions.
Finally, Applicant argues that Summan and Kagami still lack subsequent steps including features taught in claims 9-12. However, each of claims 9-12 are found in separate dependency chains from claim 6; thus, the features recited in claims 9-12 are not interpreted as being included in the scope of claim 6.
If Applicant believes that the features of claims 9-12 in combination with those recited in claim 6 would distinguish over the applied art, the Examiner recommends amending independent claim 1 to include the features of claims 6 and 9-12. In the meantime, no claim captures this combined scope.
For all the foregoing reasons, Applicant’s argument A1 is not found convincing. Thus, the prior art rejection of dependent claim 6 is maintained.
Dependent claim 8 (Section D, on page 15)
On page 15 of the Amendment, Applicant argues that Chang’s plane-fitting algorithm is developed for general geometric fitting, not for cylindrical pipe models.
However, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). While Chang does indeed disclose general geometric fitting, not for cylindrical pipe models, Summan and Kagami clearly do disclose geometric fitting for cylindrical pipe models (see at least Section III of Kagami). Thus, it is the combination of Summan, Kagami, and Chang that teaches the feature in question, at least for the reasons set forth in the rejection.
Applicant further argues that the proposed modification would have required a non-obvious adaptation because:
PNG
media_image7.png
172
588
media_image7.png
Greyscale
This list of alleged difficulties that an ordinarily-skilled artisan might have encountered in combining the teachings of the applied art merely amounts to a conclusory statement that is not grounded in evidence. Indeed, the Applicant fails to provide any evidence in support of this general assertion.
As best understood, Applicant appears to assert that the claimed invention would produce unexpected results, a secondary consideration of obviousness. However, “A showing of unexpected results must be based on evidence, not argument or speculation” (See MPEP 2145). Since the argument is submitted absent any supporting evidence, it is not found convincing.
Finally, Applicant argues that Summan and Kagami still lack subsequent steps including features taught in claims 9-12. However, each of claims 9-12 are found in separate dependency chains from claim 8; thus, the features recited in claims 9-12 are not interpreted as being included in the scope of claim 8.
If Applicant believes that the features of claims 9-12 in combination with those recited in claim 8 would distinguish over the applied art, the Examiner recommends amending independent claim 1 to include the features of claims 8 and 9-12. In the meantime, no claim captures this combined scope.
For all the foregoing reasons, Applicant’s argument D is not found convincing. Thus, the prior art rejection of dependent claim 8 is maintained.
Dependent claim 10 (Section A3, on pages 9-10)
On pages 9-10 of the Amendment, Applicant argues that Summan does not teach or suggest “creating an unwrapped image…using multiple rays projected onto the virtual pipe model”, as recited in claim 10. In support of this assertion, Applicant contends that the claim limitation involves a series of steps that are not recited in the claim. However, limitations from the specification are not read into the claims. If Applicant believes these steps distinguish over the applied art, the Examiner recommends amending independent claim 1 to recite these steps.
Applicant further asserts that the Examiner incorrectly conflates Summan’s texture mapping procedure with Applicant’s ray-based unwrapping, arguing that they are fundamentally different techniques serving different purposes. The Examiner maintains that Summan teaches the broadest reasonable interpretation of the claim.
For example, Summan discloses a “curve fitting procedure” which is used to fit each ring to the cross sectional point clouds comprising the main point cloud”, wherein “the position and orientation of each ring was set to the camera pose associated with each cross section” (See p. 1339 and Fig. 6 of Summan, reproduced below). Fig. 5 of the subject application, also reproduced below, shows “an example of projected rays fitted to the geometry of an interior of a round pipe” ([0021]). Notably, Fig. 5 shows the claimed “rays” as rings similar to the rings in Summan’s point cloud. Accordingly, the Examiner maintains that Summan teaches the limitation in question in a manner similar to the subject invention.
PNG
media_image6.png
418
428
media_image6.png
Greyscale
PNG
media_image8.png
518
388
media_image8.png
Greyscale
In view of the foregoing, Applicant’s argument A3 is not found convincing. Thus, the Examiner maintains that the applied are renders dependent claim 10 obvious.
Moreover, in view of the amendment of claim 10 to be dependent on multiple other dependent claims (and not in an alternative manner), the claim and its dependent claims 11-13 are not treated on their merits (See “Claim Objections” Section below).
Dependent claim 13 (Section E, on page 16)
On page 16 of the Amendment, Applicant argues that Khwaja does not teach or suggest the claimed feature of stitching unwrapped images with averaging of the pixels from related images, as required by claim 13.
In support of this assertion, Applicant contends that Khwaja’s energy minimization approach selects optimal seam paths, rather than averaging pixel values from multiple images to achieve seamless transitions. However, the claim is silent about seamless transitions. Even if the claim recited such language, it would be directed to an intended result, rather than any positively recited step. Applicant does not appear to dispute that Khwaja discloses image stitching with averaging of the pixels from related images. The Examiner maintains that the claim limitations of claim 13 are taught by the reference.
Applicant further argues that Khwaja’s scope is narrow and does not address the constraints of small-bore cylindrical geometry, radial illumination correction, or depth-image-based rendering for non-centralized cameras. However, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). Summan clearly teaches small-bore cylindrical geometry and depth-image-based rendering, and Jagadeesan clearly discloses radial illumination correction for all the reasons discussed above in conjunction with the upstream claims from which claim 13 depends and for the reasons set forth in the rejections. Thus, the combination of references teaches the limitation in question, contrary to Applicant’s assertions.
In view of the foregoing, Applicant’s argument E is not found convincing. Thus, the Examiner maintains that the applied are renders dependent claim 13 obvious.
Moreover, in view of the amendment of claim 10 to be dependent on multiple other dependent claims (and not in an alternative manner), claim 13 is not treated on its merits since the claim is dependent on claim 10 (See “Claim Objections” Section below).
Claim Objections
Claim 10, dependent on claim 9, has been amended to reference claims 7 and 8. Accordingly, claim 10 and its dependent claims 11-13 are objected to under 37 CFR 1.75(c) as being in improper form because a multiple dependent claim should refer to other claims in the alternative only. See MPEP § 608.01(n). Accordingly, claims 10-13 have not been further treated on the merits.
Claim Interpretation – “thereby” clauses
According to MPEP 2111.04 “a whereby clause in a method claim is not given weight when it simply expresses the intended result of a process step positively recited.”
Despite using the term “thereby” in place of “whereby”, each of claims 6 and 11 recites clauses that amount to an intended result of a positively recited process step. In particular, claim 6 recites “thereby reducing defects and discontinuities of the 3D textured model caused by insufficient lighting during image acquisition in step (c)” which are intended results of the positively recited steps in the claim. Claim 11 recites “thereby creating unwrapped and weighted images and increasing the sharpness of the panoramic image” which are intended results of the positively recited steps in the claim.
Accordingly, these clauses are not given patentable weight. As detailed below in the prior art mapping of these limitations, the applied art is capable of producing the intended result by virtue of teaching the positively recited steps in the respective claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7-8 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 7 recites “creating a virtual pipe”. Claim 6 (from which claim 7 depends) also recites “creating a virtual pipe”, It is unclear whether the virtual pipes in these two limitations are the same or different. When considering claim 7 on its merits, the above limitation will be interpreted as “creating [[a]] the virtual pipe”.
Claim 8 inherits the deficiencies of claim 7 by virtue of its dependency on claim 7.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4-5, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over “Image Mosaicing for Automated Pipe Scanning” by Summan et al. (hereinafter “Summan”) in view of U.S. Patent Application Publication No. 2022/0051431 to Jagadeesan et al. (hereinafter “Jagadeesan”).
As to independent claim 1, Summan discloses a video inspection method for an interior surface of a pipe (Abstract and Introduction disclose that Summan is directed to “remote visual inspection (RVI)…of the interior condition of pipelines” based on “video”, wherein an “image sequence returned from the camera is processed” to “create a 3D model for the interior surface of a pipeline”), the method comprising the steps of: a. defining geometrical size and shape of the interior surface of the pipe (Fig. 2a shows the cylindrical geometry and size of the pipe, and p. 1337 discloses that the signal processing is performed “by incorporating prior knowledge of the pipe diameter”), b. providing a videoscope sized to fit inside the pipe, c. advancing the videoscope through the pipe while acquiring raw images from a video camera of the videoscope (Fig. 1 and Probe Design section disclose that the probe is designed to fit within the pipe walls and includes a “camera”, and p. 1337 discloses that the camera captures “a sequence of 60 images covering 400 mm of linear travel inside the pipe”), d. estimating the video camera pose for at least some of the raw images of step (c) (Signal Processing Section discloses that the “pose of the probe must be estimated at each image acquisition point”), e. sequentially building a pipe point cloud, and a 3D textured model of the interior surface of the pipe from the raw images with adjustments for video camera poses (p. 1336 discloses tracking the “motion of the camera and comput[ing] a surface point cloud” and p. 1339 discloses that “a curve fitting procedure was used to fit each ring to the cross sectional point clouds comprising the main point cloud”, wherein “the position and orientation of each ring was set to the camera pose associated with each cross section”; that is, a 3D model of the interior surface of the pipe is sequentially updated with each captured image and its calculated pose; see also Fig. 6), f. unwrapping the images from step (e) using corresponding video camera poses and the pipe point cloud to create unwrapped images of the interior surface of the pipe (p. 1339 discloses performing “an unwrapping operation whereby the annular image is converted into a rectangular image”; for example, the “unwrapped view of the image in Fig. 3(a) is shown in Fig. 8”, wherein each unwrapped image is “mapped as texture to a virtual cylinder and inserted into a 3D environment” and “the radii of these cylinders are set to the mean radii obtained from the ring fitting procedure” discussed above involving the pose of each ring being fit to the point cloud), and g. creating a panoramic image of the interior surface of the pipe by stitching the unwrapped images together (p. 1340 discloses that “following the application of the unwrapped images…the images are stitched” together for performing the pipe inspection).
Summan does not expressly disclose that the building of the 3D textured model is followed by adjusting of image brightness based on a light distribution model and the video camera pose, wherein the light distribution model is that of a gradual decrease of lighting from a center of the video camera pose towards a periphery of the raw image.
Jagadeesan, like Summan, is directed to a “3D reconstruction method” in which a “3D point cloud” is obtained from “video frames”, “camera pose” is estimated, and an “extended surface model is merged” with the results to add new points thereto (16A-B, 18A, [0006-0007, 0108, 0234-0242]). Although Jagadeesan’s disclosure is focused on use in medical images, the reference contemplates applying its teachings to “industrial or other exploratory procedures” such as the pipe modeling of Summan ([0003]). Jagadeesan further discloses that the light source of the endoscopic device “is often equipped at the tip of the imaging modality, hence the image edge is often darker than the central area”; accordingly, “in order to generate smooth texture of the [3D] model”, Jagadeesan proposes applying an “updating weight” of “1.0 if the point is at the center area of the image,” and decreasing the weight “as it approaches the edge image” ([0235]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Summan to equip the light source at the tip of the probe and to weight the image brightness values that contribute to the 3D model in manner that reflects the gradual decrease in light from center to periphery, as taught by Jagadeesan, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. It is predictable that the proposed modification would have illuminated a wider field of view while also “generat[ing] smooth texture of the model”, as taught by Jagadeesan ([0235]).
As to claim 4, Summan as modified above further teaches that in step (d) the estimating of video camera poses is done by extracting and matching unique features from consecutive raw images (pp. 1336-1337 of Summan disclose extracting SIFT features “to track the motion of the camera…between successive image frames” by detecting “matches” and using “triangulation of image features” to “recover 3D coordinates” of a same point/feature “observed from at least two images”, thereby achieving “estimation of the camera pose”).
As to claim 5, Summan as modified above further teaches that said step of extracting and matching unique features is performed using a Structure-from-Motion (SfM) technique (Introduction of Summan discloses that the “image sequence returned from the camera is processed by a structure from motion “SFM” algorithm to reconstruct a 3D surface model of the pipe” using the processing discussed above with respect to claim 4).
As to claim 9, Summan as modified above further teaches that the step of unwrapping the raw images is performed using Depth-Image-Based-Rendering (DIBR) and ray tracing techniques (p. 1339 of Summan discloses performing “an unwrapping operation whereby the annular image is converted into a rectangular image”; for example, the “unwrapped view of the image in Fig. 3(a) is shown in Fig. 8”, wherein each unwrapped image is “mapped as texture to a virtual cylinder and inserted into a 3D environment”, wherein this technique of creating virtual viewpoints based on a reference view reads on the claimed DIBR; p. 1339 further discloses that the unwrapping involves “a curve fitting procedure was used to fit each ring to the cross sectional point clouds comprising the main point cloud”, wherein “the position and orientation of each ring was set to the camera pose associated with each cross section”, wherein the projection of the rings onto the virtual cylinder corresponds to the claimed ray tracing technique, particularly in view of the analogous procedure described in the specification of the subject application with respect to Figs. 4 and 5).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Summan in view of Jagadeesan and further in view of “A Vision-based System for Mapping the Inside of a Pipe” by El Kahi et al. (hereinafter “El Kahi”).
As to claim 2, Summan discloses that the probe includes additional sensors (laser profiler and IMU), but notes that these sensors are optional, and that “the optimal use of these additional sensor measurements shall be investigated in future work” (pp.1335-1336). Thus, Summan contemplates a probe that is absent additional sensors beyond the “camera” and “LEDs”. However, Summan does not expressly disclose such an arrangement. That is, Summan as modified above does not expressly disclose that the videoscope is equipped with only the video camera and illuminating lights, and does not include any camera centering hardware, laser projection hardware, or any other sensor.
El Kahi, like Summan, is directed to the “fault assessment of pipes” using an “automated 3D pipe reconstruction system” (Abstract). El Kahi discloses that “most pipe analysis systems rely on active sensors such as laser” similar to Summan’s system, but notes that “using a single monocular camera as the only sensor” provides “advantages in terms of cost and safety” (Abstract).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the proposed combination of Summan and Jagadeesan to simplify the probe by including a single monocular camera as the only sensor, as taught by El Kahi, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. It is predictable that the proposed modification would have provided “advantages in terms of cost and safety” (Abstract of El Kahi).
Claims 3 and 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Summan in view of Jagadeesan and further in view of “3D Pipe Network Reconstruction Based on Structure from Motion with Incremental Conic Shape Detection and Cylindrical Constraint” by Kagami et al. (cited in the IDS filed 8/14/23; hereinafter “Kagami”).
As to claim 3, Summan discloses that the probe is “designed to operate in pipes with a minimum diameter of 60mm” (p. 1335). Thus, Summan as modified above does not expressly disclose that the pipe is a small bore pipe with an internal diameter or an internal size characterizing a cross-section of the pipe being 25 mm or smaller.
Kagami, like Summan, is directed to “pipe inspection” based on “a 3D pipe reconstruction system using sequential images” using “Structure-from-Motion (SfM)” (Abstract). Kagami discloses that the reconstruction uses an “endoscope” that can traverse a pipe network with “a narrower inner diameter (8.0 mm) than others (16.1mm)” (Section IV).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to replace Summan’s probe with an endoscope small enough to traverse pipes as small as 8mm in diameter, as taught by Kagami, to arrive at the claimed invention discussed above. Such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, Summan’s probe and Kagami’s endoscope perform the same general and predictable function, the predictable function being traversing a network of pipes and capturing images thereof in order to create a 3D reconstruction of the pipe network. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of Summan’s probe by replacing it with Kagami’s endoscope. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious. It is predictable that the proposed modification would have allowed pipe reconstruction of a wider range of pipe networks including those with pipes having a small diameter.
As to claim 6, Summan as modified above further teaches that the step of building the pipe point cloud comprising (a) building a first point cloud from the raw images, and (c) building a second pipe point cloud by fitting to the first point cloud (pp. 1337-1338 of Summan discloses that a point cloud of the pipe surface (Fig. 4) is generated “through estimation of the camera pose and triangulation of image features”; p. 1339 discloses “a curve fitting procedure was used to fit each ring to the cross sectional point clouds comprising the main point cloud”, thus resulting in a new (second) point cloud), thereby reducing defects and discontinuities of the 3D textured model caused by insufficient lighting during image acquisition in step (c) (As noted above under the “Claim Interpretation – ‘thereby’ clauses” Section of the present action, “reducing defects and discontinuities” is an intended result of the positively-recited “building” step in the claim and is thus not given patentable weight. Summan as modified by Jagadeesan is capable of reducing defects and discontinuities by virtue of Summan’s teaching of the point cloud building step. Furthermore, Jagadeesan’s weighting step ([0235]) corrects for insufficient lighting in which the image edge is often darker than the central area. Accordingly, Summan as modified by Jagadeesan is capable of achieving the intended result set forth in the “thereby” clause of the claim).
Although Summan generally discloses that the point cloud generation is performed “by incorporating prior knowledge of the pipe diameter” (p. 1337), Summan does not provide specific details regarding how this prior knowledge is used in building the virtual pipe point cloud that replaces the initial pipe point cloud. That is, Summan does not expressly disclose (b) creating a virtual pipe using known pipe dimensions or fitting the virtual pipe to the first point cloud.
Kagami, like Summan, is directed to “pipe inspection” based on “a 3D pipe reconstruction system using sequential images” using “Structure-from-Motion (SfM)” involving “fitting 3D points to the known properties” including the known shape of the pipe (Abstract and Section III(C)). In particular, Kagami discloses that “incremental SfM firstly initializes the 3D model for a selected image pair”, then “incrementally registers the input images to the model, while enriching the model by adding new 3D points correspond to 2D local feature tracks”, thus constructing a “temporary model” (Section III(A)). Kagami then discloses that “the system refines the temporary 3D model after each input image registration” (Section III(A)). Subsequently, the model can be refined by “fitting 3D points to the known pipe surface”, using “the known inner diameter of the pipe” (Sections II and III(C-D)). That is, Kagami discloses: (a) building a first point cloud from the raw images (initialized temporary model); (b) creating a virtual pipe using known pipe dimensions (known pipe surface using known inner diameter of pipe); and (c) building a second pipe point cloud by fitting the virtual pipe to the first point cloud (fitting 3D points to the known pipe surface).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Summan’s fitting process to leverage the knowledge of the inner diameter of the pipe to fit the initialized model to the known pipe surface to build a finalized model, as taught by Kagami, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. It is predictable that the proposed modification would have obtained a “further complete and accurate 3D model” compared to systems that use the “standard SfM” such as Summan (See Section III(D) of Kagami).
As to claim 7, the proposed combination of Summan, Jagadeesan and Kagami further teaches that the step of building the pipe point cloud further comprises a step of creating a virtual pipe using known pipe dimensions followed by a step of fitting pipe poses onto the virtual pipe to generate the second pipe point cloud (pp. 1337-1339 of Summan discloses that a point cloud of the pipe surface (Fig. 4) is generated “by incorporating prior knowledge of the pipe diameter” and “a curve fitting procedure was used to fit each ring to the cross sectional point clouds comprising the main point cloud”, wherein each image is “mapped as texture to a virtual cylinder”, the radii of which are “obtained from the ring fitting procedure”; Section III(A, C-D) of Kagami similarly discloses a process of fitting the 3D points to a virtual cylinder involving a “cylinder constraint” that relies on the “known inner diameter of the pipe”; the reasons for combining the references are the same as those discussed above in conjunction with claim 6).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Summan in view of Jagadeesan and Kagami and further in view of U.S. Patent Application Publication No. 2016/0138914 to Chang et al. (hereinafter “Chang”).
As to claim 8, Summan as modified above does not expressly disclose that the step of fitting pipe poses onto the virtual pipe is conducted by minimizing an average of Euclidean distances between the virtual pipe and the first pipe point cloud.
Chang, like Summan and Kagami, is directed to fitting data to a point cloud (Abstract). Chang discloses that the fitting process is performed “by calculating the fitting position of the virtual plane iteratively using the least square method and the second point cloud data”, wherein “the average Euclidean distance is minimized”, the distance referring to “the Euclidean distance from any point of point cloud space to the fitting plane” ([0047]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the proposed combination of Summan, Jagadeesan, and Kagami to perform the fitting of the virtual pipe and the pipe point cloud by minimizing an average Euclidean distance between the virtual fitting plane and the point cloud, as taught by Chang, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. It is predictable that the proposed modification would have ensured “satisfying the accuracy evaluation criterion” of the point cloud fitting process ([0055] of Chang).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEAN M CONNER whose telephone number is (571)272-1486. The examiner can normally be reached 10 AM - 6 PM Monday through Friday, and some Saturday afternoons.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Greg Morse can be reached at (571) 272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEAN M CONNER/Primary Examiner, Art Unit 2663