DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claim 2 has been amended changing the scope and contents of the claim.
Applicant’s amendment filed July 6, 2026 overcomes the following objection/rejection(s) from the last Office Action of April 3, 2026:
Objections to the claims for minor informalities
Response to Arguments
Applicant's arguments filed July 6, 2026 have been fully considered but they are not persuasive.
Applicant argues, “Uchiyama fails to disclose or suggest that a projection image is generated by projecting three-dimensional data including the specified position of each of the phenomena and compositing the phenomena included in the images in such a manner that the phenomena for the images are distinguishable from each other (Remarks, 8)” as recited in claim 1.
Examiner respectfully disagrees. In addition to the previous references associated with claim 1, the examiner emphasizes Uchiyama specifically discloses at column 6, line 62, “The illustration in FIG. 7A shows that the position (estimated three-dimensional position) of a generated cylindrical image 701 corresponds to the actual position of the human body. Next, the consistency determination unit 106 projects the cylindrical image 701 onto camera images 702 and 703. At this time, when the estimated three-dimensional position is correct as illustrated in FIG. 7A, the cylindrical image is present at a correct distance from the cameras, so the sizes of the images (projection images) projected on the camera images 702 and 703 are similar to the sizes of detection frames as illustrated in FIG. 7B.” Figure 7A displays two cameras imaging the same object from different angles and determining the difference between the projection image and the detection frame (see Figure 7B).
For the ease of argument, each limitation will be analyzed individually with a corresponding citation and explanation below.
a projection image is generated by projecting three-dimensional data including the specified position of each of the phenomena and compositing the phenomena included in the images
The 3D data is read as the 3D position of the original object as noted in Figure 7A. Further, the specified position of the phenomena is read the “estimated 3D position” in Figure 7A. Finally, compositing the data is read as the outputs in Figure 7B, combining all data into one set.
See also, “Next, the consistency determination unit 106 projects the cylindrical image 701 onto camera images 702 and 703 (column 6, line 65).”
“in such a manner that the phenomena for the images are distinguishable from each other”
As noted in Figure 7B the projected image and the detected frame are overlaid (and in this case represented as different bounding boxes. The bounding boxes are read as being able to be distinguishable from one another. Further, as noted at column 7, line 1, “the cylindrical image is present at a correct distance from the cameras , so the sizes of the images ( projection images ) projected on the camera images 702 and 703 are similar to the sizes of detection frames as illustrated in FIG . 7B .” Thus, had they not been the same size, and rendered differently, they would be distinguishable based on size.
Further, though figure 7A and 7B only contain one object, this is read as exemplary. Had there been additional objects in the scene, one of ordinary skill in the art before the effective filing date would be able to understand how to manipulate the system with multiple objects present.
Based on the interpretation and analysis above and the additional citations within Uchiyama previously cited, the examiner maintains the 35 USC § 102 rejection and makes this action final.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-2 and 10-11 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by U.S. Patent No. 9,877,012 to Uchiyama et al. (hereinafter Uchiyama).
Regarding independent claim 1, Uchiyama discloses An image processing apparatus (abstract, “An image processing apparatus”) comprising:
at least one memory storing instructions (column 3, line 28, “A central processing unit (CPU) 303 executes a control program stored in a read-only memory (ROM) 304 to control the entire image detection apparatus.”); and
at least one processor configured to execute the instructions (column 3, line 28, “A central processing unit (CPU) 303 executes a control program stored in a read-only memory (ROM) 304 to control the entire image detection apparatus.”) to:
estimate a position and a posture of a camera when the camera has captured each of at least two images including phenomena the same as each other based on the at least two images and three-dimensional data of a structure (column 5, line 3, “First, in step S401, camera calibration is performed to estimate intrinsic parameters, positions, and orientations of the respective cameras;” column 5, line 17, “Alternatively, the positions and orientations of the respective cameras may be estimated by extracting feature points such as corners in camera images, scale-invariant feature transform (SIFT) features, etc., associating the feature point groups between the images, and then calculating the positions and orientations of the respective cameras and the positions of the feature point groups (refer to Pierre Moulon, Pascal Monasse, and Renaud Marlet “Adaptive structure from motion with a contrario model estimation” ACCV2012). Further, the intrinsic parameters and the positions and orientations of the cameras may be calculated simultaneously.;” column 3, line 63, “The camera information holding unit 103 holds information about intrinsic parameters, positions, and orientations of respective cameras that are acquired by camera calibration. The image acquisition unit 101 acquires images from the respective cameras;” column 4, line 35, “For example, suppose that first, second, third cameras are arranged to face in the direction of an object A to attempt to simultaneously capture images of the object A and the object information acquisition unit 102 successfully detects the object A from the camera images of the first and second cameras.”);
select an area of the phenomenon in each of the at least two images (Figure 7B; the detection frame of the object is read as the detected object (i.e. phenomena));
specify a position of each of the phenomena in a coordinate system of the three-dimensional data based on the estimated position and posture of the camera and the area selected in each of the at least two images (column 4, line 14, “The position estimation unit 107 estimates the coordinates of the three-dimensional position of the object based on the positions of the associated objects and the positional relationship of the cameras”); and
generate a projection image by projecting three-dimensional data including the specified position of each of the phenomena and compositing the phenomena included in the images in such a manner that the phenomena for the images are distinguishable from each other (column 4, line 61, “The display unit 109 displays on a display a result of the object detection and the three-dimensional positions with the camera images.;” Figure 7B; the projection image is projection on the detection frame; column 6, line 62, “The illustration in FIG. 7A shows that the position (estimated three-dimensional position) of a generated cylindrical image 701 corresponds to the actual position of the human body. Next, the consistency determination unit 106 projects the cylindrical image 701 onto camera images 702 and 703. At this time, when the estimated three-dimensional position is correct as illustrated in FIG. 7A, the cylindrical image is present at a correct distance from the cameras, so the sizes of the images (projection images) projected on the camera images 702 and 703 are similar to the sizes of detection frames as illustrated in FIG. 7B;” column 9, line 7, “Then, like the real image 701 in FIG. 7 and a virtual image 801 in FIG. 8, the image 1004 is projected onto the respective camera images;” distinguishable via the outlines).
Regarding independent claim 2, the rejection of claim 1 applies directly. Additionally, Uchiyama further discloses wherein the at least one processor is configured to execute the instructions to change, for each of the phenomena in the projection image, at least one of transmittance to be set or a display color to be assigned (NOTE: at least one of A and B is read that only one is required; based on the specification it appears only one is required, not both; column 8, line 17, “The frames of the same person on different cameras are specified in the same color so that the user can recognize with ease whether human bodies on different cameras are the same human body”).
Regarding independent claim 10, the rejection of claim 1 applies directly. Additionally, Uchiyama further discloses An image processing method (column 1, line 9, “The present invention relates to an image processing apparatus configured to estimate the three-dimensional position of an object from images captured by a plurality of cameras and a method therefore.”) comprising:
estimating a position and a posture of a camera when the camera has captured each of at least two images including phenomena the same as each other based on the at least two images and three-dimensional data of a structure (column 5, line 3, “First, in step S401, camera calibration is performed to estimate intrinsic parameters, positions, and orientations of the respective cameras;” column 5, line 17, “Alternatively, the positions and orientations of the respective cameras may be estimated by extracting feature points such as corners in camera images, scale-invariant feature transform (SIFT) features, etc., associating the feature point groups between the images, and then calculating the positions and orientations of the respective cameras and the positions of the feature point groups (refer to Pierre Moulon, Pascal Monasse, and Renaud Marlet “Adaptive structure from motion with a contrario model estimation” ACCV2012). Further, the intrinsic parameters and the positions and orientations of the cameras may be calculated simultaneously.;” column 3, line 63, “The camera information holding unit 103 holds information about intrinsic parameters, positions, and orientations of respective cameras that are acquired by camera calibration. The image acquisition unit 101 acquires images from the respective cameras;” column 4, line 35, “For example, suppose that first, second, third cameras are arranged to face in the direction of an object A to attempt to simultaneously capture images of the object A and the object information acquisition unit 102 successfully detects the object A from the camera images of the first and second cameras.”);
selecting an area of the phenomenon in each of the at least two images (Figure 7B; the detection frame of the object is read as the detected object (i.e. phenomena));
specifying a position of each of the phenomena in a coordinate system of the three-dimensional data based on the estimated position and posture of the camera and the area selected in each of the at least two images (column 4, line 14, “The position estimation unit 107 estimates the coordinates of the three-dimensional position of the object based on the positions of the associated objects and the positional relationship of the cameras”); and
generating a projection image by projecting three-dimensional data including the specified position of each of the phenomena and compositing the phenomena included in the images in such a manner that the phenomena for the images are distinguishable from each other (column 4, line 61, “The display unit 109 displays on a display a result of the object detection and the three-dimensional positions with the camera images.;” Figure 7B; the projection image is projection on the detection frame; column 6, line 62, “The illustration in FIG. 7A shows that the position (estimated three-dimensional position) of a generated cylindrical image 701 corresponds to the actual position of the human body. Next, the consistency determination unit 106 projects the cylindrical image 701 onto camera images 702 and 703. At this time, when the estimated three-dimensional position is correct as illustrated in FIG. 7A, the cylindrical image is present at a correct distance from the cameras, so the sizes of the images (projection images) projected on the camera images 702 and 703 are similar to the sizes of detection frames as illustrated in FIG. 7B;” column 9, line 7, “Then, like the real image 701 in FIG. 7 and a virtual image 801 in FIG. 8, the image 1004 is projected onto the respective camera images;” distinguishable via the outlines).
Regarding independent claim 11, the rejection of claim 1 applies directly. Additionally, Uchiyama further discloses A non-transitory computer readable medium storing a program for causing a computer to execute processing (column 3, line 28, “A central processing unit ( CPU ) 303 executes a control program stored in a read - only memory ( ROM ) 304 to control the entire image detection apparatus . The ROM 304 stores the control program to be executed by the CPU 303 and various types of parameter data . The control program is executed in the CPU 303 to cause the image detection apparatus to function as various types of units that execute processing illustrated in a flow chart 35 described below”) comprising:
estimating a position and a posture of a camera when the camera has captured each of at least two images including phenomena the same as each other based on the at least two images and three-dimensional data of a structure (column 5, line 3, “First, in step S401, camera calibration is performed to estimate intrinsic parameters, positions, and orientations of the respective cameras;” column 5, line 17, “Alternatively, the positions and orientations of the respective cameras may be estimated by extracting feature points such as corners in camera images, scale-invariant feature transform (SIFT) features, etc., associating the feature point groups between the images, and then calculating the positions and orientations of the respective cameras and the positions of the feature point groups (refer to Pierre Moulon, Pascal Monasse, and Renaud Marlet “Adaptive structure from motion with a contrario model estimation” ACCV2012). Further, the intrinsic parameters and the positions and orientations of the cameras may be calculated simultaneously.;” column 3, line 63, “The camera information holding unit 103 holds information about intrinsic parameters, positions, and orientations of respective cameras that are acquired by camera calibration. The image acquisition unit 101 acquires images from the respective cameras;” column 4, line 35, “For example, suppose that first, second, third cameras are arranged to face in the direction of an object A to attempt to simultaneously capture images of the object A and the object information acquisition unit 102 successfully detects the object A from the camera images of the first and second cameras.”);
selecting an area of the phenomenon in each of the at least two images (Figure 7B; the detection frame of the object is read as the detected object (i.e. phenomena));
specifying a position of each of the phenomena in a coordinate system of the three-dimensional data based on the estimated position and posture of the camera and the area selected in each of the at least two images (column 4, line 14, “The position estimation unit 107 estimates the coordinates of the three-dimensional position of the object based on the positions of the associated objects and the positional relationship of the cameras”); and
generating a projection image by projecting three-dimensional data including the specified position of each of the phenomena and compositing the phenomena included in the images in such a manner that the phenomena for the images are distinguishable from each other (column 4, line 61, “The display unit 109 displays on a display a result of the object detection and the three-dimensional positions with the camera images.;” Figure 7B; the projection image is projection on the detection frame; column 6, line 62, “The illustration in FIG. 7A shows that the position (estimated three-dimensional position) of a generated cylindrical image 701 corresponds to the actual position of the human body. Next, the consistency determination unit 106 projects the cylindrical image 701 onto camera images 702 and 703. At this time, when the estimated three-dimensional position is correct as illustrated in FIG. 7A, the cylindrical image is present at a correct distance from the cameras, so the sizes of the images (projection images) projected on the camera images 702 and 703 are similar to the sizes of detection frames as illustrated in FIG. 7B;” column 9, line 7, “Then, like the real image 701 in FIG. 7 and a virtual image 801 in FIG. 8, the image 1004 is projected onto the respective camera images;” distinguishable via the outlines).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3-4 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Uchiyama as applied to claim 1 above, and further in view of U.S. Patent No. 9,760,987 to Ueno et al. (hereinafter Ueno).
Regarding dependent claim 3, the rejection of claim 1 is incorporated herein. Additionally, Uchiyama fails to explicitly disclose wherein the at least two images are captured at times different from each other.
However, Ueno discloses wherein the at least two images are captured at times different from each other (column 2, line 47, “In an inspection service and the like, a plurality of images of an object taken at different clock times are compared. ”).
Uchiyama is directed toward, “An image processing apparatus includes a holding unit configured to hold a positional relationship of a plurality of image capturing units, an acquisition unit configured to detect objects from respective images captured by the plurality of image capturing units and acquire positions of the objects on the captured images and geometric attributes of the objects, an associating unit configured to associate the detected objects, based on the positional relationship, the positions and the geometric attribute (abstract).” Ueno is directed toward, “A guiding method includes obtaining data of a first image, detecting with a computer reference image data corresponding to a reference object in the data of the first image, calculating with the computer a first condition based on an appearance of the reference object in the first image, the first condition indicating an operational condition of an imaging apparatus when first image was captured (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Uchiyama and Ueno are directed toward similar methods of endeavor of image processing for object detection and camera parameter estimation. Further, one of ordinary skill in the art before the effective filing date would easily understand there are often times a user is interested in analyzing an object over time, as opposed to a static object. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Ueno in order to ensure a user can see a change in an object over time.
Regarding dependent claim 4, the rejection of claim 3 is incorporated herein. Additionally, Uchiyama and Ueno in the combination fail to explicitly disclose wherein the at least one processor is configured to execute the instructions to overlap, in a state in which the positions of the phenomena are shifted, the images of the phenomena on each other in the projection image in a chronological order or in a reverse chronological order of the dates on which the images of the phenomena have been captured. However, Ueno does disclose obtaining images from various time points (column 2, line 47, “In an inspection service and the like, a plurality of images of an object taken at different clock times are compared. ”). Further, as noted above, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand the benefit of displaying objects over time. Additionally, displaying images as overlaid in different orders is a design matter that could easily be adjusted by one of ordinary skill in the art before the effective filing date of the claimed invention based on the interest of the reviewer.
Regarding dependent claim 8, the rejection of claim 1 is incorporated herein. Additionally, Uchiyama fails to explicitly disclose wherein the at least one processor is configured to execute the instructions to display a scale symbol in the projection image.
However, Ueno discloses wherein the at least one processor is configured to execute the instructions to display a scale symbol in the projection image (column 4, line 24, “Meanwhile, since the marker M also exists in the real space, when the marker M is recognized from the taken image in which the real space is taken, the virtual space and the real space are associated via the marker M. In the AR technique, an object to be a reference for a virtual space, such as the marker M, is referred to as a reference object;” the marker determines a relationship between the real and virtual space, and is read as a scale symbol; column 4, line 51, “The marker M exemplified in FIG. 3 is in a square shape and is established in size in advance (for example, the length of each side is 5 cm and the like);” knowing the predetermined size of the marker allows the marker itself to be a scale; column 8, line 28, “This is because the picture of the marker 306 in the image 300 and the picture of a marker 314 in the image 308 are present in different size and in different positions, so that the image 308 is supposed to be taken in an imaging position different from the image 300. When the imaging direction is also different, the shape of the picture of the marker 306 turns out to be different from the shape of the picture of the marker 314 as well.”).
Uchiyama is directed toward, “An image processing apparatus includes a holding unit configured to hold a positional relationship of a plurality of image capturing units, an acquisition unit configured to detect objects from respective images captured by the plurality of image capturing units and acquire positions of the objects on the captured images and geometric attributes of the objects, an associating unit configured to associate the detected objects, based on the positional relationship, the positions and the geometric attribute (abstract).” Ueno is directed toward, “A guiding method includes obtaining data of a first image, detecting with a computer reference image data corresponding to a reference object in the data of the first image, calculating with the computer a first condition based on an appearance of the reference object in the first image, the first condition indicating an operational condition of an imaging apparatus when first image was captured (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Uchiyama and Ueno are directed toward similar methods of endeavor of image processing for object detection and camera parameter estimation. Further, one of ordinary skill in the art before the effective filing date would easily understand there are often times a user is interested in analyzing an object size. Knowing an object size relative to a scale is beneficial to understand if an object has changed in size. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Ueno in order to ensure a user can determine an accurate size of an object in an image to understand if objects size have changed over time.
Allowable Subject Matter
Claims 5-7 and 9 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 5-7:
The following is an examiner’s statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of detecting objects in an image based on multiple images obtained. However, none of them alone or in any combination teaches determining a surface of the detected object based on a position of the object, and determine a projection range in the image in the projection surface.
The closest prior art being previously cited Ueno discloses “A guiding method includes obtaining data of a first image, detecting with a computer reference image data corresponding to a reference object in the data of the first image, calculating with the computer a first condition based on an appearance of the reference object in the first image, the first condition indicating an operational condition of an imaging apparatus when first image was captured, and for a second image to be captured (abstract).” Additionally, Uemo allows for detection of a crack as seen in Figures 1A, 1B and 2, however, there is no detection of the surface the crack itself is on.
Thus, Uemo fails to disclose determining a surface of the detected object based on a position of the object, and determine a projection range in the image in the projection surface.
Claim 9:
The following is an examiner’s statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of detecting objects in an image based on multiple images obtained. However, none of them alone or in any combination teaches determining a database of the object, area data, and estimated camera position and posture.
The closest prior art being previously cited Uemo discloses methods of guiding a user to a specific camera position/pose based on prior data. However, this guidance is based on utilizing a physical marker to align with, as opposed to comparing to stored data of the object and camera position.
Thus, Uemo fails to disclose determining a database of the object, area data, and estimated camera position and posture.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Windsor whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COURTNEY JOAN NELSON/Primary Examiner, Art Unit 2661
Prosecution Insights