DETAILED ACTION
Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-11, 13-17, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bolvin et al. (US 2015/0084951) In view of Sudo (US 2023/0098053).
[Claim 1] Bolvin teaches An information processing apparatus comprising:
circuitry configured to
estimate first position information (i.e. using lineup algorithm) of the first camera in relation to the display (i.e. 3D offset) based on the at least one first image (i.e. pattern) (p. 0239-0240),
obtain offset information between the first camera and the second camera (i.e. 3D offset) (p. 0239-0240), and
estimate second position information (i.e. positional information for witness camera or film camera from patterns) of the second camera and third position information of the at least one marker (i.e. natural markers) in relation to the display based on the first position information (i.e. utilizing lineup algorithm), the at least one second image (i.e. 2nd pattern), and the offset information (i.e. 3D offset) (p. 0189, 0200, 0239-0240). natural markers are automatically learned and computed in 3D by the system as the user scans the environment walking around. As soon as the user has started to do the survey, he has to also capture the registration object (see section 2.1.3.2) in order to automatically estimate the scale, origin and orientation of the world;
wherein a positional relation between the first camera and the second camera that is fixed (i.e. witness and film cameras are attached to each other) (p. 0179, 0240).
Bolvin is not entirely clear in teaching the specific feature of:
obtain at least one first image of at least one first marker displayed on a display, the at least one first image being acquired by a first camera,
obtain at least one second image of at least one second marker arranged on a surface other than the display screen in an imaging space around the display, the at least one second image being acquired by a second camera,
“wherein the offset information is obtained using a positional relation”;
wherein a type of the at least one first marker is different from a type of the at least one second marker.
Sudo teaches the specific feature of:
obtain at least one first image of at least one first marker displayed on a display (i.e. first markers K.sub.1 on the display screen SC), the at least one first image being acquired by a first camera (i.e. projectors 10) (fig. 7-8; p. 0061-0064, 0066),
obtain at least one second image of at least one second marker (i.e. markers K.sub.2) arranged on a surface other than the display screen (i.e. K2-13 is display just below the SC) in an imaging space around the display, the at least one second image being acquired by a second camera (i.e. projectors 10) (fig. 7-8; p. 0061-0064, 0066),
“wherein the offset information (i.e. overlap marker coordinates) is obtained using a positional relation” (i.e. coordinates positions first and second image according to the markers) (p. 0066);
wherein a type of the at least one first marker is different from a type of the at least one second marker (i.e. markers are different colors) (p. 0064).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 2] Bolvin teaches the information processing apparatus according to claim 1, wherein the circuitry is further configured to control output of a display image by the display according to a position of the first camera (i.e. projecting sensor depths of the film camera) (p. 0017).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen” (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 3] Bolvin teaches the information processing apparatus according to claim 2,
wherein the circuitry is further configured to control the output of the display image by the display during virtual production (i.e. virtual modeling) in which the display image output by the display is included in images acquired by the first camera (constructing depth parameters) (p. 0297).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen” (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 4] Bolvin teaches The information processing apparatus according to claim 1, wherein the at least one first image of the display includes a plurality of first markers (i.e. natural markers) (p. 0189).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen” (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 5] Bolvin teaches The information processing apparatus according to claim 4, wherein each first marker includes an augmented reality marker (i.e. augmented reality framework for natural markers) (p. 0192-0193).
[Claim 6] Bolvin teaches The information processing apparatus according to claim 4, wherein each first marker is displayed at known coordinates in a coordinate system of the display (i.e. calibration chart uses known coordinates) (p. 140, 0240-0244).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen” (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 7] Bolvin teaches The information processing apparatus according to claim 6, wherein the circuitry is configured to estimate the first position information of the first camera in relation to the display
using the known coordinates of the at least one first marker (i.e. calibration chart uses known coordinates with natural markers) (p. 140, 0240-0244).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen” (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 8] Bolvin teaches The information processing apparatus according to claim 6, wherein the circuitry is further configured to control output of a display image by the display
according to a position and orientation (i.e. positional information) of the first camera in the coordinate system of the display (i.e. witness camera or film camera, both can read the patterns) (p. 0239-0240).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen(fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 9] Bolvin teaches The information processing apparatus according to claim 1, wherein the circuitry is configured to obtain a plurality of first images of the display (i.e. white or black circles) (p. 0240), and
wherein the circuitry is configured to obtain a plurality of second images of the at least one second marker (i.e. natural markers) corresponding to the plurality of first images (i.e. pattern) (p. 0189, 0200, 0239-0240). natural markers are automatically learned and computed in 3D by the system as the user scans the environment walking around. As soon as the user has started to do the survey, he has to also capture the registration object (see section 2.1.3.2) in order to automatically estimate the scale, origin and orientation of the world.
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen(fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 10] Bolvin teaches The information processing apparatus according to claim 9, wherein the plurality of first images of the display are acquired by the first camera (i.e. film camera) from a position corresponding to the first position information (i.e. positional information) and the plurality of second images are acquired by the second camera (i.e. witness camera) from a position corresponding to the second position information (i.e. positional information whilst cameras attached to each other) (p. 0239-0240).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen” (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 11] Bolvin teaches The information processing apparatus according to claim 9, wherein the plurality of first images are acquired by the first camera (i.e. film camera) from a plurality of positions and the plurality of second images are acquired by the second camera (i.e. witness camera) from a plurality of positions corresponding to the plurality of positions of the first camera (i.e. using lineup algorithm to track objects or natural markers) (p. 0194, 0239-0240).
[Claim 13] Bolvin teaches The information processing apparatus according to claim 1, wherein the circuitry is configured to estimate the second position information of the second camera in relation to the display based on the first position information and the offset information (i.e. using lineup algorithm to track objects or natural markers and determine the 3D offset) (p. 0194, 0239-0240).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen” (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 14] Bolvin teaches The information processing apparatus according to claim 13, wherein the circuitry is configured to estimate the third position information (i.e. all position information utilized by the lineup algorithm) of the at least one second marker in relation to the display based on the at least one second image and the second position information (i.e. using lineup algorithm to track objects or natural markers and determine the 3D offset) (p. 0194, 0239-0240).
Bolvin is not entirely clear in teaching the specific feature of:
“images of a display screen”.
Sudo teaches the specific feature of:
“images of a display screen” (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 15] Bolvin teaches The information processing apparatus according to claim 14, wherein the at least one second marker includes a plurality of second markers (i.e. natural markers) provided around the second camera in the imaging space, and wherein the circuitry is further configured to generate a map of the plurality of second markers based on the third position information (i.e. creating a map of 3D point cloud markers) (p. 0224-0228).
Bolvin is silent regarding the specific feature of:
“second markers provided around the second camera on the surface other than the display screen”.
Sudo teaches the specific feature of:
“second markers provided around the second camera on the surface other than the display screen” (i.e. K2-13 is display just below the SC) (fig. 7-8; p. 0061-0064, 0066).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided display markers as taught by Sudo to the system of Bolvin to coordinate multiple images together (p. 0066).
[Claim 16] Bolvin teaches The information processing apparatus according to claim 1, wherein the obtained offset information (i.e. 3D offset) is obtained based on predetermined offset information (i.e. utilizing lineup algorithm) between the second camera and a third camera (i.e. numerous witness cameras) (p. 0189, 0200, 0239-0240). natural markers are automatically learned and computed in 3D by the system as the user scans the environment walking around. As soon as the user has started to do the survey, he has to also capture the registration object (see section 2.1.3.2) in order to automatically estimate the scale, origin and orientation of the world; and
wherein a positional relation between the second camera and the third camera is fixed (i.e. cameras can be attached to each other) (p. 0239-0240).
[Claim 17] Bolvin teaches The information processing apparatus according to claim 1,
wherein the circuitry is configured to estimate the offset information (i.e. 3D offset) by performing calibration based on a plurality of relative poses (i.e. determines positional information based on both cameras location which is determined, the cameras can be located anywhere) of the first camera and the second camera (i.e. film and witness cameras) (p. 0189, 0239-0240).
Claim 19 is analyzed and interpreted as a method of claim 1.
Claim 20 recites “A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer” to perform the steps of claim 1. Bolvin inherently discloses “A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer” to perform the steps of claim 1.
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bolvin et al. (US 2015/0084951) In view of Sudo (US 2023/0098053), and further in view of Grabli et al. (US 2020/0286284).
[Claim 12] Bolvin is silent regarding The information processing apparatus according to claim 1, wherein the at least one second marker is a retroreflective material, and wherein the second camera is an infrared camera.
Grabli teaches The information processing apparatus according to claim 1, wherein the at least one second marker is a retroreflective material (i.e. retroreflective markers), and wherein the second camera is an infrared camera (i.e. IR) (p. 0025, 0037).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided camera types and reflective markers as taught by Grabli to the system of Bolvin to generate content (p. 0025)
Claim(s) 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bolvin et al. (US 2015/0084951) In view of Sudo (US 2023/0098053), and further in view of Drixler (US 2020/0238525), and further in view of Tsuzuki (US 2015/0104097)
[Claim 18] Bolvin is silent regarding The information processing apparatus according to claim 1, wherein the circuitry is configured to estimate the offset information by solving hand-eye calibration based on a seven-degrees-of-freedom parameter including
a six-degrees-of-freedom parameter relating to relative positions and orientations of the first camera and the second camera, and a single-degree-of-freedom parameter relating to scale invariance.
Drixler teaches Tsuzuki teaches the specific feature of:
wherein the circuitry is configured to estimate the offset information by solving hand-eye calibration based on a seven-degrees-of-freedom parameter (i.e. degrees of freedom) (p. 0022).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided hand eye calibration as taught by Drixler to the system of Bolvin to provide calibration using degrees of freedom (p. 0022).
Tsuzuki teaches the specific feature of:
a six-degrees-of-freedom parameter (i.e. seven degrees of freedom) relating to relative positions and orientations of the first camera and the second camera (i.e. first and second cameras), and a single-degree-of-freedom parameter relating to scale invariance (p. 0036).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the present invention to have provided degrees of freedom as taught by Tsuzuki to the system of Bolvin to provide a solution for scale indetermination (p. 0036).
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Claims 1-20 are rejected.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20230080888 A1 FUJIMORI; Toshiki – image markers for multiple cameras
US 20180367681 A1 Xu; Shan et al. – multiple camera calibration chart
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MUSHFIKH I ALAM whose telephone number is (571)270-1710. The examiner can normally be reached 1:00PM-9:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Nasser Goodarzi can be reached at 571-272-4195. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MUSHFIKH I. ALAM
Primary Examiner
Art Unit 2426
/MUSHFIKH I ALAM/Primary Examiner, Art Unit 2426 9/3/2026
Prosecution Insights