DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on January 2, 2025 and January 27, 2025 were filed on/after the mailing date of the application on January 2, 2025. The submissions are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Drawings
The drawings were received on January 2, 2025. These drawings are accepted.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 13, 18, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ogasawara (US 2022/0277520) in view of Ito (US 11,451,759).
As to claim 1, Ogasawara discloses an information processing system (Figure 1A, information processing system 100) comprising one or more memories storing instructions (e.g. Figure 2B, RAM 212 and/or ROM 213 of image generating apparatus 104 of information processing system 100) and one or more processors that execute the instructions (e.g. Figure 2B, CPU 211 of image generating apparatus 104 of information processing system 100)([0044]-[0046] notes CPU 211 performs processing as described using the programs and data stored in the RAM 212 and ROM 213) to: obtain three-dimensional models of one or more objects at a designated time from a storage that stores three-dimensional models indicating three-dimensional shapes of objects at respective times (e.g. obtain 3D models from database 103)([0019]-[0022] notes plurality of cameras of the plurality of sensor systems 101 perform image capture, [0023] notes image recording apparatus 102 obtains a plurality of images from the plurality of sensor systems 101, combines the plurality of images and a time code used for the image capture, and stores them in the database 103, [0024] notes database 103 manages event information, 3D model information, and the like, [0036]-[0037] notes based on a plurality of captured images, the 3D model generation unit 202 generates a 3D model representing the three-dimensional shape of an object within the region 120, e.g. the 3D model generation unit 202 generates a 3D model of each of foreground objects based on the plurality of foreground images, [0038] notes the 3D model generation unit 202 records the generated 3D models in the database 103 along with time codes, [0039] notes 3D model generation unit 202 may be included in the image recording apparatus, [0040] notes image generation unit 203 obtains (e.g. reads) 3D models from the database 103); using the three-dimensional models of the one or more objects at the designated time, generate a virtual viewpoint image corresponding to the designated time from a virtual viewpoint based on a designated virtual viewpoint parameter ([0040] notes the image generation unit 203 (e.g. of image generating apparatus 104) obtains the 3D models from database 103 and generates a virtual viewpoint image based on the 3D models obtained, where [0026] further the virtual viewpoint image generated using the 3D models from database 103 and information on the viewpoint of the virtual camera, the viewpoint information on the virtual camera includes information indicating the position and orientation of the virtual viewpoint, more specifically, the viewpoint information includes a parameter representing the three-dimensional position of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in a pan direction, a tilt direction, and a roll direction (“virtual viewpoint parameter”), Figures 6A, 6B, [0086]-[0088] notes in response to a certain time code being specified, 3D model information on each object related to the time code can be obtained using the information illustrated); output the generated virtual viewpoint image (e.g. [0028] notes tablet 105 displays the virtual viewpoint image generated by the image generating apparatus 104 or stores the virtual viewpoint image in database 103); and… record information on the three-dimensional models of the objects corresponding to the generated virtual viewpoint image (Figure 6A, [0084] notes event information table 610 which shows the storage location of 3D model information for each of the objects for every time code of the captured event, Figure 6B, [0085] notes 3D model information table 620 managed by database 103 of information managed by database 103, [0086] notes in response to a certain time code being specified, 3D model information on each object related to the time code specified can be obtained using the information shown in Figures 6A and 6B).
Ogasawara differs from the invention as defined in claim 1 in that Ogasawara does not disclose “…determine whether a predetermined condition is satisfied, the predetermined condition being a condition related to at least one of the time and the virtual viewpoint parameter; and in response to satisfaction of the predetermined condition, record information on the three-dimensional models of the objects corresponding to the generated virtual viewpoint image.”
Ito further discloses generate a virtual viewpoint image corresponding to the designated time from a virtual viewpoint based on a designated virtual viewpoint parameter (Figure 5, step S502, column 3, lines 22-29, column 6, lines 31-38 notes the virtual viewpoint parameter designation unit 102 designates a virtual viewpoint parameter for the virtual viewpoint image generation unit 104, the virtual viewpoint parameter includes one or more parameters among direction information indicating a direction from a virtual viewpoint, position information indicating a position of a virtual viewpoint, and angle information indicating a viewing angle of a virtual viewpoint, and step S504, column 3, lines 37-45, column 6, lines 42-47 notes using the image data obtained by the image data obtaining unit 103, the virtual viewpoint image generation unit 104 generates virtual viewpoint image data according to the virtual viewpoint parameter designated by the virtual viewpoint parameter designation unit 102); output the generated virtual viewpoint image (Figure 1 illustrates output from virtual viewpoint image generation unit 104 to virtual viewpoint parameter assignment unit (denoted by arrows)); determine whether a predetermined condition is satisfied, the predetermined condition being a condition related to at least one of the time and the virtual viewpoint parameter (column 3, lines 55-59 notes the image capturing time information and the virtual viewpoint parameter to be used to generate the virtual viewpoint image data are obtained from the virtual viewpoint manipulation unit 110, where Figure 5, step S501, column 3, lines 16-21, column 6, lines 25-31 notes the image capturing time designation unit 101 designates an image capturing time for the image data obtaining unit 103, which is obtained from the virtual viewpoint manipulation unit 110, step S502, column 3, lines 22-29, column 6, lines 31-38 notes the virtual viewpoint parameter designation unit 102 designates a virtual viewpoint parameter for the virtual viewpoint image generation unit 104, then in step S503, column 3, lines 30-36, column 6, lines 38-42 notes from the multi-viewpoint image data 120, the image data obtaining unit 103 obtains the image data at the image capturing time designated by the image capturing time designation unit 101, where column 2, lines 58-65 notes the multi-viewpoint image data 120 is image data obtained from a plurality of cameras at different positions and viewpoints, the multi-viewpoint image data 120 may contain image capturing times, information on the cameras, and so on, thus considered the “predetermined condition” may be considered the capturing time information and virtual viewpoint parameter obtained from the virtual viewpoint manipulation unit 110 corresponds with at least the image capturing times of the multi-viewpoint image data 120); and in response to satisfaction of the predetermined condition (e.g. in response to the multi-viewpoint image data 120 corresponding to certain image capturing times and the designated virtual viewpoint parameter), record information on the three-dimensional models of the objects corresponding to the generated virtual viewpoint image (column 3, lines 46-54 notes the virtual viewpoint parameter assignment unit 105 assigns the virtual viewpoint parameter designated by the virtual viewpoint parameter designation unit 102 to the virtual viewpoint image data generated by the virtual viewpoint image generation unit 104 to thereby generate the second virtual viewpoint image data 130, and outputs it, the second virtual viewpoint image data 130 may be a one-frame still image, image data with one or more frames of still images, or image data of a moving image, e.g. where Figures 6A and 6B, column 7, lines 4-26 notes storing the virtual viewpoint parameters for respective pieces of virtual viewpoint image data of frames, and column 3, lines 37-45 notes virtual viewpoint image data may be generated by constructing a three-dimensional model and mapping image data, thus may be considered the virtual viewpoint parameter may be stored for the three-dimensional models of the objects corresponding to the generated virtual viewpoint image).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify Ogasawara’s system and method of generating a virtual viewpoint image with Ito’s method of storing (e.g. recording) virtual viewpoint parameters with the generated virtual viewpoint image to provide ease of obtaining information on the virtual viewpoint as desired (see column 8, lines 15-45 of Ito).
As to claim 2, Ogasawara modified with Ito disclose the one or more processors execute the instructions to record at least one of the time and the virtual viewpoint parameter as information used to specify the three-dimensional models of the objects (Ogasawara, Figure 6A, [0084] notes event information table 610 which shows the storage location of 3D model information for each of the objects for every time code of the captured event, Figure 6B, [0085] notes 3D model information table 620 managed by database 103 of information managed by database 103, [0086] notes in response to a certain time code being specified, 3D model information on each object related to the time code specified can be obtained using the information shown in Figures 6A and 6B; modified with Ito, e.g. Figures 6A, 6B, column 7, lines 4-26 notes a format in which virtual viewpoint parameters 602 to 605 for respective pieces of virtual viewpoint image data 606 to 609 of all frames are stored together in a header in the head of a file, and a format in which the virtual viewpoint parameters 602 to 605 for the pieces of virtual viewpoint image data 606 to 609 are stored immediately before the respective pieces of data, e.g. in a chronological order of frames).
As to claim 3, Ogasawara modified with Ito disclose the storage stores sequence data indicating three-dimensional models that respectively pertain to a plurality of times with respect to one or more objects, and the one or more processors execute the instructions to record information that specifies the sequence data used to generate the virtual viewpoint image as information on the three-dimensional models of the objects (Ogasawara, Figure 6A, [0084] notes event information table 610 which shows the storage location of 3D model information for each of the objects for every time code of the captured event, Figure 6B, [0085] notes 3D model information table 620 managed by database 103 of information managed by database 103, [0086] notes in response to a certain time code being specified, 3D model information on each object related to the time code specified can be obtained using the information shown in Figures 6A and 6B; modified with Ito, Figures 6A, 6B, column 7, lines 4-26 notes a format in which virtual viewpoint parameters 602 to 605 for respective pieces of virtual viewpoint image data 606 to 609 of all frames are stored together in a header in the head of a file, and a format in which the virtual viewpoint parameters 602 to 605 for the pieces of virtual viewpoint image data 606 to 609 are stored immediately before the respective pieces of data, e.g. in a chronological order of frames).
As to claim 4, Ogasawara modified with Ito disclose the one or more processors execute the instructions to record at least a part of the generated virtual viewpoint image (Ogasawara, e.g. as noted in claim 1, the generated virtual viewpoint image may be stored in database 103; modified with Ito, e.g. as noted for claim 3, Figures 6A, 6B, storing virtual viewpoint parameters 602 to 605 for respective pieces of virtual viewpoint image data 606 to 609 of frames).
As to claim 5, Ogasawara modified with Ito disclose the one or more processors execute the instructions to record user information of a user who has issued an instruction for generating the virtual viewpoint image (modified with Ito, column 8, lines 46-56 notes further storing identification information (such as a user name) of the user who set the virtual viewpoint, the date and time when the virtual viewpoint was set the date and time when the virtual viewpoint image data was generated, and/or the like).
As to claim 6, Ogasawara modified with Ito disclose the virtual viewpoint parameter includes at least one of a location and an orientation of the virtual viewpoint (modified with Ito, e.g. as noted in claim 1, the virtual viewpoint parameter may include one or more parameters among an orientation parameter 221 (e.g. indicating the orientation of the virtual camera), a position parameter 222 (e.g. indicating a position of the virtual camera), and a viewing angle parameter 223 (e.g. indicating the horizontal viewing angle of the virtual viewpoint)).
As to claim 13, Ogasawara modified with Ito disclose the one or more processors execute the instructions to, in response to satisfaction of the predetermined condition, record log information indicating that the virtual viewpoint image has been generated using the three-dimensional models of the objects at the time (Ogasawara, e.g. as noted in claim 1, the virtual viewpoint image generated using the three-dimensional model, where 3D model information may be stored, e.g. via a 3D model information table; modified with Ito, column 8, lines 46-56 notes further storing identification information (such as a user name) of the user who set the virtual viewpoint, the date and time when the virtual viewpoint was set the date and time when the virtual viewpoint image data was generated, and/or the like).
As to claim 18, Ogasawara modified with Ito disclose an information processing method comprising the steps as performed by the information processing system of claim 1. Please see the rejection and rationale of claim 1.
As to claim 19, Ogasawara modified with Ito disclose a non-transitory computer-readable medium storing a program (Ogasawara, e.g. Figure 2B, RAM 212 and/or ROM 213 of image generating apparatus 104 of information processing system 100) executable by a computer (Ogasawara, e.g. Figure 2B, CPU 211 of image generating apparatus 104 of information processing system 100)(Ogasawara, [0044]-[0046] notes CPU 211 performs processing as described using the programs and data stored in the RAM 212 and ROM 213) to perform a method comprising the steps as performed by the information processing system of claim 1. Please see the rejection and rationale of claim 1.
Claim(s) 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ogasawara (US 2022/0277520).
As to claim 14, Ogasawara discloses an information processing system (Figure 1A, information processing system 100) comprising one or more memories storing instructions (e.g. Figure 2B, RAM 212 and/or ROM 213 of image generating apparatus 104 of information processing system 100) and one or more processors that execute the instructions (e.g. Figure 2B, CPU 211 of image generating apparatus 104 of information processing system 100)([0044]-[0046] notes CPU 211 performs processing as described using the programs and data stored in the RAM 212 and ROM 213) to: obtain a three-dimensional model that is an inspection target ([0042] notes the modeling data generation unit 204 calculates and determines the rendering range or projection range of the virtual camera using the position and attitude of the virtual camera and the time code for the virtual viewpoint image, the modeling data generation unit 204 then determines a modeling data generation range based on the determined rendering range of the virtual camera, [0083] notes the modeling data generation unit 204 obtains, from the database 103, 3D models of objects included in the modeling data generation range); with reference to a record of generation of a virtual viewpoint image, obtain, as a three-dimensional model that is a comparison target, a three-dimensional model that has been used to generate the virtual viewpoint image in the past from a storage that stores three-dimensional models indicating three-dimensional shapes of objects ([0036], [0037] notes the 3D model generation unit 202 generates a 3D model representing the three-dimensional shape of an object within the region 120, [0038] notes these generated 3D models are stored in database 103 along with time codes, [0040] further notes the image generation unit 203 obtains 3D models from database 103 and generates a virtual viewpoint image based on the 3D models obtained, thus these 3D models have been used to generate the virtual viewpoint image in the past from database 103 and may be further obtained by the modeling data generation unit 204 as determined in the modeling data generation range as described above); and compare the three-dimensional model that is the inspection target with the three-dimensional model that is the comparison target (Figure 6A, [0084] notes event information table 610 which shows the storage location of 3D model information for each of the objects for every time code of the captured event, Figure 6B, [0085] notes 3D model information table 620 managed by database 103 of information managed by database 103, [0086] notes in response to a certain time code being specified, 3D model information on each object related to the time code specified can be obtained using the information shown in Figures 6A and 6B, [0087], [0088] notes the modeling data generation unit 204 refers to the 3D model information on objects associated with the time code specified, and determines for each of those objects whether the object is included in the modeling data generation range determined (e.g. performs comparison of 3D models in the modeling data generation range, which may include the 3D models used for generating the virtual viewpoint image)).
As noted above, Ogasawara described obtaining 3D models from database 103, where 3D models are generated and stored via the 3D model generation unit which are further obtained by the image generation unit to generate the virtual viewpoint image described. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the invention to recognize that the 3D models obtained by the modeling data generation unit may include the 3D models obtained by the image generation unit to generate the virtual viewpoint image, e.g. in the past, thus obtains a three-dimensional model that is an “inspection target” and a “comparison target” encompassed in the modeling data generation range, yielding predictable results, without changing the scope of the invention.
As to claim 15, Ogasawara discloses the one or more processors execute the instructions to store three-dimensional models that respectively pertain to a plurality of times with respect to one or more objects (e.g. storing 3D models with time codes as described below), the record of generation of the virtual viewpoint image includes at least one of the time and a virtual viewpoint parameter corresponding to the virtual viewpoint image (e.g. [0026] the virtual viewpoint image generated using the 3D models from database 103 and information on the viewpoint of the virtual camera, the viewpoint information on the virtual camera includes information indicating the position and orientation of the virtual viewpoint, more specifically, the viewpoint information includes a parameter representing the three-dimensional position of the virtual viewpoint and a parameter representing the orientation of the virtual viewpoint in a pan direction, a tilt direction, and a roll direction (“virtual viewpoint parameter”), [0028] notes the virtual viewpoint image generated by the image generating apparatus 104 may be stored in database 103), and the one or more processors execute the instructions to obtain the three-dimensional model that has been used to generate the virtual viewpoint image in the past with reference to at least one of the time and the virtual viewpoint parameter corresponding to the virtual viewpoint image (e.g. as noted in claim 14, 3D models are obtained (e.g. read) by image generation unit 203 from database 103 to generate the virtual viewpoint image, where the modeling data generation unit 204 further obtains, from the database 103, 3D models of objects included in the modeling data generation range (may include 3D models obtained by image generation unit 203 to generate the virtual viewpoint image), where Figure 6A, [0084] notes event information table 610 which shows the storage location of 3D model information for each of the objects for every time code of the captured event, Figure 6B, [0085] notes 3D model information table 620 managed by database 103 of information managed by database 103, [0086] notes in response to a certain time code being specified, 3D model information on each object related to the time code specified can be obtained using the information shown in Figures 6A and 6B).
As to claim 16, Ogasawara discloses the one or more processors execute the instructions to reconstruct the virtual viewpoint image that has been generated in the past using the three-dimensional model that is the comparison target ([0042] notes after determining a modeling data generation range based on the determined rendering range of the virtual camera, the modeling data generation unit 204 generates modeling data from 3D models included in the modeling data generating range, where [0030] notes modeling apparatus 106, e.g. a 3D printer, models a three-dimensional object of an object to be modeling indicated by the modeling data, e.g. provided via the modeling data generation unit 204 of image generating apparatus 104).
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ogasawara (US 2022/0277520) as applied to claim 14 above, and further in view of Ito (US 11,451,759).
As to claim 17, Ogasawara modified with Ito disclose the record of generation of the virtual viewpoint image is information recorded by the information processing system (Ogasawara, Figure 1A, information processing system 100) comprising one or more memories storing instructions (Ogasawara, e.g. Figure 2B, RAM 212 and/or ROM 213 of image generating apparatus 104 of information processing system 100) and one or more processors that execute the instructions (Ogasawara, e.g. Figure 2B, CPU 211 of image generating apparatus 104 of information processing system 100)(Ogasawara, [0044]-[0046] notes CPU 211 performs processing as described using the programs and data stored in the RAM 212 and ROM 213) to perform the steps as performed by the information processing system of claim 1. Please see the rejection and rationale of claim 1.
Allowable Subject Matter
Claims 6-12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding each of claims 6-12, the prior art of record fails to teach or suggest, singly or combined, the limitations of each of the claims as recited.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yoshikawa et al. (US 11,494,975) disclose a system and method for analyzing a three-dimensional model of an object which includes: obtaining the three-dimensional model generated based on images of the object, the images being imaged by respective cameras from respective viewpoints, the three-dimensional model including three-dimensional points each of which indicating a position of the object; obtaining a camera parameter of one camera among the respective cameras; generating, based on the camera parameter and the three-dimensional model, a depth image indicating a distance between the one camera and the object; generating a foreground image indicating an area in which the object is present in the one image imaged by the one camera; comparing the depth image and the foreground image to determine whether there is a deficiency of a three-dimensional point in the three-dimensional model; and outputting deficiency information if it is determined that there is the deficiency of the three-dimensional point in the three-dimensional model;
Yoneda (US 2022/0051469) discloses a system and method of obtaining virtual viewpoint information indicating at least a position of a virtual viewpoint and a view direction from the virtual viewpoint and generating a virtual viewpoint image based on the virtual viewpoint information and a plurality of images captured from a plurality of viewpoints by hiding processing a specific region of the virtual viewpoint image, the specific region being determined based on that the obtained virtual viewpoint information satisfies a condition;
Ota (US 2021/0224322) discloses a system and method of accumulating virtual viewpoint video image data generated based on image data obtained by capturing an object from a plurality of directions by a plurality of cameras and a virtual viewpoint parameter used for generation of the virtual viewpoint video image data in association with each other, then the image search system extracts, in a case where a search condition is input via an input unit, virtual viewpoint video image data associated with a virtual viewpoint parameter corresponding to the search condition from the accumulated virtual viewpoint video image data, the image search system further presents information of the extracted virtual viewpoint video image data as results of the search;
Adachi (US 2020/0394394) discloses a system comprising a first obtaining unit configured to obtain a first three-dimensional model expressing a position and a shape of a first object in a three-dimensional space based on images of the first object captured by a plurality of image-capturing devices at different locations; a second obtaining unit configured to obtain a second three-dimensional model expressing a position and a shape of a second object in the three-dimensional space; an identifying unit configured to identify a plurality of parts of the first object; a determining unit configured to determine a contact between a certain part of the first object and the second object based on the obtained first three-dimensional model and second three-dimensional model and an identification result of the plurality of parts; and an output unit configured to output information according to a determination result of the contact; and
MARANZANA et al. (US 2017/0004621) disclose a system and method of searching and comparing 3D models based on descriptors.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACINTA M CRAWFORD whose telephone number is (571)270-1539. The examiner can normally be reached 8:30a.m. to 4:30p.m.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACINTA M CRAWFORD/Primary Examiner, Art Unit 2617
Prosecution Insights