DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 13 is objected to because of the following informalities: at the end of the “displaying a first prompt interface…” clause, there is both a comma and a semicolon next to each other (“,:”). The comma should be removed. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 9 and 11-18 are rejected under 35 U.S.C. 103 as being unpatentable over Sarkis (U.S. Patent App. Pub. No. 2016/0005211; cited in IDS) in view of Tsoref (U.S. Patent App. Pub. No. 2023/0410364).
Regarding claim 1:
Sarkis teaches: a method (claim 1, a method) comprising:
obtaining a first image sequence by photographing around a target object (Fig. 6: 601, capturing a sequence of images of an object (a target object));
obtaining an initial three-dimensional (3D) model of the target object according to 3D modeling based on the first image sequence (Fig. 6: 601, generate a 3D model of the object based on the captured sequence of images);
obtaining, based on a missing region of the initial 3D model… (a second image sequence) … filling, based on the second image sequence, the missing region to generate a target 3D model (Fig. 6: 606 based on a detected anomaly, which, per para. 39 can be a missing or incomplete region, Sarkis teaches, at paras. 40-44, a user-selectable option to capture additional images (a second image sequence) of the object to correct the anomaly or missing region).
However, Sarkis does not teach obtaining, based on a missing region a camera pose sequence for performing photographing on the target object;
obtaining a second image sequence by photographing on the target object based on the camera pose sequence. Consider the following.
In analogous art, Tsoref teaches “enrollment images” which comprise “images taken from many different camera poses, comprising spatial positions around and/or within an example of the item of manufacture” (para. 115) to generate 3D models (Fig. 1A). Tsoref also teaches that it is known to identify or obtain camera poses that would be useful in model generation (here, the purpose of Tsoref is inspection of target objects). See Fig. 1B: 105, 106, 108, 110. Tsoref also teaches obtaining different or changed camera poses to “provide an image with increased suitability for enrolling the identified inspection target compared to the initial camera pose” (quoting part of claim 2. See also claim 3 and para. 121, which also teaches camera poses to supplement images deemed missing).
Modifying Sarkis, such that to take additional images to address anomalies such as missing regions, is done via determining camera poses that would be useful to achieve such goal, per Tsoref (i.e. would also make inspection more useful, by the teachings of Tsoref), is all of taught and suggested by the prior art, and would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Additional motivation would be to aid in obtaining further images that are deemed beneficial to end goals (i.e. a properly complete inspection or otherwise obtaining a complete 3D model of the target object).
The prior art included each element recited in claim 1, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as mapped and described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 2:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the method of claim 1, wherein obtaining the camera pose sequence comprises: determining, based on a center of the missing region, a first camera pose; and
determining, based on the first camera pose, the camera pose sequence, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Tsoref teaches obtaining a series of images by positioning a camera at the center as a first camera pose, and determining a pose sequence based on the first (Fig. 2A-C, para. 181). Modifying the applied references, in view of Tsoref, such to use this pose technique to obtain pose and images of the missing region, as mapped in claim 1, is taught and suggested by the prior art, and would have been obvious and predictable to one of ordinary skill in the art.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 9:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the method of claim 1, wherein filling the missing region comprises: generating, based on the second image sequence, a second 3D point cloud; and
filling, based on the second 3D point cloud, the missing region to generate the target 3D model, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Sarkis teaches alternate generation of a 3D point cloud and/or a 3D model. Modifying the applied references, in view of Sarkis, such that a 3D point cloud is generated to use as region information to fill the missing region, is all of taught and suggested by the prior art, with additional motivation as point cloud data is used to generate 3D models, also taught by Sarkis, and would have been obvious to one of ordinary skill.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 11:
Sarkis teaches the method of claim 1, further comprising determining that the initial 3D model comprises the missing region when the initial 3D model comprises a side used by only one triangular face of the initial 3D model (this defines a missing region as a second triangular face (missing) that would share the side/edge with a first triangular face) or
identifying that one of a plurality of vertices of the initial 3D model has a quantity of corresponding sides that is not equal to a quantity of corresponding triangular faces.
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to identify and address different types of anomalies in image models, to achieve desired results.
Regarding claim 12:
Sarkis teaches: the method of claim 1, wherein the missing region comprises a hole (e.g. para. 44).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied reference(-s) in view of same to have obtained the above, motivated to identify and address different types of anomalies in image models, to achieve desired results.
Regarding claim 13: please see also claim 1.
It would have been obvious for one of ordinary skill in the art to have combined and modified the applied reference(-s), in view of same, to have obtained: a method (Sarkis, claim 1, a method) comprising:
obtaining a first image sequence by photographing a target object (see mapping in claim 1);
obtaining an initial three-dimensional (3D) model of the target object according to 3D modeling based on the first image sequence (see mapping in claim 1);
displaying a first prompt interface when the initial 3D model comprises a missing region, wherein the first prompt interface is asking a user whether to perform supplementary photographing on the target object, (Sarkis, paras. 39-40, display an interface with selectable options to correct an anomaly, such as a missing region. One of the options can be capturing additional images (i.e., perform supplementary photography));
obtaining, on the first prompt interface, a first confirmation operation (Id. user can select an option, which can be confirming the capturing of additional images);
displaying, in response to the first confirmation operation, prompt information guiding the user to perform photographing on the target object (e.g. Sarkis, Figs. 3-5, information guiding user to perform photographing),
wherein the prompt information is based on a camera pose sequence for performing photographing on the target object (see mapping to claim 1, Tsoref teaches determining a camera pose sequence for obtaining desired images of a target object), and
wherein the camera pose sequence is based on the missing region (see also mapping to claim 1);
photographing a second image sequence for the target object in response to a photographing operation (see mapping to claim 1);
filling, based on the second image sequence, the missing region to obtain a target 3D model (see mapping to claim 1); and
displaying the target 3D model (Sarkis, Fig. 6: 610, 612).
Modifying the applied references, such that the prompt information to guide a user to perform photographing, per Sarkis, is based on a camera pose sequence, per Tsoref and mapped in claim 1, is all of taught and suggested by the prior art, and would have been obvious to one of ordinary skill.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 14:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the method of claim 13, wherein the prompt information comprises a first rendered image (Sarkis, Fig. 5),
wherein the first rendered image comprises a contour marking line of the target object (Sarkis, Fig. 5, see right panel of images),
wherein a first transparency of a first region framed by the contour marking line is a preset value (Sarkis, Fig. 5, firs region can be outer or inner surface),
wherein the preset value is between a minimum transparency and a maximum transparency (present transparency between two limits is an obvious design choice),
wherein the first rendered image is based on a second rendered image that is based on the camera pose sequence and the initial 3D model (Sarkis Fig. 5), and
wherein displaying the prompt information comprises displaying the first rendered image on a photographing interface (Sarkis, Fig. 5), and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
The prior art included each element recited in claim 14, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as mapped and described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 15:
Sarkis teaches: the method of claim 14, wherein a second transparency of a second region outside the first region in the first rendered image is the maximum transparency (see Fig. 5, there are two regions of the cup, and two different transparencies. The embodiment of Fig. 15 would be an obvious design choice to one of ordinary skill. Applicant’s specification describes no criticality to a specific transparency to the claimed invention).
Regarding claim 16:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the method of claim 13, further comprising:
displaying, after photographing the second image sequence, a second prompt interface asking the user whether to fill the missing region;
obtaining, on the second prompt interface, a second confirmation operation; and
displaying, in response to the second confirmation operation, the target 3D model, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Sarkis teaches prompts and an interface to allow a user ability to provide input and control generation of the 3D model (see e.g. paras. 39-41, 44, 57, 70 and Figs. 3-6). Modifying Sarkis, to have included a prompt to ask a user whether they want to fill the missing region as a selectable option (e.g. para. 40), and then upon confirmation, displaying the target 3D model (Fig. 6), is taught/suggested by Sarkis to allow a user to control model generation.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 17: please see also claim 1.
Sarkis teaches: an electronic device (Fig. 9: 900 device) comprising:
a memory configured to store instructions (Fig. 9: 932 memory, claim 28, store instructions); and
one or more processors coupled to the memory (Fig. 9: 910 processor coupled to memory), wherein when executed by the one or more processors, the instructions cause the electronic device to (para.69-70 and claim 28):
The instructions of claim 17 correspond to the method of claim 1; the same rationale for rejection applies.
Regarding claim 18: please see claim 2.
These claims are similar; the same rationale for rejection applies.
Allowable Subject Matter
Claims 3-8, 10, 19 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 3 and similarly claim 19 (as well as claims 4-6 and 20, which variously depend from claims 3 or 19, the prior art does not disclose, and would not have rendered obvious, the combination of features as per claim 3:
Claim 3:
the method of claim 1, wherein the initial 3D model comprises vertices,
wherein the vertices comprise edge vertices located at edges of the missing region, and wherein obtaining the camera pose sequence comprises:
determining, based on a center of the missing region, a first camera pose;
determining, second camera poses corresponding to the edge vertices, wherein one edge vertex of the edge vertices corresponds to one of the second camera poses; and
determining, based on the first camera pose and the second camera poses, the camera pose sequence.
The closest prior art to the above claim features is of record and cited herein. While Tsoref does teach determining centers of regions for image capture (see e.g. Figs. 2A-2D), as well as determining camera poses (e.g. Figs. 1B and 7), the instant reference does not teach determining second camera poses corresponding to edge vertices, whereby one edge vertex corresponds to one of the second camera poses, as recited in claim 3.
Likewise, with regard to claim 7:
Claim 7:
the method of claim 2, wherein the initial 3D model comprises vertices, wherein the vertices comprise edge vertices located at edges of the missing region,
wherein the first camera pose comprises a camera orientation and a first photographing distance, and wherein the first camera pose method further comprises:
determining, based on the edge vertices, the center;
determining, based on a direction from the center to a second center of the initial 3D model, the camera orientation; and
determining, based on a second photographing distance corresponding to the first image sequence, the first photographing distance.
The prior art does not teach the above claim features as they relate to edge vertices, edges of the missing region, determining the center based on same, and determining a camera orientation based on a direction from a center to a second center, in addition to the remaining determining steps of claim 7.
Similarly regarding claims 8 and 10:
Claim 8:
the method of claim 1, further comprising:
performing, based on the camera pose sequence and the initial 3D model, rendering to obtain first rendered images corresponding to one photographing angle of view, and
wherein the photographing angles of view correspond to one camera pose in the camera pose sequence;
marking contours of the target object in the first rendered images to obtain second rendered images, wherein the second rendered images comprise contour marking lines; and
adjusting transparencies of regions framed by the contour marking lines to a preset value to obtain third rendered images,
wherein the preset value is between a minimum transparency and a maximum transparency, and wherein the second image sequence is based on the third rendered images.
The prior art does not render obvious the above features as they relate to distinguished photographing angle of view and a camera pose, with a correspondence therebetween. Sarkis does teach contours and transparencies in rendered image (see Fig. 5), but does not teach the sequence of first to third rendered images in relation to transparency adjustments.
Claim 10:
the method of claim 9, wherein the initial 3D model comprises vertices that form a first 3D point cloud based on the first image sequence,
wherein the vertices comprise edge vertices located at edges of the missing region, and wherein filling the missing region comprises:
performing point cloud fusion on the second 3D point cloud and the first 3D point cloud to obtain an intermediate 3D model,
wherein the intermediate 3D model comprises a third 3D point cloud that is located in a second missing region of the intermediate 3D model, and
wherein the third 3D point cloud is a subset of the second 3D point cloud;
gridding the edge vertices and points comprised in the third 3D point cloud to obtain a gridded intermediate 3D model; and
generating, based on the gridded intermediate 3D model, the target 3D model.
While Sarkis does teach point cloud generation in addition to 3D model generation (e.g. para. 5), the instant reference, alone or in varied combination with remaining references of record, the above features as they relate to generation of an intermediate model, point cloud fusion, a second and third 3D point cloud, and generation of a target model based on a gridded intermediate model.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sarah Lhymn whose telephone number is (571)270-0632. The examiner can normally be reached M-F, 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Sarah Lhymn
Primary Examiner
Art Unit 2613
/Sarah Lhymn/Primary Examiner, Art Unit 2613
Prosecution Insights