DETAILED ACTION
1. Claims 1-9, and 11-12 are pending in this application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §102 and §103 (or as subject to pre-AIA 35 U.S.C. §102 and §103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Amendment
3. This office action is in response to applicant’s amendment filed on 02/17/2026 in response to the non-final action mailed on 11/21/2025. Claims 1 and 12 have been amended. Claims 2-9 have been kept original. Claim 10 have been cancelled. Amendment has been entered.
Response to Arguments
4. The applicant has not confirmed the or denied that the claims interpretation under 35 U.S.C. 112(f). The claims interpretation will be maintained.
Applicant's arguments, filed on 02/17/2026, with respect to the rejection of claims
1-9, and 11-12 under 35 U.S.C. §101 (abstract idea - mental process) (Applicant’s arguments, pages 8-19), have been fully considered and are but are not persuasive. The amended claims do not overcome the 35 U.S.C. §101 (abstract idea - mental process).
The rejection analysis follows the Revised Patent Subject Matter Eligibility Guidance (2019 PEG).
Applicant Argues that “Applicant's independent claims 1 and 12 are directed to an improvement in the functioning of a computer. Further, Applicant respectfully submits that generating a multi-modal image from each viewpoint of a 3D object for a plurality of viewpoints of the 3D object, and then generating an image for each viewpoint from the respective multi-modal image determined for each viewpoint is too complex to be practically performed in the human mind or with a pen and paper.” Examiner Reposefully disagrees.
A human can mentally observe data and visualize a multi-modal image (an image integrating information from multiple senses or data types) from that observed data. For example, a human can read a list of dry data (ingredients: 2 cups flour, 1 tsp salt, 350°F oven). As you process this, your mind visualizes a multi-modal image of the final golden-brown bread, its warm smell, and the crusty texture.
A human can observe a collection of image data and look for pieces/parts of the image data. For example, a human reviews a set of photos of manufactured parts (e.g., printed circuit boards or car doors) to identify defective pieces, such as scratches, dents, or missing components.
There is nothing so complex in the limitations that could not be doing in the human mind.
It is noted that the Applicant mentioned several parts of the specification in their argument; however, the details described therein are not part of the claimed elements. Therefore, arguments linked to those specification parts are moot. The possible improvements described by the applicant in the arguments must reflect both the specification and the claims. See MPEP 2106.05(a)
Applicant Argues that “That is, the searching accuracy can be improved by generating the multi-modal image from the 3D object generated using a predetermined algorithm or the like, generating an image from the multi-modal image, and using the image generated from the multimodal image for the search.” However, neither the claims or the specification describes the futures of the predetermined algorithm or the like. It appears that the applicant has an idea of a possible solution but has not described it in detail in the claims or specification. See MPEP 2106.05(f)(1)– “Whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished.”.
The Applicant cite the court cases of “SRI Int'/, Inc. v. Cisco Systems, Inc. CyberSource, …; BASCOM Global Internet Services, Inc., v. AT&T Mobility LLC, AT&T Corp.; and Ancora Techs. v. HTC Am., Inc.” and argued that the claims cannot be accomplish in the human mind. First, each application is examined on its own merit. Second, the claims in the cited court cases are slightly different from the examined claims.
It is also noted that the courts do not distinguish between claims that recite mental processes performed by humans and claims that recite mental processes performed on a computer. See MPEP 2106.04(a)(2)(III) – Mental Processes.
Applicant Argues that “Applicant's claims 1 and 12, when considered as a whole, integrate the alleged abstract idea into a practical application at least because claims 1 and 12 recite elements that improve the operation of a computer by increasing the searching accuracy of the computer and enabling the
computer to perform a 3D object search without having to train a model to search for the particular 3D object.” However, the claims do not describe in detail how such improvements in operation and increases in accuracy are accomplished. See MPEP 2106.05(f)(1)– “Whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished.”.
For those reasons claims 1-9 and 12 remain rejected under 35 U.S.C. §101 (abstract idea - mental process). See the rejection below.
Applicant's arguments, filed on 06/30/2023, with respect to the rejection of claims 1-9, and 11-12 under 35 U.S.C. §103 (Applicant' s arguments, pages 19-24), have been fully considered and are but are moot because the independent claims are amended and introduce new limitations that were not previously presented newly found prior art has been applied.
Claim Interpretation
5. The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f), is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f):
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f), is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f). The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f), is rebutted when the claims limitations recite function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f), except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f), except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f), because the claims limitations use a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
“a 3D object generation unit configured to; an image generation unit configured to; an image feature extraction unit configured to; an image search unit configured to; an image search result integration unit configured to” in independent claim 1.
“a display unit configured to; an input unit configured to;” in dependent claim 4.
“a registration unit configured to; ” in dependent claim 5.
It is noted that the deepened claims 2 and 7-8 also recite “the image search unit”.
It is noted that the deepened claims 2-4 and 9 also recite “the image search result integration unit”.
It is noted that the deepened claim 5 also recites “the 3D object generation unit”.
It is noted that the deepened claim 5 and 10-11 also recite “the image generation unit”.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid it/them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 112
6. The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims limitation “a 3D object generation unit configured to generate an input 3D object from an input image;” invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The specification fails to describe a structure to perform the function of the 3D object generation unit. While it notes that the process is performed by a known algorithm, it does not specify which known algorithm is being used. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claim Rejections - 35 USC § 101
7. 35 U.S.C. §101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 are rejected under 35 U.S.C. §101 because the claimed invention is directed to an abstract idea (Mental Process) without significantly more. The claims similarly describe a search system and method for searching for a 3D object.
The following is an analysis based on 2019 Revised Patent Subject Matter Eligibility Guidance (2019 PEG).
Step 1, Statutory Category?
Claims 1-9 and 11 are directed to a system.
Claim 12 is directed to a method.
Therefore, claims 1-9, 11 and 12 fall into at least one of the four statutory categories.
Step 2A, Prong I: Judicial Exception Recited?
The examiner submits that the foregoing claim limitations constitute a “Mental Process”, as the claims cover performance of the limitations in the human mind, given the broadest reasonable interpretation.
As per claims 1 and 12, the claims similarly recite the limitations of:
“a 3D object generation unit configured to generate an input 3D object from an input image;” A human observes an image and draws a 3D object. For example, when a human observes a 2D image and draws a 3D object, they are using their knowledge of perspective and visual cues to translate depth information onto a flat surface. The 3D object generation unit is merely an element used as a tool to implement an abstract concept. There is nothing so complex in the limitation that could not be doing in the human mind.
“an image generation unit configured to generate, as a search image group, a plurality of images obtained by viewing the input 3D object from a plurality of viewpoints;” A human can observe a collection of image data and look for pieces/parts of the image data. The image generation unit is merely an element used as a tool to implement an abstract concept. There is nothing so complex in the limitation that could not be doing in the human mind.
“wherein for each viewpoint, the image generation unit is configured to generate a multi-modal image from the input 3D object” A human can mentally observe data and visualize a multi-modal image (an image integrating information from multiple senses or data types) from that observed data. There is nothing so complex in the limitation that could not be doing in the human mind.
“an image feature extraction unit configured to calculate an image feature from each of the images in the search image group;” A human can observe an image and make a judgment of it to identify a specific feature of the image. The image feature extraction unit is merely an element used as a tool to implement an abstract concept. There is nothing so complex in the limitation that could not be doing in the human mind.
“an image search unit configured to search, by using each of the images in the search image group as a search query, a database in which a plurality of images obtained by viewing a plurality of 3D objects as search targets from a plurality of viewpoints are registered;” A human can observe a collection of images and select a few to be used as criteria to search for other images. The database, plurality of images and viewing a plurality of 3D objects as search targets from a plurality of viewpoints are registered are merely an element used as a tool to implement an abstract concept. There is nothing so complex in the limitation that could not be doing in the human mind.
As per dependent claim 2, the claim recites the limitations of:
“wherein each of the images registered in the database is associated with identification information of the 3D object as the search target,” The identification information of the 3D object as the search target is merely an element used to implement the abstract idea.
“the image search unit obtains a similarity to the search query for a plurality of images which are search results for the search query,” A human can observe a few images and mentally obtain a visualization of the similar portions of the images. There is nothing so complex in the limitation that could not be doing in the human mind.
As per dependent claim 3, the claim recites the limitation of:
“wherein the image search result integration unit adds up the similarities of the images associated with the identification information of the same 3D object by applying a predetermined weight,” A human can mentally visualize an image and apply criteria such as a predetermined weight to get better similarity results. There is nothing so complex in the limitation that could not be doing in the human mind.
“the weight is determined according to a difference between a viewpoint of each of the images included in the search image group and a viewpoint of the input image.” A human can observe differences between images and, based on the observations, determine a weight for the image. There is nothing so complex in the limitation that could not be doing in the human mind.
As per dependent claim 4, the claim recites the limitation of:
“wherein the image search result integration unit adds up the similarities of the images associated with the identification information of the same 3D object by applying the specified weight.” A human can mentally visualize an image and apply criteria such as a predetermined weight to get better similarity results. There is nothing so complex in the limitation that could not be doing in the human mind.
As per dependent claim 6, the claim recites the limitation of:
“wherein the database associates 3D object identification information with 3D object data indicating a 3D object as the search target, and associates the 3D object identification information, an image feature, and a viewpoint position with image data indicating an image as the search target.” A human can mentally observe images and associate information with the observed mentally. Humans have the capacity for mental imagery, which means they can mentally recreate and manipulate images in their minds even when the original image is no longer present. There is nothing so complex in the limitation that could not be doing in the human mind.
As per dependent claim 7, the claim recites the limitation of:
“wherein the image search unit searches for a representative image among the images registered in the database, and the representative image is selected for each cluster obtained by dividing a feature space.” A human can mentally search for an image, or specific features within a mental representation, after observing a plurality of images. A human can mentally select and group images based on their location, and subsequently subdivide those selected images into further groups. There is nothing so complex in the limitation that could not be doing in the human mind.
As per dependent claim 8, the claim recites the limitation of:
“wherein the image search unit searches for a representative image among the images registered in the database, and the representative image is an image present in a region where density is less than a threshold in a feature space.” The representative image among the images registered in the database, representative image is the image present in the region and the density is less than a threshold in a feature space are merely instructions used to implement the abstract idea.
As per dependent claim 9, the claim recites the limitation of:
“wherein the image search result integration unit corrects, according to a degree of coincidence between a combination of the viewpoints in the search image group and a combination of viewpoints in the images associated with the identification information of the same 3D object, the similarity of the 3D object as the search target.” A human can mentally observe an image and, based on specific criteria or goals, make changes or transformations to that image in their mind. The human’s ability to mentally modify images is crucial for creative thinking and problem-solving tasks. There is nothing so complex in the limitation that could not be doing in the human mind.
As per dependent claim 11, the claim recites the limitation of:
“wherein the 3D object generation unit combines a plurality of 3D objects, and the image generation unit generates a plurality of images obtained by viewing a combination of the plurality of 3D objects from a plurality of viewpoints as a search image group.” A human can mentally observe images and visualize a combination of one or more of those images. A human can mentally create new images based on images they have previously visualized and observed. A human can mentally combine previously visualized and observed images. There is nothing so complex in the limitation that could not be doing in the human mind.
Accordingly, claims 1-9 and 11-12 recite at least one abstract idea.
Step 2A, Prong II: Integrated into a Practical Application?
The claims recite the following additional limitations/elements:
As per independent claim 1, the claim recites the elements of:
“a 3D object generation unit, an image generation unit, an image feature extraction unit, an image search unit, a database, and an image search result integration unit” These elements are example of mere instruction to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (see MPEP § 2106.05(f)). Specifically, the additional elements of the limitations invoke computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general-purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) do not provide improvements to the functioning of a computer or to any other technology or technical field; and do not integrate a judicial exception into a practical application.
As per independent claim 1, the claim recites the limitation of:
“an image search result integration unit configured to integrate a search result for each of the images in the search image group for each of the 3D objects as the search targets, and output, as a similar 3D object, a 3D object similar to the input image among the 3D objects as the search targets.” Integrating a search result associated with a plurality of images is nothing more than gathering all the information about the plurality of images and displaying it. This limitation is example of adding insignificant extra-solution activity to the judicial exception (see MPEP § 2106.05(g)). Specifically, the additional limitation exemplifies mere data gathering, without any further processing or analysis. The image search result integration unit is simply an element used as a tool for gathering the search results (see MPEP § 2106.05(f)).
“output an image generated from the multi-modal image as an image in the search image group;” This limitation is example of adding insignificant extra-solution activity to the judicial exception (see MPEP § 2106.05(g)). Specifically, the additional limitation exemplifies mere data gathering and transmitting, without any further processing or analysis.
As per dependent claim 2, the claim recites the limitation of:
“the image search result integration unit adds up similarities of images associated with identification information of the same 3D object from search results of each of the images in the search image group to obtain a similarity of the 3D object.” Obtaining data from an image is a way to gather data. This limitation is example of adding insignificant extra-solution activity to the judicial exception (see MPEP § 2106.05(g)). Specifically, the additional limitation exemplifies mere data gathering, without any further processing or analysis.
As per dependent claim 4, the claim recites the limitations of:
“a display unit configured to display the input image and the images in the search image group;” Displaying data is a way of transmitting data that was gathered. This limitation is example of adding insignificant extra-solution activity to the judicial exception (see MPEP § 2106.05(g)). Specifically, the additional limitation exemplifies mere data gathering and transmitting, without any further processing or analysis. The display unit is a mere element used as a tool to implement the abstract idea (see MPEP § 2106.05(f)).
“an input unit configured to receive specification of a weight for the images in the search image group” This limitation is example of adding insignificant extra-solution activity to the judicial exception (see MPEP § 2106.05(g)). Specifically, the additional limitation exemplifies mere data gathering, without any further processing or analysis. The input unit is a mere element used as a tool to implement the abstract idea (see MPEP § 2106.05(f)).
As per dependent claim 5, the claim recites the limitation of:
“a registration unit configured to register images in the database, wherein when an image is received as the search target, the 3D object generation unit generates a 3D object as the search target, the image generation unit uses a 3D object received as the search target or the 3D object generated from the image received as the search target to generate a plurality of images obtained by viewing the 3D object from a plurality of viewpoints as a registration image group, and the registration unit associates the image group generated by the image generation unit with identification information of the 3D object as the search target, and registers the image group in the database.” Registering data into a database is nothing more than storing the data into the database. This limitation is example of adding insignificant extra-solution activity to the judicial exception (see MPEP § 2106.05(g)). Specifically, the additional limitation exemplifies mere data gathering, without any further processing or analysis. The registration unit is a mere element used as a tool to implement the abstract idea (see MPEP § 2106.05(f)).
As per independent claim 12, the claim recites the elements of:
“a database” This element is example of mere instruction to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea (see MPEP § 2106.05(f)). Specifically, the additional elements of the limitations invoke computers or other machinery merely as a tool to perform an existing process. Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general-purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) do not provide improvements to the functioning of a computer or to any other technology or technical field; and do not integrate a judicial exception into a practical application.
“output an image generated from the multi-modal image as an image in the search image group;” This limitation is example of adding insignificant extra-solution activity to the judicial exception (see MPEP § 2106.05(g)). Specifically, the additional limitation exemplifies mere data gathering and transmitting, without any further processing or analysis.
Therefore, claims 1-9 and 11-12 do not integrate the recited abstract ideas into a practical application.
Step 2B: Claim provides an Inventive Concept?
With respect to the limitations identified as insignificant extra-solution activity above the conclusions are carried over, and both the “outputting …; obtaining …; display …; receiving …; and … register images in the database …. ” is well-understood, routine, and conventional operations.
For support as being well-understood, routine, and conventional for “outputting …; obtaining …; display …; receiving …; and … register images in the database …. (store images in the database) ” as noted by the courts is well understood routine and conventional, see MPEP 2106.05(d)(ii) “i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); … buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network);” and/or MPEP 2106.05(d)(ii) “iv. Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93;”, and/or MPEP 2106.05(d)(II) “iii. Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755 (updating an activity log);” See also see [Display Interface - an overview | ScienceDirect Topics, Introducing ASP.NET Web Pages - Displaying Data | Microsoft Docs, Execute DBCC PAGE command to Display Contents of Data Pages in SQL Server (kodyaz.com) and Load and display paged data | Android Developers]; and Lei et al. (US 20250022111 A1) – “an image obtained by a conventional imaging display device and an image that can be displayed by the conventional imaging display device”; and He (US 20240357200 A1) – “performing image rendering on the plurality of multimedia materials according to the target color display standard, so as to display the plurality of multimedia materials according to the image rendering result or export the plurality of multimedia materials as a video file in a specific format according to the image rendering result.”.
Looking at the limitations in combination and the claim as a whole does not change this conclusion and the claim is ineligible.
Therefore, the claims 1-9 and 11-12 are not patent eligible.
Claim Rejections - 35 USC § 103
8. In the event the determination of the status of the application as subject to AIA 35 U.S.C. § 102 and § 103 (or as subject to pre-AIA 35 U.S.C. § 102 and § 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section § 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under pre-AIA 35 U.S.C. § 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claims 1-7 and 11-12 are rejected under 35 U.S.C. § 103 as being unpatentable over Besecker et al. (US 20230177832 A1) in view of Nagano et al. (US 20230081641 A1) in further view of Sreedhar et al. (US 20240029148 A1) still in further view of Tan et al. (US 20220101048 A1).
As per claim 1, Besecker teaches a search system (i.e. “Systems and methods for enhancing computer-generated visualization of items”; Abstract, para. [0004]; The search system is interpreted as the systems) comprising:
an image feature extraction unit (i.e. “Mobile computing device 102 (e.g., via image analyzer 110) may be configured to extract relevant feature data from the image”; fig. 1, para. [0057], [0079]; Examiner note: the image feature extraction unit is interpreted as the mobile computing device) configured to
calculate an image feature from each of the images in the search image group (i.e. “Mobile computing device 102 (e.g., via image analyzer 110) may be configured to extract relevant feature data from the image (e.g., resolution, the field of view (FOV), focal length, imaging center, etc.), as well as to measure (see, e.g., the IMU sensor 216) information usable to track the movement of mobile computing device 102 relative to the environment, such as position/orientation data.”; fig. 1, para. [0057]); Examiner note: the calculate an image feature from each of the images in the search image group is interpreted as the mobile computing device configured to measure (see, e.g., the IMU sensor 216) information usable to track the movement of mobile computing device 102 relative to the environment, such as position/orientation data);
an image search result integration unit (i.e. “a set of inspiration images 1710 is provided at a user interface”; fig. 17, para. [0154]-[0156]; Examiner note: the image search result integration unit is interpreted as the user interface) configured to
integrate a search result for each of the images in the search image group for each of the 3D objects as the search targets (i.e. “The products from the AR list link 1720 may be incorporated with products included in each inspiration image.”; fig. 17, para. [0154]-[0157]; Examiner note: where integrate is considered herein as the incorporated),
and output, as a similar 3D object, a 3D object similar to the input image among the 3D objects as the search targets (i.e. “At least some of the components of the inspiration image 1710 may be identified by an image analysis algorithm. For example, the image analysis algorithm can identify outlines of products in the image by identifying similar lines, textures, or other image qualities and attributing the similar pixels with stored images of room features or products (e.g., walls, floors, windows, couches, seats, curtains, etc.).” and “The user interface may be updated to display potential room layouts that the user can select and may also identify the products that have AR-enabled images corresponding with them.”; figs. 12, 17, para. [0156]-[0157]; Examiner note: the output is considered herein as the display potential room layouts; the similar 3D object, the 3D object similar to the input image among the 3D objects as the search targets is interpreted as the inspiration image 1710, see para. [0156]).
However, it is noted that the prior art of Besecker does not explicitly teach “a 3D object generation unit configured to generate an input 3D object from an input image; wherein for each viewpoint, the image generation unit is configured to generate a multi-modal image from the input 3D object, and output an image generated from the multi-modal image as an image in the search image group; an image search unit configured to search, by using each of the images in the search image group as a search query, a database in which a plurality of images obtained by viewing a plurality of 3D objects as search targets from a plurality of viewpoints are registered;”
On the other hand, in the same field of endeavor, Nagano teaches a 3D object generation unit (i.e. “a 2D convolutional neural network (CNN)”; fig. 1, para. [0018]; Examiner note: the 3D object generation unit is interpreted as the a 2D convolutional neural network (CNN)) configured to
generate an input 3D object from an input image (i.e. “a three-dimensional (3D) representation of the single 2D image is generated utilizing a 2D convolutional neural network (CNN).”; figs. 1, 6, para. [0018]-[0019], [0095]; Examiner note: the generate an input 3D object from an input image is interpreted as the three-dimensional (3D) representation of the single 2D image is generated);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nagano that teaches generating a three-dimensional model from a two-dimensional image into Besecker that teaches data lineage graphs. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to create 3D representations using a feedforward method, as it can improve the performance of computing hardware that implements 3D representation generation (Nagano, para. [0026]).
However, it is noted that the combination of the prior arts of Besecker and Nagano do not explicitly teach “wherein for each viewpoint, the image generation unit is configured to generate a multi-modal image from the input 3D object, and output an image generated from the multi-modal image as an image in the search image group; an image search unit configured to search, by using each of the images in the search image group as a search query, a database in which a plurality of images obtained by viewing a plurality of 3D objects as search targets from a plurality of viewpoints are registered;”
On the other hand, in the same field of endeavor, Sreedhar teaches an image search unit (i.e. “the server 110 may search the catalog to identify images of products ….”; figs. 1-2, para. [0047], [0049]; Examiner note: the image search unit is interpreted as the server) configured to
search, by using each of the images in the search image group as a search query (i.e. “The server 110 may identify, based on the received information, a number of candidate product images, such as image 220 shown in FIG. 2, from a retailer's product catalog that includes various images of the first type of product (e.g., images of multiple different products of the first type, such as accent chair).”; fig. 2, para. [0047], [0049]; Examiner note: the server is identifying number of candidate product images based on the candidate product image 220 herein the candidate product image 220 is interpreted to be used as a search query; where the identity is searching for possible candidate product images),
a database in which a plurality of images obtained (i.e. “multiple proxy product models may be placed in a user's space and a product image obtained from the server may include more than one product in the image. In some embodiments, images of the user's space may include existing images of the user's space (e.g., home) or images obtained from third-party sources.”; para. [0083]; Examiner note: the database is interpreted as the server herein)
by viewing a plurality of 3D objects as search targets (i.e. “the catalog 240 of images of products that is utilized to identify candidate product images may be stored at the server 110”; figs. 2, 13, para. [0056]; Examiner note: the viewing a plurality of 3D objects as search targets is interpreted as the catalog of images of products that is utilized to identify candidate product images)
from a plurality of viewpoints are registered (i.e. “An overview of the processing is shown in FIGS. 10A-10D. In some embodiments, the processing of a product silo or lifestyle image may include running it through a machine learning model (e.g., a regression-based machine learning model) that detects the angle the product is facing in the image, also referred to herein as the shot angle of the product, and labels it.”; fig. 2, para. [0056]; Examiner note: the plurality of viewpoints is registered is interpreted as the shot angle of the product, see fig. 10A-D where multiples shot angle of the product is illustrated);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface into the combination of the prior art of Besecker that teaches data lineage graphs, and Nagano that teaches generating a three-dimensional model from a two-dimensional image. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to use augmented reality (AR) to improve the shopping experience for customers because it can provide a visualization of the product in the physical scene (Sreedhar, para. [0026]).
However, it is noted that the combination of the prior arts of Besecker, Nagano and Sreedhar do not explicitly teach “an image generation unit configured to generate, as a search image group, a plurality of images obtained by viewing the input 3D object from a plurality of viewpoints, wherein for each viewpoint, the image generation unit is configured to generate a multi-modal image from the input 3D object, and output an image generated from the multi-modal image as an image in the search image group;”
On the other hand, in the same field of endeavor, Tan teaches an image generation unit (i.e. “FIG. 8 presents another example multimodal training data generation module 800 in accordance with one or more embodiments of the disclosed subject matter.”; fig. 8, para. [0125]; Examiner note: the image generation unit is interpreted as the multimodal training data generation module 800) configured to
generate, as a search image group (i.e. “At 1502, the annotation transfer component 114 can further transfer the projected ground truth data 1308 onto the corresponding native 2D image 1506 as shown in box 1504, resulting in the generation of the corresponding native 2D image 1508 with transferred ground truth data.”; fig. 15:1508, para. [0153]; Examiner note: the search image group is interpreted as the native 2D image 1508),
a plurality of images obtained (i.e. “generating the projected ground truth data 1308.”; fig. 15:1308, para. [0153]; Examiner note: the plurality of images obtained is interpreted as the ground truth data; the ground of truth data has a plurality of images)
by viewing the input 3D object (i.e. “Process 1300 involves transferring ground truth data 1302 for the native 3D image onto the corresponding synthetic 2D image 1304 to generate a corresponding synthetic 2D image 1316 with transferred ground truth data.”; figs. 13:1302, 15:1302, para. [0147]; Examiner note: the viewing the input 3D object is interpreted as the transferring ground truth data 1302 for the native 3D image)
from a plurality of viewpoints (i.e. “apply various augmented features and/or additional adjustments to the respective copies 1204 to generate variational training data images that are more representative of the different native 2D images that will be encountered in the field (e.g., to simulate different patient data).”; figs. 12:1204, 14:1404, 17, 20, para. [0141], [0149]; Examiner note: the plurality of viewpoints is interpreted as the copies 1204),
wherein for each viewpoint, the image generation unit is configured to generate a multi-modal image from the input 3D object (i.e. “This results in generation of annotated 2D image data 2024 in the target capture modality. For example, image 2026 corresponds to the annotated native 2D image and image 2028 corresponds to the annotated synthetic 2D image.”; fig. 20:2024, para. [0167]-[0169]; Examiner note: the multi-modal image is interpreted as the annotated 2D image data 2024. Further, “Process 2000 can be repeated for each patient for all the paired images from all the patients to obtain a high-quality, annotated images of the second modality and the resulting annotated synthetic and/or native 2D imaged data can be added to a training dataset 2303.”; para. [0170]), and
output an image generated from the multi-modal image as an image in the search image group (i.e. “the annotation component 106 can present the eSXR image with the transferred GT to one or more annotators for manual review and adjustment.”; figs. 17, 20, 22A-B, para. [0069], [0180]; Examiner note: the output the image generated is intpreted as the present the eSXR image. Further, i.e. “For example, in some embodiments, the GT data can include mark-up image data applied to the CT image data and transferred to the eSXR lung ROI image 2204 using the same projection parameters used to generate the eSXR image 2117.”; fig. 22B, para. [01786]; Examiner note: image in the search image group is interpreted as the eSXR lung ROI image 2204);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tan that teaches a training image data generation and usage thereof for developing mono-modality image inferencing models into the combination of the prior art of Besecker that teaches data lineage graphs, Nagano that teaches generating a three-dimensional model from a two-dimensional image, and Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to improve a quality and realistic appearance of the synthetic 2D output image using one or more pre-projection processing steps because it can significantly reduce the simulation-to-reality (sim2real) gap by correcting distortions, compensating for radiometric differences, and removing artifacts, allowing the synthetic data to closely mimic real-world visual characteristics (Tan, para. [0050]-[0051]).
As per claim 2, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 1 above.
Additionally, Besecker teaches wherein each of the images registered in the database is associated with identification information of the 3D object as the search target (i.e. “the extracted object identifiers may be used to guide the identification and/or selection of similarly functional products available within the local resource database 106.”; para. [0090], [0128]; Examiner note: the identification information of the 3D object is interpreted as the object identifiers),
the image search unit obtains a similarity to the search query for a plurality of images which are search results for the search query (i.e. “the extracted object identifiers may be used to guide the identification and/or selection of similarly functional products available within the local resource database 106.”; fig. 12, para. [0090], [0128], [0143]-[0144]; Examiner note: the image search unit obtains a similarity to the search query for a plurality of images which are search results for the search query is interpreted as the selection of similarly functional products available within the local resource database), and
the image search result integration unit adds up similarities of images associated with identification information of the same 3D object from search results of each of the images in the search image group to obtain a similarity of the 3D object (i.e. “For example, the user may select first product 1200A and second product 1200B as products to add to a favorite list. The selected products may correspond with a “heart” icon or other similar label 1210 (illustrated as first icon 1210A and second icon 1210B).” and “The selected products may be identified by selecting an “add to AR list” (or similarly labeled) icon (illustrated as first icon 1220A and second icon 1220B).”; fig. 12, para. [0143]-[0144]).
As per claim 3, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 2 above.
However, it is noted that the combination of the prior arts of Besecker, Nagano and Tan do not explicitly teach “wherein the image search result integration unit adds up the similarities of the images associated with the identification information of the same 3D object by applying a predetermined weight, and the weight is determined according to a difference between a viewpoint of each of the images included in the search image group and a viewpoint of the input image.”
On the other hand, in the same field of endeavor, Sreedhar teaches wherein the image search result integration unit adds up the similarities of the images associated with the identification information of the same 3D object by applying a predetermined weight (i.e. “the dimensions of the product may be determined to be compatible with the dimensions of the proxy product model when all the dimensions (e.g., width, height, and depth) match or match within a particular tolerance (e.g., +/−a certain threshold value).”; figs. 1-2, para. [0049]-[0050]; Examiner note: Examiner note: the applying the predetermined weight is interpreted as the particular tolerance), and
the weight is determined according to a difference between a viewpoint of each of the images included in the search image group and a viewpoint of the input image (i.e. “an image of a product taken at a 45 degree shot angle may be determined to be compatible with an image of a user's space taken at a 45 degree shot angle+/−a threshold tolerance.”’; para. [0049]-[0050]; Examiner note: the threshold tolerance is considered as the difference).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface into the combination of the prior art of Besecker that teaches data lineage graphs, Nagano that teaches generating a three-dimensional model from a two-dimensional image, and Tan that teaches a training image data generation and usage thereof for developing mono-modality image inferencing models. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to use augmented reality (AR) to improve the shopping experience for customers because it can provide a visualization of the product in the physical scene (Sreedhar, para. [0026]).
As per claim 4, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 2 above.
Additionally, Besecker teaches further comprising: a display unit (i.e. “a display”; fig. 2, para. [0069], [0085]) configured to display the input image and the images in the search image group (i.e. “Mobile computing device 102 may display a web page 502 that includes one or more AR-enabled images of products 504 (e.g., room décor, furniture, or other products).”’ figs. 5, 7, 9, 12, para. [0099]).
However, it is noted that the combination of the prior arts of Besecker, Nagano and Tan do not explicitly teach “an input unit configured to receive specification of a weight for the images in the search image group, wherein the image search result integration unit adds up the similarities of the images associated with the identification information of the same 3D object by applying the specified weight.”
On the other hand, in the same field of endeavor, Sreedhar teaches an input unit (i.e. “a mobile device”; para. [0004]-[0005]; Examiner note: the input unit is interpreted as the mobile device) configured to receive specification of a weight for the images in the search image group (i.e. “the user may set the dimensions of the proxy product model using a GUI element 1210 having a scrollable list of values for each of multiple dimensions (width, height, and depth, in this example).”; figs. 3, 5A-C, 15, para. [0058]-[0059], [0084]; Examiner note: the receive specification of the weight for the images in the search image group is interpreted as the user may set the dimensions of the proxy product model using a GUI element 1210 having a scrollable list of values for each of multiple dimensions (width, height, and depth, in this example)),
wherein the image search result integration unit adds up the similarities of the images associated with the identification information of the same 3D object by applying the specified weight (i.e. “A screenshot of a prompt, shown in FIG. 5L, enables the user to ensure that a marked space (e.g., a particular area around or in front of the proxy product model)—indicated by a dotted pattern on the surface—is clear to move around and confirm the space (for example, by clicking “Confirm Space”).”; figs. 5A-L, 12-13; para. [0070], [0084]-[0085]: Examiner notes: as illustrated the images is adjusted based on the input dimensions and space. It is considered that the fig. 5L has all the similarities adds up to the Same 3D Object herein illustrated as proxy product model 504).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface into the combination of the prior art of Besecker that teaches data lineage graphs, Nagano that teaches generating a three-dimensional model from a two-dimensional image, and Tan that teaches a training image data generation and usage thereof for developing mono-modality image inferencing models. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to use augmented reality (AR) to improve the shopping experience for customers because it can provide a visualization of the product in the physical scene (Sreedhar, para. [0026]).
As per claim 5, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 1 above.
Additionally, Besecker teaches further comprising: a registration unit (i.e. “the user may select fourth product 1200D and fifth product 1200E as products to add to AR list 1240”; fig. 12, [0144]; Examiner: the registration is interpreted as the AR list) configured to
register images in the database (i.e. “The (e.g., automatic) placement may identify within the local asset resource database 106 and/or remote resource database 122 one or more additional fill models which, when placed accordingly, may close the gap.”; fig. 12, para. [0142]-[0144], [0167]; Examiner note: the databases herein saving products),
wherein when an image is received as the search target (i.e. “For example, as illustrated in FIG. 14, a QR code 1410 may be presented at the interface of the first device to enable handoff from the first device to a second device.”; figs. 13-14, para. [0147]; Examiner: the image is received as the search target is interpreted as the QR code 1410 may be presented at the interface of the first device to enable handoff from the first device to a second device),
the 3D object generation unit generates a 3D object as the search target (i.e. “In any of the examples illustrated in FIGS. 14-16, upon scanning or activating the QR code, application 114 may be launched at mobile computing device 102 to allow the user to browse for products and visualize them in an AR-enabled environment.”; figs. 14-16, para. [0147]-[0148]; Examiner note: the generates the 3D object as the search target is interpreted as the generates activating the QR code allow the user to browse for products and visualize them in an AR-enabled environment),
the image generation unit uses a 3D object received as the search target or the 3D object generated from the image received as the search target to generate a plurality of images obtained by viewing the 3D object from a plurality of viewpoints as a registration image group (i.e. “illustrated in FIG. 16, a QR code 1610 may be presented at the interface of the computing device as an overlay or adjacent to product information. The QR code may be provided concurrently with product information in the overlay.”; fig. 16, para. [0149]; Examiner note: the 3D object generated from the image received as the search target to generate a plurality of images obtained by viewing the 3D object from a plurality of viewpoints as a registration image group is interpreted as the QR code 1610 may be presented at the interface of the computing device as an overlay or adjacent to product information), and
the registration unit associates the image group generated by the image generation unit with identification information of the 3D object as the search target (i.e. “At least some of the components of the inspiration image 1710 may be identified by an image analysis algorithm. For example, the image analysis algorithm can identify outlines of products in the image by identifying similar lines, textures, or other image qualities and attributing the similar pixels with stored images of room features or products (e.g., walls, floors, windows, couches, seats, curtains, etc.).”; figs. 16-17, para. [0149], [0156]), and
registers the image group in the database (i.e. “the products in the inspiration image 1710 (and AR list 1720, if they are selected) may be placed in the user's AR-room in accordance with the rules corresponding with a room layout (e.g., an L-shape, G-shape, T-shape, and the like).”; fig. 17, para. [0157]; Examiner note: the registers the image group in the database is interpreted as the products in the inspiration image is placed in the user's AR-room in accordance with the rules corresponding with a room layout).
As per claim 6, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 1 above.
Additionally, Besecker teaches wherein the database associates 3D object identification information with 3D object data indicating a 3D object as the search target, and associates the 3D object identification information, an image feature, and a viewpoint position with image data indicating an image as the search target (i.e. “At least some of the components of the inspiration image 1710 may be identified by an image analysis algorithm. For example, the image analysis algorithm can identify outlines of products in the image by identifying similar lines, textures, or other image qualities and attributing the similar pixels with stored images of room features or products (e.g., walls, floors, windows, couches, seats, curtains, etc.). The image analysis algorithm may match the identified products and colors in the inspiration image with a stored inspiration image. When a match is determined (e.g., based on a match score comparing the received and stored images), the stored inspiration image with the highest match rate may be identified to retrieve additional information and metadata about the received inspiration image.”; fig. 17, [0156]-[0157]).
As per claim 7, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 1 above.
Additionally, Besecker teaches wherein the image search unit searches for a representative image among the images registered in the database (i.e. “The user may save one or more AR-enabled images of products using the “add to AR list” feature while searching for products on a desktop and then later transfer the list/images locally to mobile computing device 102 in the local resource database 106 (e.g. when the user is in the room for which they are shopping or at other times).”; para. [0151]-[0152]; Examiner note: where the AR list have a plurality of images), and
the representative image is selected for each cluster obtained by dividing a feature space (i.e. “This may help enable a selective display provided via the user interface. In some examples, the selective display enables the user to select one of the AR-enabled image to selectively display an AR-enabled view of the corresponding product in room.”; fig. 17, para. [0153]. Further, i.e. “For example, the user may design and/or configure a virtual environment comprising a virtual living space.”; para. [0058], [0094], [0161]-[0162]; the feature space is interpreted as the virtual living space).
As per claim 11, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 1 above.
However, it is noted that the combination of the prior arts of Besecker, Sreedhar and Tan do not explicitly teach “wherein the 3D object generation unit combines a plurality of 3D objects, and the image generation unit generates a plurality of images obtained by viewing a combination of the plurality of 3D objects from a plurality of viewpoints as a search image group.”
On the other hand, in the same field of endeavor, Nagano teaches wherein the 3D object generation unit combines a plurality of 3D objects (i.e. “Volumetric rendering 508 is then run on the 3D representation to aggregate neural features within one or more viewing directions, and the aggregated neural features are provided as input to a multilayer perceptron (MLP) 510 that predicts and outputs color and density values 512 for each point within the 3D representation.”; fig. 6, para. [0092], [0118]), and
the image generation unit generates a plurality of images obtained (i.e. “a three-dimensional (3D) representation of a single 2D image is generated utilizing a 2D convolutional neural network (CNN).”; fig. 6, para. [0095])
by viewing a combination of the plurality of 3D objects from a plurality of viewpoints as a search image group (i.e. “Further, as shown in operation 606, the realism of a rendering of the 3D representation by the 2D CNN from one or more different viewpoints shown within the single 2D image is ensured by combining GAN losses.”; fig. 6, para. [0095]; Examiner note: the viewing the combination of the plurality of 3D objects from a plurality of viewpoints as a search image group is interpreted as the rendering of the 3D representation by the 2D CNN from one or more different viewpoints shown within the single 2D image is ensured by combining GAN losses).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nagano that teaches generating a three-dimensional model from a two-dimensional image into the combination of the prior art of Besecker that teaches data lineage graphs, Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface, and Tan that teaches a training image data generation and usage thereof for developing mono-modality image inferencing models. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to create 3D representations using a feedforward method, as it can improve the performance of computing hardware that implements 3D representation generation (Nagano, para. [0026]).
As per claim 12, Besecker teaches a search method (i.e. “Systems and methods for enhancing computer-generated visualization of items”; Abstract, para. [0004]; The search system is interpreted as the systems) comprising:
an image feature calculation step of (i.e. “Mobile computing device 102 (e.g., via image analyzer 110) may be configured to extract relevant feature data from the image”; fig. 1, para. [0057], [0079]; Examiner note: the image feature extraction unit is interpreted as the mobile computing device)
calculating an image feature from each of the images in the search image group (i.e. “Mobile computing device 102 (e.g., via image analyzer 110) may be configured to extract relevant feature data from the image (e.g., resolution, the field of view (FOV), focal length, imaging center, etc.), as well as to measure (see, e.g., the IMU sensor 216) information usable to track the movement of mobile computing device 102 relative to the environment, such as position/orientation data.”; fig. 1, para. [0057]); Examiner note: the calculate an image feature from each of the images in the search image group is interpreted as the mobile computing device configured to measure (see, e.g., the IMU sensor 216) information usable to track the movement of mobile computing device 102 relative to the environment, such as position/orientation data);
an image search result integration step of (i.e. “a set of inspiration images 1710 is provided at a user interface”; fig. 17, para. [0154]-[0156]; Examiner note: the image search result integration unit is interpreted as the user interface)
integrating a search result for each of the images in the search image group for each of the 3D objects as the search targets (i.e. “The products from the AR list link 1720 may be incorporated with products included in each inspiration image.”; fig. 17, para. [0154]-[0157]; Examiner note: where integrate is considered herein as the incorporated),
and outputting, as a similar 3D object, a 3D object similar to the input image among the 3D objects as the search targets (i.e. “At least some of the components of the inspiration image 1710 may be identified by an image analysis algorithm. For example, the image analysis algorithm can identify outlines of products in the image by identifying similar lines, textures, or other image qualities and attributing the similar pixels with stored images of room features or products (e.g., walls, floors, windows, couches, seats, curtains, etc.).” and “The user interface may be updated to display potential room layouts that the user can select and may also identify the products that have AR-enabled images corresponding with them.”; figs. 12, 17, para. [0156]-[0157]; Examiner note: the output is considered herein as the display potential room layouts; the similar 3D object, the 3D object similar to the input image among the 3D objects as the search targets is interpreted as the inspiration image 1710, see para. [0156]).
However, it is noted that the prior art of Besecker does not explicitly teach “by a search device, a 3D object generation step of generating an input 3D object from an input image; wherein for each viewpoint, the image generation unit is configured to generate a multi-modal image from the input 3D object, and output an image generated from the multi-modal image as an image in the search image group; and an image search step of searching, by using each of the images in the search image group as a search query, a database in which a plurality of images obtained by viewing a plurality of 3D objects as search targets from a plurality of viewpoints are registered;”
On the other hand, in the same field of endeavor, Nagano teaches by a search device (i.e. “the 3D representation of the single 2D image may be generated utilizing one or more local computing devices, one or more networked/distributed computing devices, one or more cloud-computing devices, etc.”; fig, 1, para. [0017], [0025]),
a 3D object generation step of (i.e. “a 2D convolutional neural network (CNN)”; fig. 1, para. [0018]; Examiner note: the 3D object generation unit is interpreted as the a 2D convolutional neural network (CNN))
generating an input 3D object from an input image (i.e. “a three-dimensional (3D) representation of the single 2D image is generated utilizing a 2D convolutional neural network (CNN).”; figs. 1, 6, para. [0018]-[0019], [0095]; Examiner note: the generate an input 3D object from an input image is interpreted as the three-dimensional (3D) representation of the single 2D image is generated);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Nagano that teaches generating a three-dimensional model from a two-dimensional image into Besecker that teaches data lineage graphs. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to create 3D representations using a feedforward method, as it can improve the performance of computing hardware that implements 3D representation generation (Nagano, para. [0026]).
However, it is noted that the combination of the prior arts of Besecker and Nagano do not explicitly teach “wherein for each viewpoint, the image generation unit is configured to generate a multi-modal image from the input 3D object, and output an image generated from the multi-modal image as an image in the search image group; an image search step of searching, by using each of the images in the search image group as a search query, a database in which a plurality of images obtained by viewing a plurality of 3D objects as search targets from a plurality of viewpoints are registered;”
On the other hand, in the same field of endeavor, Sreedhar teaches an image search step of (i.e. “the server 110 may search the catalog to identify images of products ….”; figs. 1-2, para. [0047], [0049]; Examiner note: the image search unit is interpreted as the server)
searching, by using each of the images in the search image group as a search query (i.e. “The server 110 may identify, based on the received information, a number of candidate product images, such as image 220 shown in FIG. 2, from a retailer's product catalog that includes various images of the first type of product (e.g., images of multiple different products of the first type, such as accent chair).”; fig. 2, para. [0047], [0049]; Examiner note: the server is identifying number of candidate product images based on the candidate product image 220 herein the candidate product image 220 is interpreted to be used as a search query; where the identity is searching for possible candidate product images),
a database in which a plurality of images obtained (i.e. “multiple proxy product models may be placed in a user's space and a product image obtained from the server may include more than one product in the image. In some embodiments, images of the user's space may include existing images of the user's space (e.g., home) or images obtained from third-party sources.”; para. [0083]; Examiner note: the database is interpreted as the server herein)
by viewing a plurality of 3D objects as search targets (i.e. “the catalog 240 of images of products that is utilized to identify candidate product images may be stored at the server 110”; figs. 2, 13, para. [0056]; Examiner note: the viewing a plurality of 3D objects as search targets is interpreted as the catalog of images of products that is utilized to identify candidate product images)
from a plurality of viewpoints are registered (i.e. “An overview of the processing is shown in FIGS. 10A-10D. In some embodiments, the processing of a product silo or lifestyle image may include running it through a machine learning model (e.g., a regression-based machine learning model) that detects the angle the product is facing in the image, also referred to herein as the shot angle of the product, and labels it.”; fig. 2, para. [0056]; Examiner note: the plurality of viewpoints is registered is interpreted as the shot angle of the product, see fig. 10A-D where multiples shot angle of the product is illustrated);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface into the combination of the prior art of Besecker that teaches data lineage graphs, and Nagano that teaches generating a three-dimensional model from a two-dimensional image. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to use augmented reality (AR) to improve the shopping experience for customers because it can provide a visualization of the product in the physical scene (Sreedhar, para. [0026]).
However, it is noted that the combination of the prior arts of Besecker, Nagano and Sreedhar do not explicitly teach “an image generation step of generating, as a search image group a plurality of images obtained by viewing the input 3D object from a plurality of viewpoints, wherein the image generation unit generates a multi-modal image from the input 3D object, and outputs an image generated from the multi-modal image as an image in the search image group.”
On the other hand, in the same field of endeavor, Tan teaches an image generation step (i.e. “FIG. 8 presents another example multimodal training data generation module 800 in accordance with one or more embodiments of the disclosed subject matter.”; fig. 8, para. [0125]; Examiner note: the image generation unit is interpreted as the multimodal training data generation module 800) of
generating, as a search image group (i.e. “At 1502, the annotation transfer component 114 can further transfer the projected ground truth data 1308 onto the corresponding native 2D image 1506 as shown in box 1504, resulting in the generation of the corresponding native 2D image 1508 with transferred ground truth data.”; fig. 15:1508, para. [0153]; Examiner note: the search image group is interpreted as the native 2D image 1508),
a plurality of images obtained (i.e. “generating the projected ground truth data 1308.”; fig. 15:1308, para. [0153]; Examiner note: the plurality of images obtained is interpreted as the ground truth data; the ground of truth data has a plurality of images)
by viewing the input 3D object (i.e. “Process 1300 involves transferring ground truth data 1302 for the native 3D image onto the corresponding synthetic 2D image 1304 to generate a corresponding synthetic 2D image 1316 with transferred ground truth data.”; figs. 13:1302, 15:1302, para. [0147]; Examiner note: the viewing the input 3D object is interpreted as the transferring ground truth data 1302 for the native 3D image)
from a plurality of viewpoints (i.e. “apply various augmented features and/or additional adjustments to the respective copies 1204 to generate variational training data images that are more representative of the different native 2D images that will be encountered in the field (e.g., to simulate different patient data).”; figs. 12:1204, 14:1404, 17, 20, para. [0141], [0149]; Examiner note: the plurality of viewpoints is interpreted as the copies 1204),
wherein for each viewpoint, the image generation unit is configured to generate a multi-modal image from the input 3D object (i.e. “This results in generation of annotated 2D image data 2024 in the target capture modality. For example, image 2026 corresponds to the annotated native 2D image and image 2028 corresponds to the annotated synthetic 2D image.”; fig. 20:2024, para. [0167]-[0169]; Examiner note: the multi-modal image is interpreted as the annotated 2D image data 2024. Further, “Process 2000 can be repeated for each patient for all the paired images from all the patients to obtain a high-quality, annotated images of the second modality and the resulting annotated synthetic and/or native 2D imaged data can be added to a training dataset 2303.”; para. [0170]), and
output an image generated from the multi-modal image as an image in the search image group (i.e. “the annotation component 106 can present the eSXR image with the transferred GT to one or more annotators for manual review and adjustment.”; figs. 17, 20, 22A-B, para. [0069], [0180]; Examiner note: the output the image generated is intpreted as the present the eSXR image. Further, i.e. “For example, in some embodiments, the GT data can include mark-up image data applied to the CT image data and transferred to the eSXR lung ROI image 2204 using the same projection parameters used to generate the eSXR image 2117.”; fig. 22B, para. [01786]; Examiner note: image in the search image group is interpreted as the eSXR lung ROI image 2204);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Tan that teaches a training image data generation and usage thereof for developing mono-modality image inferencing models into the combination of the prior art of Besecker that teaches data lineage graphs, Nagano that teaches generating a three-dimensional model from a two-dimensional image, and Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to improve a quality and realistic appearance of the synthetic 2D output image using one or more pre-projection processing steps because it can significantly reduce the simulation-to-reality (sim2real) gap by correcting distortions, compensating for radiometric differences, and removing artifacts, allowing the synthetic data to closely mimic real-world visual characteristics (Tan, para. [0050]-[0051]).
10. Claims 8-9 are rejected under 35 U.S.C. § 103 as being unpatentable over Besecker et al. (US 20230177832 A1) in view of Nagano et al. (US 20230081641 A1) in further view of Sreedhar et al. (US 20240029148 A1) still in further view of Tan et al. (US 20220101048 A1) still in further view of Monaghan et al. (US 11113875 B1).
As per claim 8, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 1 above.
Additionally, wherein the image search unit searches for a representative image among the images registered in the database (i.e. “The user may save one or more AR-enabled images of products using the “add to AR list” feature while searching for products on a desktop and then later transfer the list/images locally to mobile computing device 102 in the local resource database 106 (e.g. when the user is in the room for which they are shopping or at other times).”; para. [0151]-[0152]; Examiner note: where the AR list have a plurality of images).
However, it is noted that the combination of the prior arts of Besecker, Nagano, Sreedhar and Tan do not explicitly teach “the representative image is an image present in a region where density is less than a threshold in a feature space.”
On the other hand, in the same field of endeavor, Monaghan teaches the representative image is an image present in a region where density is less than a threshold in a feature space (i.e. “For instance, if more than 20% of the point cloud is determined to have a density that is below a threshold amount, 3D analysis tool 200 may produce overlay visualization 1006 that identifies 20% of the point cloud data points that do not satisfy the density threshold, and may notify the user that the point cloud has insufficient quality to validated due to the data point density.”; figs. 2, 10, Col. 14, Lines 28-39).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Monaghan that teaches a three-dimensional (“3D”) analysis tool into the combination of the prior art of Besecker that teaches data lineage graphs, Nagano that teaches generating a three-dimensional model from a two-dimensional image, Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface, and Tan that teaches a visual media-based multimodal chatbot. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to analyze one or more attributes across the entire 2D image in order to improve image acquisition, processing, editing, and/or delivery because it can enhance image quality for human viewing, provide better input for automated image processing techniques, or enable high-throughput extraction of quantitative data for analysis (Monaghan, Col. 1, Lines 1-17).
As per claim 9, Besecker, Nagano, Sreedhar and Tan teach all the limitations as discussed in claim 2 above.
However, it is noted that the combination of the prior arts of Besecker, Nagano, Sreedhar and Tan do not explicitly teach “wherein the image search result integration unit corrects, according to a degree of coincidence between a combination of the viewpoints in the search image group and a combination of viewpoints in the images associated with the identification information of the same 3D object, the similarity of the 3D object as the search target.”
On the other hand, in the same field of endeavor, Monaghan teaches wherein the image search result integration unit corrects (i.e. “The user may also determine that an accurate capture of a multi-faceted second object with different colors may have up to 5% outlying data points, and may set the threshold for visualization 1102 to be 5% for the representing the second object.”; fig. 11; Col. 15, Lines 36-46; Examine note: the wherein the image search result integration unit corrects is interpreted as the set the threshold for visualization 1102 to be 5% for the representing the second object),
according to a degree of coincidence between a combination of the viewpoints in the search image group and a combination of viewpoints in the images associated with the identification information of the same 3D object, the similarity of the 3D object as the search target (i.e. “visualization 1102 may display a value that quantifies the detected outlying data points, or may provide a first color when the percentage of outlying data points in the selected region of data points is less than a threshold (e.g., <5%), and a second color when the percentage of outlying data points in the selected region of data points is greater than the threshold (e.g., >=5%).”; fig. 11, Col. 15, Lines 23-35).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Monaghan that teaches a three-dimensional (“3D”) analysis tool into the combination of the prior art of Besecker that teaches data lineage graphs, Nagano that teaches generating a three-dimensional model from a two-dimensional image, Sreedhar that teaches an e-commerce business may display images of its products and/or computer-generated (e.g., 2D or 3D) product models on a webpage and/or any other software interface, and Tan that teaches a training image data generation and usage thereof for developing mono-modality image inferencing models. Additionally, this can provide a menu option to select other displays or features, which may include functionality that enables the user to add one or more products (e.g., 3D models) to a virtual room planner.
The motivation for doing so would be to analyze one or more attributes across the entire 2D image in order to improve image acquisition, processing, editing, and/or delivery because it can enhance image quality for human viewing, provide better input for automated image processing techniques, or enable high-throughput extraction of quantitative data for analysis (Monaghan, Col. 1, Lines 1-17).
Prior Art of Record
11. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Park et al. (US 20250225175 A1), teaches a device and a method for image search in a surveillance camera system.
Tsuji et al. (US 20200401616 A1), teaches an image search device receives an image search request from a user, selects from a plurality of images a first group of images that match search conditions based on the search request, selects from the first group of images a second group of images based on a score calculated by a predetermined evaluation formula, and outputs information about the second group of images as a search result.
Watanabe et al. (US 20200065324 A1), teaches image search method.
Sodhani et al. (US 20190171906 A1), teaches digital image search based on arbitrary image features, a server computing device maintains an images database of digital images, and includes an image search system that receives a search input as a digital image depicting image features, and receives search criteria of depicted image features in the digital image.
Bradley et al. (US 20230154101 A1), teaches computer science and image generation and, more specifically, to techniques for multi-view neural object modeling.
Conclusion
12. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTONIO CAIA DO whose telephone number is (469)295-9251. The examiner can normally be reached on Monday - Friday / 06:30 to 16:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ng, Amy can be reached on (571) 270-1698. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTONIO J CAIA DO/
Examiner, Art Unit 2164
/AMY NG/Supervisory Patent Examiner, Art Unit 2164
Prosecution Insights