DETAILED ACTION
Remarks
Claims 1 and 3-19 have been examined and rejected. This Office action is responsive to the amendment filed on 07/27/2026, which has been entered in the above identified application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1 and 3-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, claim 1 recites:
determining, based on the virtual camera parameter, one or more virtual object groups as one or more recommendation targets from among a plurality of virtual object groups
selecting one or more virtual object groups from among the plurality of virtual object groups
the one or more virtual object groups determined as the one or more recommendation targets
selecting one or more virtual object groups from among the plurality of virtual object groups
The relationship between these elements is unclear. For the purposes of examination, this limitation is interpreted as:
determining, based on the virtual camera parameter, one or more first virtual object groups as one or more recommendation targets from among a plurality of virtual object groups
selecting one or more second virtual object groups from among the plurality of virtual object groups
the one or more virtual first object groups
selecting one or more third virtual object groups from among the plurality of virtual object groups
Claim 1 further recites:
an input for selecting one or more virtual object groups
receiving, from the user via the user interface, a user input selecting one or more virtual object groups
the received input
the one or more virtual object groups selected by the user input
It is unclear how how “the received input” and “the user input” are intended to relate to the previously recited “an input” or the previously recited “a user input”. For the purposes of examination, this limitation is interpreted as:
a first input for selecting one or more virtual object groups
receiving, from the user via the user interface, a second user input selecting one or more virtual object groups
a third received input
the one or more virtual object groups selected by a fourth user input
Regarding claims 12 and 13, claims 12 and 13 contain substantially similar limitations to those found in claim 1. Consequently, claims 12 and 13 are rejected for the same reasons.
Regarding claims 2-11, claims 2-11 are also rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for depending on an indefinite parent claim. Claims 3, 6, 7, 8-11, 14, 16, and 18 recite additional limitations regarding “object groups”. Claims 3, 8, and 11 recite additional limitations regarding “input”. These limitations are likewise rejected and interpreted.
Regarding claim 6, claim 6 recites “the range corresponding to the virtual object group”. The claim does not previously recite a virtual object group or a range corresponding the virtual object group. It is unclear how this limitation is intended to relate the previously recited “range of the at least one of the position, the orientation, or the angle of view of the virtual camera”. For the purposes of examination, this limitation is interpreted as: a range corresponding to a virtual object group
Regarding claims 7 and 8, claims 7 and 8 recite “the information indicating the range corresponding to each of the plurality of virtual object groups”. It is unclear how this limitation is intended to relate the previously recited “information indicating a range of the at least one of the position, the orientation, or the angle of view of the virtual camera” in parent claim 6. For the purposes of examination, this limitation is interpreted as: second information indicating a second range corresponding to each of the plurality of virtual object groups
Regarding claims 7 and 9, claims 7 and 9 recite “the position of the virtual camera”. It is unclear how this limitation is intended to relate the previously recited “a position, an orientation, or an angle of view of a virtual camera in a virtual space”. For the purposes of examination, this limitation is interpreted as: a second position of the virtual camera
Regarding claim 9, claim 9 recites “virtual camera parameters corresponding parameters corresponding to different frames”. It is unclear what is meant by parameters corresponding parameters corresponding. It is unclear how the virtual camera parameters are intended to relate to different frames. For the purpose of examination, this limitation is interpreted as: virtual camera parameters and different frames
Regarding claim 9, claim 9 further recites “maintaining, without change, an indication of the one or more virtual object groups determined as the one or more recommendation targets in the user interface in a case where the moving speed of the virtual camera is equal to or greater than a threshold value”. It is unclear whether “in the user interface” is intended to modify the maintaining, the indication, the one or more virtual objects groups, or the recommendations targets. It is unclear whether “in a case where the moving speed of the virtual camera is equal to or greater than a threshold value” is intended to modify the maintaining, the indication, the one or more virtual objects groups, the recommendations targets, or the user interface. For the purpose of examination, this limitation is interpreted as: in a case where the moving speed of the virtual camera is equal to or greater than a threshold value, maintaining, in the user interface without change, an indication of the one or more virtual object groups determined as the one or more recommendation targets
Regarding claims 11, 16, and 19, claims 11, 16, and 19 recites “the one or more virtual objects belonging to the virtual object group” (claim 19 recites “the corresponding virtual object group”. The claims does not previously recite an object group, a corresponding virtual object group, or one or more virtual objects belonging to the virtual object group. It is unclear how this limitation is intended to relate to the previously recited one or more virtual object groups limitations in the parent claims. For the purpose of examination, this limitation is interpreted as: second one or more virtual objects belonging to a virtual object group
Regarding claim 14, claim 11 recites “a recommendation indicator, separate from the selectable control, associated with the virtual object group”. It is unclear whether the limitation “associated with the virtual object group” (which is separated from the other limitations by a comma) is associated with the recommendation indicator or the selectable control (referring back to the previously recited “a selectable control associated with the virtual object group”. It is further unclear how “the virtual object group” is intended to relate to the previously recited virtual object groups of claim 1. For the purpose of examination, these limitation is interpreted as: “a recommendation indicator, separate from the selectable control” and “a first virtual object group”.
Regarding claims 15 and 19, claims 15 and 19 recite “the corresponding virtual object group”. It is further unclear how “the corresponding virtual object group” is intended to relate to the previously recited virtual object groups of claim 1 and “the virtual object group” of claim 14. For the purpose of examination, these limitation is interpreted as: a second corresponding virtual object group
Regarding claim 16, claim 16 recites “the recommendation indicator being associated with the same virtual object group”. The claims do not previously recite a same virtual object group or a recommendation indicator associated with the same object group. For the purpose of examination, these limitation is interpreted as: a second recommendation indicator being associated with a same virtual object group
Regarding claim 17, claim 17 recites “the selectable control associated with a virtual object group that is not determined as one of the one or more recommendation targets”. It is unclear how this limitation is intended to relate to the previously recited “a selectable control associated with the virtual object group”. For the purpose of examination, this limitation is interpreted as: a second selectable control associated with a virtual object group that is not determined as one of the one or more recommendation targets
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-8, and 10-19 are rejected under 35 U.S.C. 103 as being unpatentable over Ito (US 20220182559 A1, published 06/09/2022) in view of Yalniz et al. (US 20150363943 A1, published 12/17/2015), hereinafter Yalniz.
Regarding claim 1, Ito teaches the claim comprising:
A control apparatus comprising: one or more hardware processors; and one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions for (Ito Figs. 1-16; abs. The information processing apparatus includes: one or more processors functioning by executing instructions stored in one or more memories as the following units: a setting unit configured to set a parameter with which a display condition of additional information that is displayed in a virtual viewpoint image, identification information indicating the additional information, and virtual viewpoint information on a virtual viewpoint image that is generated are associated; and an output unit configured to output a parameter that is set by the setting unit):
obtaining a virtual camera parameter including information of at least one of a position, an orientation, or an angle of view of a virtual camera in a virtual space (Ito Figs. 1-16; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0047], The virtual viewpoint information includes the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera; [0055], The generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data (three-dimensional data) is explained by using diagrams. FIG. 3A to FIG. 3C are each a diagram explaining the generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data corresponding to an image (video) of a game of soccer. FIG. 3A shows a virtual viewpoint image example generated in accordance with a virtual viewpoint based on basic 3D data. FIG. 3B shows an image example of virtual advertisement data in accordance with the virtual viewpoint of the virtual viewpoint image in FIG. 3A. FIG. 3C shows a virtual advertisement-added virtual viewpoint image example obtained by combining the virtual viewpoint image in FIG. 3A and the virtual advertisement data in FIG. 3B.);
determining one or more virtual object groups as one or more recommendation targets from among a plurality of virtual object groups, each of the plurality of virtual object groups including one or more virtual objects; generating a user interface that assists a user to make an input for selecting one or more virtual object groups from among the plurality of virtual object groups, the user interface indicating the one or more virtual object groups determined as the one or more recommendation targets; receiving, from the user via the user interface, a user input selecting one or more virtual object groups from among the plurality of virtual object groups; and based on the received input, determining the one or more virtual object groups selected by the user input (Ito Figs. 1-16; [0028], The operation reception unit 111 receives an input of a user operation (also referred to as user input). Information that is generated in accordance with the received input operation is sent to each function unit in accordance with the operation contents. The virtual advertisement setting unit 112 sets virtual advertisement data that is combined with a virtual viewpoint video (that is added onto virtual viewpoint image) in accordance with the information that is sent from the operation reception unit 111. The virtual advertisement data is, for example, image data of an advertisement signboard that does not exist in an addition-target virtual viewpoint image (basic 3D data). The virtual advertisement data that is set may be one or a plurality of pieces of virtual advertisement data may be set, and one kind or a plurality of kinds of virtual advertisement data may be set; [0064], At S413, the operation reception unit 111 receives a user operation for inputting a virtual viewpoint image generation parameter; [0065], At S414, the operation reception unit 111 determines whether or not there are instructions to display a virtual advertisement in the information that is generated in accordance with the input operation received by the operation reception unit 111 at S413; [0093], An example of a user interface (UI) for setting a condition of displaying a virtual advertisement in a virtual viewpoint image is explained by using diagrams; [0096], On the other hand, in a case where it is desired to display a virtual advertisement in a virtual viewpoint image, as shown in FIG. 8B, the radio button 803 corresponding to the virtual advertisement (“virtual advertisement A”) desired to be displayed is selected in order to set display of the virtual advertisement by the user operation at timing at which the virtual advertisement is desired to be displayed. To the selection of the radio button 803 such as this, the processing sequences at S504 and S505 in the sequence diagram in FIG. 5 correspond; [0097], Here, a user operation screen for setting a virtual viewpoint corresponding to a virtual viewpoint image and a virtual advertisement display condition is explained by using diagrams; FIG. 9A and FIG. 9B each show a case where the virtual viewpoint image corresponding to the set display condition is displayed on the display on the left side and the virtual advertisement display condition setting UI is displayed on the display on the right side; [0098], As shown in FIG. 9A, a display 901 on the left side displays a virtual viewpoint image 911 generated in accordance with the virtual viewpoint based on the basic 3D data. The virtual viewpoint is set initially and before the reception of the user operation relating to the virtual viewpoint, the virtual viewpoint that is set initially is used as the virtual viewpoint; [0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101], by registering the virtual advertisement data in advance and selecting the virtual advertisement at the timing at which the virtual advertisement is desired to be displayed on the UI displayed on the display 902 on the right side while watching the virtual viewpoint image that is displayed on the display 901 on the left side, it is possible to immediately display the virtual advertisement in the virtual viewpoint image; [0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0137-0138], a case is explained where the broadcast time of a game of soccer is from 18:00 to 20:00 and the setting is performed so that from 18:00 to 19:00, a virtual advertisement AAA is displayed in a virtual viewpoint image and from 19:00 to 20:00, a virtual advertisement BBB is displayed in the virtual viewpoint image; [0139], As shown in FIG. 15, by the user operation, in the start time setting field 1511, “18:00” is set, in the end time setting field 1512, “19:00” is set, and in the display target setting field 1513, “virtual advertisement AAA” is set. Further, in the start time setting field 1511, “19:00” is set, in the end time setting field 1512, “20:00” is set, and in the display target setting field 1513, “virtual advertisement BBB” is set)
However, Ito fails to expressly disclose determining, based on the virtual camera parameter, one or more virtual object groups as one or more recommendation targets from among a plurality of virtual object groups. In the same field of endeavor, Yalniz teaches:
determining, based on the virtual camera parameter, one or more virtual object groups as one or more recommendation targets from among a plurality of virtual object groups (Yalniz Figs. 1-24; [0041], Referring next to FIG. 3, the client device 100 is shown capturing the digital image 106; [0042], changes in a viewing direction; [0045], the image analysis engine 215 may determine characteristics for each of the empty regions 406 which may be used in determining a recommended item for a corresponding one of the empty regions 406. For example, a first empty region 406a and a second empty region 406b may be determined to be an empty space on a wall using visual image analysis. In this example, a recommended item, such as a wall ornament, a canvas painting, wallpaper, etc., may be positioned in the first empty region 406a and/or the second empty region 406b, as will be described in greater detail below; [0046], Similarly, a third empty region 406c, a fourth empty region 406d, and a fifth empty region 406e may be determined to be an empty space on a floor using the same visual image analysis. In this example, a recommended item, such as a piece of furniture, a rug, etc., may be positioned in the third empty region 406c, the fourth empty region 406d, and/or the fifth empty region 406e; [0047], a first recommended item 296a and a second recommended item 296b (collectively recommended items 296) are shown in the third empty region 406c (FIG. 4A) and the fourth empty region 406d (FIG. 4A), respectively. As the third empty region 406c and the fourth empty region 406d were located on the floor of the setting, the recommendation service 218 may recommend items suitable for placement on the floor, such as a chaise lounge chair (recommended item 296a) and a flatwoven rug (recommended item 296b).; [0048], the representative image 236 of the flatwoven rug (recommended item 296b), may be skewed, scaled and/or rotated such that it conforms with the perspective of the setting; [0049], additional information associated with the recommended items 296 may be shown to the user; [0051], a list of the recommended items 296 may be shown in association with the modified form of the digital image 290. The list of the recommended items 296 may comprise a name, a description, a price, an average review, the representative images 236a . . . 236b, and/or other information associated with the recommended item 296; [0067], given the user of the client device 100 has provided a plurality of digital images 106 of the setting (or a video of the setting), the image modification engine 221 may stitch the digital images 106 provided to generate a three-dimensional model of the setting that may be encoded for rendering in the user interface 272. Further, construction of the three-dimensional model may employ motion estimation, homography, various camera models, etc)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated determining, based on the virtual camera parameter, one or more virtual object groups as one or more recommendation targets from among a plurality of virtual object groups as suggested in Yalniz into Ito. Doing so would be desirable because computer vision relates to analyzing, interpreting, understanding, and deriving information from digital images obtained via digital cameras or similar image capturing devices. Such information may be used in automated color detection, depth perception, and object recognition in computing environments (see Yalniz [0002]). The present disclosure relates to generating recommendations utilizing visual image analysis. It remains difficult for the user to perceive how an item may look in a particular setting. Accordingly, it may be beneficial for a user to provide a digital image of a setting in which the item may be visualized (see Yalniz [0013]). According to various embodiments, the electronic commerce system may communicate with various image processing services, such as an image analysis engine and/or an image modification engine, to analyze a digital image of a setting (e.g., a room) provided by a user. The analysis of the digital image may identify various characteristics about the setting, such as colors, sizes, dimensions, or locations of various objects in the setting. Further, the image processing services may identify regions in the setting in which no object is recognized (hereinafter referred to as empty regions). The electronic commerce system may leverage a recommendation service to identify an item to recommend to the user by utilizing the various characteristics about the room identified in the digital image and/or historical data associated with the user in the electronic commerce system (see Yalniz [0004]). Additionally, the system of Yalniz would improve the system of Ito by providing suggestions that are determined based on the position, orientation, and angle of view from which a virtual camera image is determined, thereby ensuring recommended objects are contextually appropriate, which would save the user time and increase ease of use.
Regarding claims 12 and 13, claims 12 and 13 contain substantially similar limitations to those found in claim 1. Consequently, claims 12 and 13 are rejected for the same reasons.
Regarding claim 3, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein determining the one or more virtual object groups selected by the user input comprises determining identification information uniquely identifying each of the one or more the virtual object groups selected by the user input (Ito Figs. 1-16; [0031], identification information uniquely indicating the display condition of the selected virtual advertisement data and the selected virtual advertisement to the parameter; [0040], identification information on the virtual advertisement data and the information relating to the display condition of the virtual advertisement data are sent to the virtual advertisement reading unit 128; [0064], At S413, the operation reception unit 111 receives a user operation for inputting a virtual viewpoint image generation parameter; [0065], At S414, the operation reception unit 111 determines whether or not there are instructions to display a virtual advertisement in the information that is generated in accordance with the input operation received by the operation reception unit 111 at S413; [0098-0099], The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101], by registering the virtual advertisement data in advance and selecting the virtual advertisement at the timing at which the virtual advertisement is desired to be displayed on the UI displayed on the display 902 on the right side while watching the virtual viewpoint image that is displayed on the display 901 on the left side, it is possible to immediately display the virtual advertisement in the virtual viewpoint image; [0104], by registering in advance virtual advertisement data (3D model and texture of stationary image) and setting the identification information on the virtual advertisement data, the virtual advertisement display condition, the virtual viewpoint information, and the time code to the parameter, the following effects are obtained. It is possible to generate a virtual advertisement-added virtual viewpoint image or a virtual viewpoint image with no virtual advertisement in accordance with the information extracted from the parameter; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0121], The identification number of the virtual advertisement data set to the schedule by the display condition setting unit 115 is set to the parameter to which the information relating to virtual advertisement display instructions are set along with the virtual viewpoint information and the time code along the schedule; [0137-0138], a case is explained where the broadcast time of a game of soccer is from 18:00 to 20:00 and the setting is performed so that from 18:00 to 19:00, a virtual advertisement AAA is displayed in a virtual viewpoint image and from 19:00 to 20:00, a virtual advertisement BBB is displayed in the virtual viewpoint image; see also [0028], [0064-0065], [0093], [0096-0097], [0102], [0112], [0139])
Regarding claim 4, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein obtaining the virtual camera parameter comprises calculating the virtual camera parameter based on operation information received in response to an operation by the user (Ito Figs. 1-16; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0045], The combination unit 131 combines the basic 3D data read by the basic data reading unit 130 and the virtual advertisement data read by the virtual advertisement reading unit 128 and generates one piece of combined 3D data; [0046], The virtual viewpoint extraction unit 127 extracts, from the parameter received by the parameter reception unit 123, the virtual viewpoint information (virtual camera position information) indicating the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, which are included in the parameter; [0047], The virtual viewpoint information includes the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera; [0055], The generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data (three-dimensional data) is explained by using diagrams. FIG. 3A to FIG. 3C are each a diagram explaining the generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data corresponding to an image (video) of a game of soccer. FIG. 3A shows a virtual viewpoint image example generated in accordance with a virtual viewpoint based on basic 3D data. FIG. 3B shows an image example of virtual advertisement data in accordance with the virtual viewpoint of the virtual viewpoint image in FIG. 3A. FIG. 3C shows a virtual advertisement-added virtual viewpoint image example obtained by combining the virtual viewpoint image in FIG. 3A and the virtual advertisement data in FIG. 3B; [0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side; see also [0028], [0064-0065], [0093], [0096-0097], [0102], [0112], [0139])
Regarding claim 5, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein the virtual camera defines a virtual viewpoint used to generate a virtual viewpoint image based on a plurality of pieces of image data obtained by a plurality of image capturing apparatuses (Ito Figs. 1-16; [0002], a technique has been attracting attention, which performs synchronous image capturing from a plurality of viewpoints by installing a plurality of cameras at different positions and generates a virtual viewpoint image (also called free-viewpoint video) representing an appearance from an arbitrary virtual viewpoint by using a multi-viewpoint image obtained by the image capturing; [0004], in a case where the viewpoint condition corresponding to the free-viewpoint video is different from the specific viewpoint condition that is set in advance, it is not possible to add advertisement information; [0005], An object of the present disclosure is to provide a technique to generate a virtual viewpoint image in which desired information is added to a desired position; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0055], The generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data (three-dimensional data) is explained by using diagrams. FIG. 3A to FIG. 3C are each a diagram explaining the generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data corresponding to an image (video) of a game of soccer. FIG. 3A shows a virtual viewpoint image example generated in accordance with a virtual viewpoint based on basic 3D data. FIG. 3B shows an image example of virtual advertisement data in accordance with the virtual viewpoint of the virtual viewpoint image in FIG. 3A. FIG. 3C shows a virtual advertisement-added virtual viewpoint image example obtained by combining the virtual viewpoint image in FIG. 3A and the virtual advertisement data in FIG. 3B; [0056], A virtual viewpoint image 310 shown in FIG. 3A is an image in a case where an object that exists actually is viewed from an arbitrary viewpoint (virtual viewpoint) and includes a field 311, players 312 existing on the field, and advertisement signboards 313 installed on the side of the field 311. The virtual viewpoint image 310 is an image that is generated in accordance with a virtual viewpoint based on basic 3D data stored in advance in the basic data storage unit 129. The basic 3D data is, for example, shape data indicating the shape of an object, which is generated by a publicly known technique based on a plurality of captured images obtained by capturing a target area from different viewpoints with a plurality of imaging apparatuses; [0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side; see also [0028], [0064-0065], [0093], [0096-0097], [0102], [0112], [0139])
Regarding claim 6, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein the one or more programs further include instructions for obtaining, for each of the plurality of virtual object groups, information indicating a range of the at least one of the position, the orientation, or the angle of view of the virtual camera, and determining the one or more virtual object groups as the one or more recommendation targets comprises (Ito Figs. 1-16; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0045], The combination unit 131 combines the basic 3D data read by the basic data reading unit 130 and the virtual advertisement data read by the virtual advertisement reading unit 128 and generates one piece of combined 3D data; [0046], The virtual viewpoint extraction unit 127 extracts, from the parameter received by the parameter reception unit 123, the virtual viewpoint information (virtual camera position information) indicating the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, which are included in the parameter; [0047], The virtual viewpoint information includes the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera; [0055], The generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data (three-dimensional data) is explained by using diagrams. FIG. 3A to FIG. 3C are each a diagram explaining the generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data corresponding to an image (video) of a game of soccer. FIG. 3A shows a virtual viewpoint image example generated in accordance with a virtual viewpoint based on basic 3D data. FIG. 3B shows an image example of virtual advertisement data in accordance with the virtual viewpoint of the virtual viewpoint image in FIG. 3A. FIG. 3C shows a virtual advertisement-added virtual viewpoint image example obtained by combining the virtual viewpoint image in FIG. 3A and the virtual advertisement data in FIG. 3B; [0084], the obj file is explained by using diagrams. FIG. 7 is a diagram showing an example of the obj file that is designated in the 3D model format of virtual advertisement data. In the obj file, the vertex coordinate values are designated by a keyword “v”, the texture coordinate values are designated by a keyword “vt”, and a normal vector is designated by a keyword “vn”, respectively. Further, in the present embodiment, although the obj file is explained as an example of the file format representing information on the position, size, and orientation of the 3D model of virtual advertisement data, the file format is not limited to this; [0088], FIG. 6C is a diagram showing a format example of the virtual viewpoint image generation parameter (with virtual advertisement “A”) data (in the following, also referred to as parameter (with virtual advertisement “A” data)). In the format of the parameter (with virtual advertisement “A”) data, the information representing the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, and the time code are designated, which are also designated in the parameter (without virtual advertisement) shown in FIG. 6B. In the format of the parameter (with virtual advertisement “A”) data, information relating to virtual advertisement (in FIG. 6C, virtual advertisement “A”) display instructions is further designated. This parameter (with virtual advertisement “A”) includes instructions to display the virtual advertisement “A” ([“advertise ID”: “A” ] in FIG. 6A) that is instructed to be registered in the processing at S501 in the virtual viewpoint image; [0098-0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101-0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0147], FIG. 16B shows a format example of parameter (with virtual advertisement) data; see also [0028], [0064-0065], [0093], [0096-0097], [0112], [0137-0139])
Yalniz further teaches:
determining, for each of the plurality of virtual object groups, whether a value indicated by the virtual camera parameter falls within the range corresponding to the virtual object group (Yalniz Figs. 1-24; [0041], Referring next to FIG. 3, the client device 100 is shown capturing the digital image 106; [0042], changes in a viewing direction; [0045], the image analysis engine 215 may determine characteristics for each of the empty regions 406 which may be used in determining a recommended item for a corresponding one of the empty regions 406. For example, a first empty region 406a and a second empty region 406b may be determined to be an empty space on a wall using visual image analysis. In this example, a recommended item, such as a wall ornament, a canvas painting, wallpaper, etc., may be positioned in the first empty region 406a and/or the second empty region 406b, as will be described in greater detail below; [0046], Similarly, a third empty region 406c, a fourth empty region 406d, and a fifth empty region 406e may be determined to be an empty space on a floor using the same visual image analysis. In this example, a recommended item, such as a piece of furniture, a rug, etc., may be positioned in the third empty region 406c, the fourth empty region 406d, and/or the fifth empty region 406e; [0047], a first recommended item 296a and a second recommended item 296b (collectively recommended items 296) are shown in the third empty region 406c (FIG. 4A) and the fourth empty region 406d (FIG. 4A), respectively. As the third empty region 406c and the fourth empty region 406d were located on the floor of the setting, the recommendation service 218 may recommend items suitable for placement on the floor, such as a chaise lounge chair (recommended item 296a) and a flatwoven rug (recommended item 296b).; [0048], the representative image 236 of the flatwoven rug (recommended item 296b), may be skewed, scaled and/or rotated such that it conforms with the perspective of the setting; [0049], additional information associated with the recommended items 296 may be shown to the user; [0051], a list of the recommended items 296 may be shown in association with the modified form of the digital image 290. The list of the recommended items 296 may comprise a name, a description, a price, an average review, the representative images 236a . . . 236b, and/or other information associated with the recommended item 296)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated determining, for each of the plurality of virtual object groups, whether a value indicated by the virtual camera parameter falls within the range corresponding to the virtual object group as suggested in Yalniz into Ito. Doing so would be desirable because computer vision relates to analyzing, interpreting, understanding, and deriving information from digital images obtained via digital cameras or similar image capturing devices. Such information may be used in automated color detection, depth perception, and object recognition in computing environments (see Yalniz [0002]). The present disclosure relates to generating recommendations utilizing visual image analysis. It remains difficult for the user to perceive how an item may look in a particular setting. Accordingly, it may be beneficial for a user to provide a digital image of a setting in which the item may be visualized (see Yalniz [0013]). According to various embodiments, the electronic commerce system may communicate with various image processing services, such as an image analysis engine and/or an image modification engine, to analyze a digital image of a setting (e.g., a room) provided by a user. The analysis of the digital image may identify various characteristics about the setting, such as colors, sizes, dimensions, or locations of various objects in the setting. Further, the image processing services may identify regions in the setting in which no object is recognized (hereinafter referred to as empty regions). The electronic commerce system may leverage a recommendation service to identify an item to recommend to the user by utilizing the various characteristics about the room identified in the digital image and/or historical data associated with the user in the electronic commerce system (see Yalniz [0004]). Additionally, the system of Yalniz would improve the system of Ito by providing suggestions that are determined based on the position, orientation, and angle of view from which a virtual camera image is determined, thereby ensuring recommended objects are contextually appropriate, which would save the user time and increase ease of use.
Regarding claim 7, Ito in view of Yalniz teaches all the limitations of claim 6, comprising:
wherein the virtual camera parameter includes information indicating the position of the virtual camera, and the information indicating the range corresponding to each of the plurality of virtual object groups includes information indicating a range of the position of the virtual camera (Ito Figs. 1-16; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0045], The combination unit 131 combines the basic 3D data read by the basic data reading unit 130 and the virtual advertisement data read by the virtual advertisement reading unit 128 and generates one piece of combined 3D data; [0046], The virtual viewpoint extraction unit 127 extracts, from the parameter received by the parameter reception unit 123, the virtual viewpoint information (virtual camera position information) indicating the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, which are included in the parameter; [0047], The virtual viewpoint information includes the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera; [0055], The generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data (three-dimensional data) is explained by using diagrams. FIG. 3A to FIG. 3C are each a diagram explaining the generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data corresponding to an image (video) of a game of soccer. FIG. 3A shows a virtual viewpoint image example generated in accordance with a virtual viewpoint based on basic 3D data. FIG. 3B shows an image example of virtual advertisement data in accordance with the virtual viewpoint of the virtual viewpoint image in FIG. 3A. FIG. 3C shows a virtual advertisement-added virtual viewpoint image example obtained by combining the virtual viewpoint image in FIG. 3A and the virtual advertisement data in FIG. 3B; [0084], the obj file is explained by using diagrams. FIG. 7 is a diagram showing an example of the obj file that is designated in the 3D model format of virtual advertisement data. In the obj file, the vertex coordinate values are designated by a keyword “v”, the texture coordinate values are designated by a keyword “vt”, and a normal vector is designated by a keyword “vn”, respectively. Further, in the present embodiment, although the obj file is explained as an example of the file format representing information on the position, size, and orientation of the 3D model of virtual advertisement data, the file format is not limited to this; [0088], FIG. 6C is a diagram showing a format example of the virtual viewpoint image generation parameter (with virtual advertisement “A”) data (in the following, also referred to as parameter (with virtual advertisement “A” data)). In the format of the parameter (with virtual advertisement “A”) data, the information representing the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, and the time code are designated, which are also designated in the parameter (without virtual advertisement) shown in FIG. 6B. In the format of the parameter (with virtual advertisement “A”) data, information relating to virtual advertisement (in FIG. 6C, virtual advertisement “A”) display instructions is further designated. This parameter (with virtual advertisement “A”) includes instructions to display the virtual advertisement “A” ([“advertise ID”: “A” ] in FIG. 6A) that is instructed to be registered in the processing at S501 in the virtual viewpoint image; [0098-0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101-0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0147], FIG. 16B shows a format example of parameter (with virtual advertisement) data; see also [0028], [0064-0065], [0093], [0096-0097], [0112], [0137-0139])
Regarding claim 8, Ito in view of Yalniz teaches all the limitations of claim 6, comprising:
wherein the information indicating the range corresponding to each of the plurality of virtual object groups is set based on an input received from the user (Ito Figs. 1-16; [0028], The operation reception unit 111 receives an input of a user operation (also referred to as user input). Information that is generated in accordance with the received input operation is sent to each function unit in accordance with the operation contents. The virtual advertisement setting unit 112 sets virtual advertisement data that is combined with a virtual viewpoint video (that is added onto virtual viewpoint image) in accordance with the information that is sent from the operation reception unit 111; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0045], The combination unit 131 combines the basic 3D data read by the basic data reading unit 130 and the virtual advertisement data read by the virtual advertisement reading unit 128 and generates one piece of combined 3D data; [0046], The virtual viewpoint extraction unit 127 extracts, from the parameter received by the parameter reception unit 123, the virtual viewpoint information (virtual camera position information) indicating the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, which are included in the parameter; [0047], The virtual viewpoint information includes the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera; [0084], the obj file is explained by using diagrams. FIG. 7 is a diagram showing an example of the obj file that is designated in the 3D model format of virtual advertisement data. In the obj file, the vertex coordinate values are designated by a keyword “v”, the texture coordinate values are designated by a keyword “vt”, and a normal vector is designated by a keyword “vn”, respectively. Further, in the present embodiment, although the obj file is explained as an example of the file format representing information on the position, size, and orientation of the 3D model of virtual advertisement data, the file format is not limited to this; [0088], FIG. 6C is a diagram showing a format example of the virtual viewpoint image generation parameter (with virtual advertisement “A”) data (in the following, also referred to as parameter (with virtual advertisement “A” data)). In the format of the parameter (with virtual advertisement “A”) data, the information representing the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, and the time code are designated, which are also designated in the parameter (without virtual advertisement) shown in FIG. 6B. In the format of the parameter (with virtual advertisement “A”) data, information relating to virtual advertisement (in FIG. 6C, virtual advertisement “A”) display instructions is further designated. This parameter (with virtual advertisement “A”) includes instructions to display the virtual advertisement “A” ([“advertise ID”: “A” ] in FIG. 6A) that is instructed to be registered in the processing at S501 in the virtual viewpoint image; [0098-0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101-0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0147], FIG. 16B shows a format example of parameter (with virtual advertisement) data; see also [0028], [0064-0065], [0093], [0096-0097], [0112], [0137-0139])
Regarding claim 10, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein each of the one or more virtual objects belonging to each of the plurality of virtual object groups a virtual advertisement placeable in the virtual space (Ito Figs. 1-16; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0055], The generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data (three-dimensional data) is explained by using diagrams. FIG. 3A to FIG. 3C are each a diagram explaining the generation of a virtual advertisement-added virtual viewpoint image in accordance with a virtual viewpoint based on 3D data corresponding to an image (video) of a game of soccer. FIG. 3A shows a virtual viewpoint image example generated in accordance with a virtual viewpoint based on basic 3D data. FIG. 3B shows an image example of virtual advertisement data in accordance with the virtual viewpoint of the virtual viewpoint image in FIG. 3A. FIG. 3C shows a virtual advertisement-added virtual viewpoint image example obtained by combining the virtual viewpoint image in FIG. 3A and the virtual advertisement data in FIG. 3B; [0093], An example of a user interface (UI) for setting a condition of displaying a virtual advertisement in a virtual viewpoint image is explained by using diagrams; [0096], On the other hand, in a case where it is desired to display a virtual advertisement in a virtual viewpoint image, as shown in FIG. 8B, the radio button 803 corresponding to the virtual advertisement (“virtual advertisement A”) desired to be displayed is selected in order to set display of the virtual advertisement by the user operation at timing at which the virtual advertisement is desired to be displayed. To the selection of the radio button 803 such as this, the processing sequences at S504 and S505 in the sequence diagram in FIG. 5 correspond; [0097], Here, a user operation screen for setting a virtual viewpoint corresponding to a virtual viewpoint image and a virtual advertisement display condition is explained by using diagrams; FIG. 9A and FIG. 9B each show a case where the virtual viewpoint image corresponding to the set display condition is displayed on the display on the left side and the virtual advertisement display condition setting UI is displayed on the display on the right side; [0098], As shown in FIG. 9A, a display 901 on the left side displays a virtual viewpoint image 911 generated in accordance with the virtual viewpoint based on the basic 3D data. The virtual viewpoint is set initially and before the reception of the user operation relating to the virtual viewpoint, the virtual viewpoint that is set initially is used as the virtual viewpoint; [0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101], by registering the virtual advertisement data in advance and selecting the virtual advertisement at the timing at which the virtual advertisement is desired to be displayed on the UI displayed on the display 902 on the right side while watching the virtual viewpoint image that is displayed on the display 901 on the left side, it is possible to immediately display the virtual advertisement in the virtual viewpoint image; see also [0028], [0093], [0064-0065], [0096-0097], [0102], [0112], [0139])
Regarding claim 11, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein the one or more programs further include instructions for: obtaining, for each of the plurality of virtual object groups, group information including information about the one or more virtual objects belonging to the virtual object group; and generating a virtual viewpoint image based on the group information obtained for the plurality of virtual object groups, identification information uniquely identifying the one or more virtual object groups selected by the user input, and the virtual camera parameter (Ito Figs. 1-16; [0031], identification information uniquely indicating the display condition of the selected virtual advertisement data and the selected virtual advertisement to the parameter; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0040], identification information on the virtual advertisement data and the information relating to the display condition of the virtual advertisement data are sent to the virtual advertisement reading unit 128; [0045-0046], The virtual viewpoint extraction unit 127 extracts, from the parameter received by the parameter reception unit 123, the virtual viewpoint information (virtual camera position information) indicating the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, which are included in the parameter; [0047], The virtual viewpoint information includes the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera; [0064], At S413, the operation reception unit 111 receives a user operation for inputting a virtual viewpoint image generation parameter; [0065], At S414, the operation reception unit 111 determines whether or not there are instructions to display a virtual advertisement in the information that is generated in accordance with the input operation received by the operation reception unit 111 at S413; [0098-0099], The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101], by registering the virtual advertisement data in advance and selecting the virtual advertisement at the timing at which the virtual advertisement is desired to be displayed on the UI displayed on the display 902 on the right side while watching the virtual viewpoint image that is displayed on the display 901 on the left side, it is possible to immediately display the virtual advertisement in the virtual viewpoint image; [0104], by registering in advance virtual advertisement data (3D model and texture of stationary image) and setting the identification information on the virtual advertisement data, the virtual advertisement display condition, the virtual viewpoint information, and the time code to the parameter, the following effects are obtained. It is possible to generate a virtual advertisement-added virtual viewpoint image or a virtual viewpoint image with no virtual advertisement in accordance with the information extracted from the parameter; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0121], The identification number of the virtual advertisement data set to the schedule by the display condition setting unit 115 is set to the parameter to which the information relating to virtual advertisement display instructions are set along with the virtual viewpoint information and the time code along the schedule; [0137-0138], a case is explained where the broadcast time of a game of soccer is from 18:00 to 20:00 and the setting is performed so that from 18:00 to 19:00, a virtual advertisement AAA is displayed in a virtual viewpoint image and from 19:00 to 20:00, a virtual advertisement BBB is displayed in the virtual viewpoint image; see also [0028], [0064-0065], [0093], [0096-0097], [0102], [0112], [0139])
Regarding claim 14, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein the user interface includes, for each of the plurality of virtual object groups: a selectable control associated with the virtual object group; and a recommendation indicator (Ito Figs. 1-16; [0084], the obj file is explained by using diagrams. FIG. 7 is a diagram showing an example of the obj file that is designated in the 3D model format of virtual advertisement data. In the obj file, the vertex coordinate values are designated by a keyword “v”, the texture coordinate values are designated by a keyword “vt”, and a normal vector is designated by a keyword “vn”, respectively. Further, in the present embodiment, although the obj file is explained as an example of the file format representing information on the position, size, and orientation of the 3D model of virtual advertisement data, the file format is not limited to this; [0088], FIG. 6C is a diagram showing a format example of the virtual viewpoint image generation parameter (with virtual advertisement “A”) data (in the following, also referred to as parameter (with virtual advertisement “A” data)). In the format of the parameter (with virtual advertisement “A”) data, the information representing the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, and the time code are designated, which are also designated in the parameter (without virtual advertisement) shown in FIG. 6B. In the format of the parameter (with virtual advertisement “A”) data, information relating to virtual advertisement (in FIG. 6C, virtual advertisement “A”) display instructions is further designated. This parameter (with virtual advertisement “A”) includes instructions to display the virtual advertisement “A” ([“advertise ID”: “A” ] in FIG. 6A) that is instructed to be registered in the processing at S501 in the virtual viewpoint image; [0098-0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101-0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0147], FIG. 16B shows a format example of parameter (with virtual advertisement) data; see also [0028], [0033], [0045-0047], [0055], [0064-0065], [0093], [0096-0097], [0112], [0137-0139])
Yalniz further teaches:
a recommendation indicator, separate from the selectable control, associated with the virtual object group and having a display state indicating whether the virtual object group has been determined as one of the one or more recommendation targets (Yalniz Figs. 1-24; [0041], Referring next to FIG. 3, the client device 100 is shown capturing the digital image 106; [0042], changes in a viewing direction; [0045], the image analysis engine 215 may determine characteristics for each of the empty regions 406 which may be used in determining a recommended item for a corresponding one of the empty regions 406. For example, a first empty region 406a and a second empty region 406b may be determined to be an empty space on a wall using visual image analysis. In this example, a recommended item, such as a wall ornament, a canvas painting, wallpaper, etc., may be positioned in the first empty region 406a and/or the second empty region 406b, as will be described in greater detail below; [0046], Similarly, a third empty region 406c, a fourth empty region 406d, and a fifth empty region 406e may be determined to be an empty space on a floor using the same visual image analysis. In this example, a recommended item, such as a piece of furniture, a rug, etc., may be positioned in the third empty region 406c, the fourth empty region 406d, and/or the fifth empty region 406e; [0047], a first recommended item 296a and a second recommended item 296b (collectively recommended items 296) are shown in the third empty region 406c (FIG. 4A) and the fourth empty region 406d (FIG. 4A), respectively. As the third empty region 406c and the fourth empty region 406d were located on the floor of the setting, the recommendation service 218 may recommend items suitable for placement on the floor, such as a chaise lounge chair (recommended item 296a) and a flatwoven rug (recommended item 296b); [0048], the representative image 236 of the flatwoven rug (recommended item 296b), may be skewed, scaled and/or rotated such that it conforms with the perspective of the setting; [0049], additional information associated with the recommended items 296 may be shown to the user. As a non-limiting example, in FIG. 4B, the user may manipulate or otherwise engage the recommended item 296a to initiate a rendering of a dialog 412 that shows a name, a description, and an average review of the recommended item 296a; [0051], a list of the recommended items 296 may be shown in association with the modified form of the digital image 290. The list of the recommended items 296 may comprise a name, a description, a price, an average review, the representative images 236a . . . 236b, and/or other information associated with the recommended item 296; a first add to cart component 503a and a second add to cart component 503b (collectively add to cart components 503) may facilitate placement of the first recommended item 296a and the second recommended item 296b, respectively, in the virtual shopping cart upon a manipulation by the user)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated a recommendation indicator, separate from the selectable control, associated with the virtual object group and having a display state indicating whether the virtual object group has been determined as one of the one or more recommendation targets as suggested in Yalniz into Ito. Doing so would be desirable because computer vision relates to analyzing, interpreting, understanding, and deriving information from digital images obtained via digital cameras or similar image capturing devices. Such information may be used in automated color detection, depth perception, and object recognition in computing environments (see Yalniz [0002]). The present disclosure relates to generating recommendations utilizing visual image analysis. It remains difficult for the user to perceive how an item may look in a particular setting. Accordingly, it may be beneficial for a user to provide a digital image of a setting in which the item may be visualized (see Yalniz [0013]). According to various embodiments, the electronic commerce system may communicate with various image processing services, such as an image analysis engine and/or an image modification engine, to analyze a digital image of a setting (e.g., a room) provided by a user. The analysis of the digital image may identify various characteristics about the setting, such as colors, sizes, dimensions, or locations of various objects in the setting. Further, the image processing services may identify regions in the setting in which no object is recognized (hereinafter referred to as empty regions). The electronic commerce system may leverage a recommendation service to identify an item to recommend to the user by utilizing the various characteristics about the room identified in the digital image and/or historical data associated with the user in the electronic commerce system (see Yalniz [0004]). Additionally, the system of Yalniz would improve the system of Ito by providing suggestions that are determined based on the position, orientation, and angle of view from which a virtual camera image is determined, thereby ensuring recommended objects are contextually appropriate, which would save the user time and increase ease of use.
Regarding claim 15, Ito in view of Yalniz teaches all the limitations of claim 14. Yalniz further teaches:
wherein: the recommendation indicator is displayed in a first visual state when the corresponding virtual object group is determined as one of the one or more recommendation targets; and the recommendation indicator is displayed in a second visual state, visually distinguishable from the first visual state, when the corresponding virtual object group is not determined as one of the one or more recommendation targets (Yalniz Figs. 1-24; [0041], Referring next to FIG. 3, the client device 100 is shown capturing the digital image 106; [0042], changes in a viewing direction; [0045], the image analysis engine 215 may determine characteristics for each of the empty regions 406 which may be used in determining a recommended item for a corresponding one of the empty regions 406. For example, a first empty region 406a and a second empty region 406b may be determined to be an empty space on a wall using visual image analysis. In this example, a recommended item, such as a wall ornament, a canvas painting, wallpaper, etc., may be positioned in the first empty region 406a and/or the second empty region 406b, as will be described in greater detail below; [0046], Similarly, a third empty region 406c, a fourth empty region 406d, and a fifth empty region 406e may be determined to be an empty space on a floor using the same visual image analysis. In this example, a recommended item, such as a piece of furniture, a rug, etc., may be positioned in the third empty region 406c, the fourth empty region 406d, and/or the fifth empty region 406e; [0047], a first recommended item 296a and a second recommended item 296b (collectively recommended items 296) are shown in the third empty region 406c (FIG. 4A) and the fourth empty region 406d (FIG. 4A), respectively. As the third empty region 406c and the fourth empty region 406d were located on the floor of the setting, the recommendation service 218 may recommend items suitable for placement on the floor, such as a chaise lounge chair (recommended item 296a) and a flatwoven rug (recommended item 296b); [0048], the representative image 236 of the flatwoven rug (recommended item 296b), may be skewed, scaled and/or rotated such that it conforms with the perspective of the setting; [0049], additional information associated with the recommended items 296 may be shown to the user. As a non-limiting example, in FIG. 4B, the user may manipulate or otherwise engage the recommended item 296a to initiate a rendering of a dialog 412 that shows a name, a description, and an average review of the recommended item 296a; [0051], a list of the recommended items 296 may be shown in association with the modified form of the digital image 290. The list of the recommended items 296 may comprise a name, a description, a price, an average review, the representative images 236a . . . 236b, and/or other information associated with the recommended item 296; a first add to cart component 503a and a second add to cart component 503b (collectively add to cart components 503) may facilitate placement of the first recommended item 296a and the second recommended item 296b, respectively, in the virtual shopping cart upon a manipulation by the user; [0062], In 618, it is determined whether at least one empty region 406 was identified by the image analysis engine 212 in the digital image 106. If the at least one empty region 406 is not identified in the digital image 106, the process proceeds to 633 to send the list of the recommended items 296 to the client device 100, as will be discussed in greater detail below. As may be appreciated, the recommended items may be provided to the user based on the digital image 106 regardless of whether a representative image of the recommended item may be inserted in the digital image 106; [0068], In 633, a list of the recommended items 296 may be sent to the client device 100 for rendering)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein: the recommendation indicator is displayed in a first visual state when the corresponding virtual object group is determined as one of the one or more recommendation targets; and the recommendation indicator is displayed in a second visual state, visually distinguishable from the first visual state, when the corresponding virtual object group is not determined as one of the one or more recommendation targets as suggested in Yalniz into Ito. Doing so would be desirable because computer vision relates to analyzing, interpreting, understanding, and deriving information from digital images obtained via digital cameras or similar image capturing devices. Such information may be used in automated color detection, depth perception, and object recognition in computing environments (see Yalniz [0002]). The present disclosure relates to generating recommendations utilizing visual image analysis. It remains difficult for the user to perceive how an item may look in a particular setting. Accordingly, it may be beneficial for a user to provide a digital image of a setting in which the item may be visualized (see Yalniz [0013]). According to various embodiments, the electronic commerce system may communicate with various image processing services, such as an image analysis engine and/or an image modification engine, to analyze a digital image of a setting (e.g., a room) provided by a user. The analysis of the digital image may identify various characteristics about the setting, such as colors, sizes, dimensions, or locations of various objects in the setting. Further, the image processing services may identify regions in the setting in which no object is recognized (hereinafter referred to as empty regions). The electronic commerce system may leverage a recommendation service to identify an item to recommend to the user by utilizing the various characteristics about the room identified in the digital image and/or historical data associated with the user in the electronic commerce system (see Yalniz [0004]). Additionally, the system of Yalniz would improve the system of Ito by providing suggestions that are determined based on the position, orientation, and angle of view from which a virtual camera image is determined, thereby ensuring recommended objects are contextually appropriate, which would save the user time and increase ease of use.
Regarding claim 16, Ito in view of Yalniz teaches all the limitations of claim 14, comprising:
wherein the user interface further includes, for each of the plurality of virtual object groups, a group image representing a spatial layout, in the virtual space, of the one or more virtual objects belonging to the virtual object group, the group image, the selectable control, and the recommendation indicator being associated with the same virtual object group (Ito Figs. 1-16; [0084], the obj file is explained by using diagrams. FIG. 7 is a diagram showing an example of the obj file that is designated in the 3D model format of virtual advertisement data. In the obj file, the vertex coordinate values are designated by a keyword “v”, the texture coordinate values are designated by a keyword “vt”, and a normal vector is designated by a keyword “vn”, respectively. Further, in the present embodiment, although the obj file is explained as an example of the file format representing information on the position, size, and orientation of the 3D model of virtual advertisement data, the file format is not limited to this; [0088], FIG. 6C is a diagram showing a format example of the virtual viewpoint image generation parameter (with virtual advertisement “A”) data (in the following, also referred to as parameter (with virtual advertisement “A” data)). In the format of the parameter (with virtual advertisement “A”) data, the information representing the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, and the time code are designated, which are also designated in the parameter (without virtual advertisement) shown in FIG. 6B. In the format of the parameter (with virtual advertisement “A”) data, information relating to virtual advertisement (in FIG. 6C, virtual advertisement “A”) display instructions is further designated. This parameter (with virtual advertisement “A”) includes instructions to display the virtual advertisement “A” ([“advertise ID”: “A” ] in FIG. 6A) that is instructed to be registered in the processing at S501 in the virtual viewpoint image; [0098-0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101-0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0147], FIG. 16B shows a format example of parameter (with virtual advertisement) data; see also [0028], [0033], [0045-0047], [0055], [0064-0065], [0093], [0096-0097], [0112], [0137-0139])
Regarding claim 17, Ito in view of Yalniz teaches all the limitations of claim 14. Yalniz further teaches:
wherein the selectable control associated with a virtual object group that is not determined as one of the one or more recommendation targets remains selectable by the user via the user interface (Yalniz Figs. 1-24; [0041], Referring next to FIG. 3, the client device 100 is shown capturing the digital image 106; [0042], changes in a viewing direction; [0045], the image analysis engine 215 may determine characteristics for each of the empty regions 406 which may be used in determining a recommended item for a corresponding one of the empty regions 406. For example, a first empty region 406a and a second empty region 406b may be determined to be an empty space on a wall using visual image analysis. In this example, a recommended item, such as a wall ornament, a canvas painting, wallpaper, etc., may be positioned in the first empty region 406a and/or the second empty region 406b, as will be described in greater detail below; [0046], Similarly, a third empty region 406c, a fourth empty region 406d, and a fifth empty region 406e may be determined to be an empty space on a floor using the same visual image analysis. In this example, a recommended item, such as a piece of furniture, a rug, etc., may be positioned in the third empty region 406c, the fourth empty region 406d, and/or the fifth empty region 406e; [0047], a first recommended item 296a and a second recommended item 296b (collectively recommended items 296) are shown in the third empty region 406c (FIG. 4A) and the fourth empty region 406d (FIG. 4A), respectively. As the third empty region 406c and the fourth empty region 406d were located on the floor of the setting, the recommendation service 218 may recommend items suitable for placement on the floor, such as a chaise lounge chair (recommended item 296a) and a flatwoven rug (recommended item 296b); [0048], the representative image 236 of the flatwoven rug (recommended item 296b), may be skewed, scaled and/or rotated such that it conforms with the perspective of the setting; [0049], additional information associated with the recommended items 296 may be shown to the user. As a non-limiting example, in FIG. 4B, the user may manipulate or otherwise engage the recommended item 296a to initiate a rendering of a dialog 412 that shows a name, a description, and an average review of the recommended item 296a; [0051], a list of the recommended items 296 may be shown in association with the modified form of the digital image 290. The list of the recommended items 296 may comprise a name, a description, a price, an average review, the representative images 236a . . . 236b, and/or other information associated with the recommended item 296; a first add to cart component 503a and a second add to cart component 503b (collectively add to cart components 503) may facilitate placement of the first recommended item 296a and the second recommended item 296b, respectively, in the virtual shopping cart upon a manipulation by the user; [0062], In 618, it is determined whether at least one empty region 406 was identified by the image analysis engine 212 in the digital image 106. If the at least one empty region 406 is not identified in the digital image 106, the process proceeds to 633 to send the list of the recommended items 296 to the client device 100, as will be discussed in greater detail below. As may be appreciated, the recommended items may be provided to the user based on the digital image 106 regardless of whether a representative image of the recommended item may be inserted in the digital image 106; [0068], In 633, a list of the recommended items 296 may be sent to the client device 100 for rendering)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein the selectable control associated with a virtual object group that is not determined as one of the one or more recommendation targets remains selectable by the user via the user interface as suggested in Yalniz into Ito. Doing so would be desirable because computer vision relates to analyzing, interpreting, understanding, and deriving information from digital images obtained via digital cameras or similar image capturing devices. Such information may be used in automated color detection, depth perception, and object recognition in computing environments (see Yalniz [0002]). The present disclosure relates to generating recommendations utilizing visual image analysis. It remains difficult for the user to perceive how an item may look in a particular setting. Accordingly, it may be beneficial for a user to provide a digital image of a setting in which the item may be visualized (see Yalniz [0013]). According to various embodiments, the electronic commerce system may communicate with various image processing services, such as an image analysis engine and/or an image modification engine, to analyze a digital image of a setting (e.g., a room) provided by a user. The analysis of the digital image may identify various characteristics about the setting, such as colors, sizes, dimensions, or locations of various objects in the setting. Further, the image processing services may identify regions in the setting in which no object is recognized (hereinafter referred to as empty regions). The electronic commerce system may leverage a recommendation service to identify an item to recommend to the user by utilizing the various characteristics about the room identified in the digital image and/or historical data associated with the user in the electronic commerce system (see Yalniz [0004]). Additionally, the system of Yalniz would improve the system of Ito by providing suggestions that are determined based on the position, orientation, and angle of view from which a virtual camera image is determined, thereby ensuring recommended objects are contextually appropriate, which would save the user time and increase ease of use.
Regarding claim 18, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein, when a plurality of virtual object groups are determined as recommendation targets, the user interface simultaneously indicates the plurality of virtual object groups as the recommendation targets (Ito Figs. 1-16; [0084], the obj file is explained by using diagrams. FIG. 7 is a diagram showing an example of the obj file that is designated in the 3D model format of virtual advertisement data. In the obj file, the vertex coordinate values are designated by a keyword “v”, the texture coordinate values are designated by a keyword “vt”, and a normal vector is designated by a keyword “vn”, respectively. Further, in the present embodiment, although the obj file is explained as an example of the file format representing information on the position, size, and orientation of the 3D model of virtual advertisement data, the file format is not limited to this; [0088], FIG. 6C is a diagram showing a format example of the virtual viewpoint image generation parameter (with virtual advertisement “A”) data (in the following, also referred to as parameter (with virtual advertisement “A” data)). In the format of the parameter (with virtual advertisement “A”) data, the information representing the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, and the time code are designated, which are also designated in the parameter (without virtual advertisement) shown in FIG. 6B. In the format of the parameter (with virtual advertisement “A”) data, information relating to virtual advertisement (in FIG. 6C, virtual advertisement “A”) display instructions is further designated. This parameter (with virtual advertisement “A”) includes instructions to display the virtual advertisement “A” ([“advertise ID”: “A” ] in FIG. 6A) that is instructed to be registered in the processing at S501 in the virtual viewpoint image; [0098-0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101-0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0147], FIG. 16B shows a format example of parameter (with virtual advertisement) data; see also [0028], [0033], [0045-0047], [0055], [0064-0065], [0093], [0096-0097], [0112], [0137-0139])
Yalniz further teaches:
wherein, when a plurality of virtual object groups are determined as recommendation targets based on the virtual camera parameter, the user interface simultaneously indicates the plurality of virtual object groups as the recommendation targets (Yalniz Figs. 1-24; [0041], Referring next to FIG. 3, the client device 100 is shown capturing the digital image 106; [0042], changes in a viewing direction; [0045], the image analysis engine 215 may determine characteristics for each of the empty regions 406 which may be used in determining a recommended item for a corresponding one of the empty regions 406. For example, a first empty region 406a and a second empty region 406b may be determined to be an empty space on a wall using visual image analysis. In this example, a recommended item, such as a wall ornament, a canvas painting, wallpaper, etc., may be positioned in the first empty region 406a and/or the second empty region 406b, as will be described in greater detail below; [0046], Similarly, a third empty region 406c, a fourth empty region 406d, and a fifth empty region 406e may be determined to be an empty space on a floor using the same visual image analysis. In this example, a recommended item, such as a piece of furniture, a rug, etc., may be positioned in the third empty region 406c, the fourth empty region 406d, and/or the fifth empty region 406e; [0047], a first recommended item 296a and a second recommended item 296b (collectively recommended items 296) are shown in the third empty region 406c (FIG. 4A) and the fourth empty region 406d (FIG. 4A), respectively. As the third empty region 406c and the fourth empty region 406d were located on the floor of the setting, the recommendation service 218 may recommend items suitable for placement on the floor, such as a chaise lounge chair (recommended item 296a) and a flatwoven rug (recommended item 296b); [0048], the representative image 236 of the flatwoven rug (recommended item 296b), may be skewed, scaled and/or rotated such that it conforms with the perspective of the setting; [0049], additional information associated with the recommended items 296 may be shown to the user. As a non-limiting example, in FIG. 4B, the user may manipulate or otherwise engage the recommended item 296a to initiate a rendering of a dialog 412 that shows a name, a description, and an average review of the recommended item 296a; [0051], a list of the recommended items 296 may be shown in association with the modified form of the digital image 290. The list of the recommended items 296 may comprise a name, a description, a price, an average review, the representative images 236a . . . 236b, and/or other information associated with the recommended item 296; a first add to cart component 503a and a second add to cart component 503b (collectively add to cart components 503) may facilitate placement of the first recommended item 296a and the second recommended item 296b, respectively, in the virtual shopping cart upon a manipulation by the user; [0062], In 618, it is determined whether at least one empty region 406 was identified by the image analysis engine 212 in the digital image 106. If the at least one empty region 406 is not identified in the digital image 106, the process proceeds to 633 to send the list of the recommended items 296 to the client device 100, as will be discussed in greater detail below. As may be appreciated, the recommended items may be provided to the user based on the digital image 106 regardless of whether a representative image of the recommended item may be inserted in the digital image 106; [0068], In 633, a list of the recommended items 296 may be sent to the client device 100 for rendering)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated wherein, when a plurality of virtual object groups are determined as recommendation targets based on the virtual camera parameter, the user interface simultaneously indicates the plurality of virtual object groups as the recommendation targets as suggested in Yalniz into Ito. Doing so would be desirable because computer vision relates to analyzing, interpreting, understanding, and deriving information from digital images obtained via digital cameras or similar image capturing devices. Such information may be used in automated color detection, depth perception, and object recognition in computing environments (see Yalniz [0002]). The present disclosure relates to generating recommendations utilizing visual image analysis. It remains difficult for the user to perceive how an item may look in a particular setting. Accordingly, it may be beneficial for a user to provide a digital image of a setting in which the item may be visualized (see Yalniz [0013]). According to various embodiments, the electronic commerce system may communicate with various image processing services, such as an image analysis engine and/or an image modification engine, to analyze a digital image of a setting (e.g., a room) provided by a user. The analysis of the digital image may identify various characteristics about the setting, such as colors, sizes, dimensions, or locations of various objects in the setting. Further, the image processing services may identify regions in the setting in which no object is recognized (hereinafter referred to as empty regions). The electronic commerce system may leverage a recommendation service to identify an item to recommend to the user by utilizing the various characteristics about the room identified in the digital image and/or historical data associated with the user in the electronic commerce system (see Yalniz [0004]). Additionally, the system of Yalniz would improve the system of Ito by providing suggestions that are determined based on the position, orientation, and angle of view from which a virtual camera image is determined, thereby ensuring recommended objects are contextually appropriate, which would save the user time and increase ease of use.
Regarding claim 19, Ito in view of Yalniz teaches all the limitations of claim 16, comprising:
wherein the group image is generated based on information indicating respective positions, in the virtual space, of the one or more virtual objects belonging to the corresponding virtual object group (Ito Figs. 1-16; [0084], the obj file is explained by using diagrams. FIG. 7 is a diagram showing an example of the obj file that is designated in the 3D model format of virtual advertisement data. In the obj file, the vertex coordinate values are designated by a keyword “v”, the texture coordinate values are designated by a keyword “vt”, and a normal vector is designated by a keyword “vn”, respectively. Further, in the present embodiment, although the obj file is explained as an example of the file format representing information on the position, size, and orientation of the 3D model of virtual advertisement data, the file format is not limited to this; [0088], FIG. 6C is a diagram showing a format example of the virtual viewpoint image generation parameter (with virtual advertisement “A”) data (in the following, also referred to as parameter (with virtual advertisement “A” data)). In the format of the parameter (with virtual advertisement “A”) data, the information representing the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, and the time code are designated, which are also designated in the parameter (without virtual advertisement) shown in FIG. 6B. In the format of the parameter (with virtual advertisement “A”) data, information relating to virtual advertisement (in FIG. 6C, virtual advertisement “A”) display instructions is further designated. This parameter (with virtual advertisement “A”) includes instructions to display the virtual advertisement “A” ([“advertise ID”: “A” ] in FIG. 6A) that is instructed to be registered in the processing at S501 in the virtual viewpoint image; [0098-0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101-0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0147], FIG. 16B shows a format example of parameter (with virtual advertisement) data; see also [0028], [0033], [0045-0047], [0055], [0064-0065], [0093], [0096-0097], [0112], [0137-0139])
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Ito in view of Yalniz in further view of Miyata et al. (US 20230396749 A1, published 12/07/2023), hereinafter Miyata.
Regarding claim 9, Ito in view of Yalniz teaches all the limitations of claim 1, comprising:
wherein the virtual camera parameter includes information indicating the position of the virtual camera, and the one or more programs further include instructions for: calculating a motion of the virtual camera based on positions of the virtual camera by virtual camera parameters corresponding parameters corresponding to different frames, and maintaining, without change, an indication of the one or more virtual object groups determined as the one or more recommendation targets in the user interface in a case where the motion of the virtual camera (Ito Figs. 1-16; [0033], the position of the virtual viewpoint, the view direction from the virtual viewpoint, and the angle of view of the virtual viewpoint respectively correspond to the position of the virtual camera, the orientation of the virtual camera, and the angle of view of the virtual camera; [0045-0046], The virtual viewpoint extraction unit 127 extracts, from the parameter received by the parameter reception unit 123, the virtual viewpoint information (virtual camera position information) indicating the position of the virtual camera, the orientation of the virtual camera, and the angle of view corresponding to the virtual camera, which are included in the parameter; [0047], The virtual viewpoint information includes the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera; [0098-0099], As shown in FIG. 9B, the display 901 on the left side displays a virtual viewpoint image 921 generated in accordance with the virtual viewpoint based on the basic 3D data. The display 902 on the right side displays a virtual advertisement display designation UI 922 in the state where the radio button for setting display of the virtual advertisement A to the image displayed on the display 901 on the left side is selected by the user operation. Display of a virtual advertisement is selected, and therefore, the display on the left side displays the virtual viewpoint image to which the virtual advertisement is added; [0101-0102], it may also be possible to enable adjustment of the virtual viewpoint by the user operation by using the virtual viewpoint image 911 displayed on the display 901 on the left side. In a case where the virtual viewpoint is adjusted (set) by the user operation, the parameter setting unit 114 sets the virtual viewpoint information to the parameter and the parameter transmission unit 118 transmits the parameter to the image generation apparatus 120; [0112], FIG. 12C shows a virtual advertisement schedule setting UI example; [0120], As shown in FIG. 12C, a virtual advertisement schedule setting UI 1240 is a UI for designating a schedule of displaying a virtual advertisement for a virtual viewpoint image; [0123], the position of the advertisement signboard moves from the position on the left side to the position on the right side in the image; [0127], it is made possible to produce a display in which the contents of the advertisement move by switching the displays in order of the virtual advertisement 1401, the virtual advertisement 1402, and the virtual advertisement 1403; [0146], a 3D model and moving image data of a virtual advertisement are registered in advance and a virtual advertisement whose texture moves is displayed in a virtual viewpoint image by designating the moving image data or the frame number of the moving image data, which are already registered along with the parameter; [0147-0148], moving image data (parameter) of a 3D model of a virtual advertisement is stored. Here, a parameter (with virtual advertisement) data example is explained by using diagrams. In the format of the parameter (with virtual advertisement) data shown in FIG. 16B, as virtual advertisement display instructions, “AAA: FWD” representing “advertisement ID: moving image reproduction method” is designated. Here, FWD represents the uniform reproduction in the forward direction. By designating the reproduction method of a moving image at the time of setting the various kinds of information to the parameter as described above, it is made possible to display a moving virtual advertisement; see also [0028], [0064-0065], [0093], [0096-0097], [0112], [0137-0139], [0147])
Yalniz further teaches:
maintaining an indication of the one or more virtual object groups determined as the one or more recommendation targets in the user interface (Yalniz Figs. 1-24; [0041], Referring next to FIG. 3, the client device 100 is shown capturing the digital image 106; [0042], changes in a viewing direction; [0045], the image analysis engine 215 may determine characteristics for each of the empty regions 406 which may be used in determining a recommended item for a corresponding one of the empty regions 406. For example, a first empty region 406a and a second empty region 406b may be determined to be an empty space on a wall using visual image analysis. In this example, a recommended item, such as a wall ornament, a canvas painting, wallpaper, etc., may be positioned in the first empty region 406a and/or the second empty region 406b, as will be described in greater detail below; [0046], Similarly, a third empty region 406c, a fourth empty region 406d, and a fifth empty region 406e may be determined to be an empty space on a floor using the same visual image analysis. In this example, a recommended item, such as a piece of furniture, a rug, etc., may be positioned in the third empty region 406c, the fourth empty region 406d, and/or the fifth empty region 406e; [0047], a first recommended item 296a and a second recommended item 296b (collectively recommended items 296) are shown in the third empty region 406c (FIG. 4A) and the fourth empty region 406d (FIG. 4A), respectively. As the third empty region 406c and the fourth empty region 406d were located on the floor of the setting, the recommendation service 218 may recommend items suitable for placement on the floor, such as a chaise lounge chair (recommended item 296a) and a flatwoven rug (recommended item 296b).; [0048], the representative image 236 of the flatwoven rug (recommended item 296b), may be skewed, scaled and/or rotated such that it conforms with the perspective of the setting; [0049], additional information associated with the recommended items 296 may be shown to the user; [0051], a list of the recommended items 296 may be shown in association with the modified form of the digital image 290. The list of the recommended items 296 may comprise a name, a description, a price, an average review, the representative images 236a . . . 236b, and/or other information associated with the recommended item 296; [0067], given the user of the client device 100 has provided a plurality of digital images 106 of the setting (or a video of the setting), the image modification engine 221 may stitch the digital images 106 provided to generate a three-dimensional model of the setting that may be encoded for rendering in the user interface 272. Further, construction of the three-dimensional model may employ motion estimation, homography, various camera models, etc)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated maintaining an indication of the one or more virtual object groups determined as the one or more recommendation targets in the user interface as suggested in Yalniz into Ito. Doing so would be desirable because computer vision relates to analyzing, interpreting, understanding, and deriving information from digital images obtained via digital cameras or similar image capturing devices. Such information may be used in automated color detection, depth perception, and object recognition in computing environments (see Yalniz [0002]). The present disclosure relates to generating recommendations utilizing visual image analysis. It remains difficult for the user to perceive how an item may look in a particular setting. Accordingly, it may be beneficial for a user to provide a digital image of a setting in which the item may be visualized (see Yalniz [0013]). According to various embodiments, the electronic commerce system may communicate with various image processing services, such as an image analysis engine and/or an image modification engine, to analyze a digital image of a setting (e.g., a room) provided by a user. The analysis of the digital image may identify various characteristics about the setting, such as colors, sizes, dimensions, or locations of various objects in the setting. Further, the image processing services may identify regions in the setting in which no object is recognized (hereinafter referred to as empty regions). The electronic commerce system may leverage a recommendation service to identify an item to recommend to the user by utilizing the various characteristics about the room identified in the digital image and/or historical data associated with the user in the electronic commerce system (see Yalniz [0004]). Additionally, the system of Yalniz would improve the system of Ito by providing suggestions that are determined based on the position, orientation, and angle of view from which a virtual camera image is determined, thereby ensuring recommended objects are contextually appropriate, which would save the user time and increase ease of use.
However, Ito in view of Yalniz fails to expressly disclose calculating a moving speed of the virtual camera based on positions of the virtual camera by virtual camera parameters corresponding parameters corresponding to different frames, and maintaining, without change, an indication of the one or more virtual object groups determined as the one or more recommendation targets in the user interface in a case where the moving speed of the virtual camera is equal to or greater than a threshold value. In the same field of endeavor, Miyata teaches:
calculating a moving speed of the virtual camera based on positions of the virtual camera by virtual camera parameters corresponding parameters corresponding to different frames, and maintaining, without change, an indication of the one or more virtual object groups determined as the one or more recommendation targets in the user interface in a case where the moving speed of the virtual camera is equal to or greater than a threshold value (Miyata Figs. 1-24; [0005], an image processing apparatus, an image processing method, and a program which can show an object image to a viewer of a virtual viewpoint image; [0105], the viewpoint and the gaze point are set, a virtual visual line direction (imaging direction of the virtual camera) is also uniquely decided. The virtual visual line direction refers to a direction from the viewpoint to the gaze point; [0113], The movement speed information 74E is information for specifying a movement speed of the position of the viewpoint for observing the subject on the viewpoint path P1; [0114], The angle-of-view information 74F is information for specifying an angle of view (hereinafter, also simply referred to as an “angle of view”). Here, the angle of view refers to an angle of view for observing the subject on the viewpoint path P1. In the present embodiment, the angle of view is fixed to a predetermined angle (for example, 100 degrees). It should be noted that this is merely an example, and the angle of view may be decided according to the movement speed. In this case, for example, within a range in which an upper limit (for example, 150 degrees) and a lower limit (for example, 15 degrees) of the angle of view are decided, the angle of view is narrower as the movement speed is lower. In addition, the angle of view may be narrower as the movement speed is higher; [0120], the virtual viewpoint image generation unit 28C acquires the virtual viewpoint image 76 including an advertisement sign image 77 (see FIG. 11) showing the advertisement sign 6 based on the viewpoint information 74 and the positional information 39; [0126], generates the virtual viewpoint images 76 of a plurality of frames according to the viewpoint path P2 (see FIG. 9); [0186], in a case in which the required time indicated by the required time information 74C (see FIG. 8) is equal to or longer than a first threshold value (for example, 30 seconds), the virtual viewpoint image generation unit 28C may change the visual line direction with reference to the positional information 39 such that the advertisement sign image 77 is included in the virtual viewpoint image 76 during a period until the required time elapses. In addition, for example, in a case in which the elapsed time indicated by the elapsed time information 74D (see FIG. 8) is equal to or longer than a second threshold value (for example, 10 seconds), the virtual viewpoint image generation unit 28C may change the visual line direction with reference to the positional information 39 such that the advertisement sign image 77 is included in the virtual viewpoint image 76 at a position of the viewpoint corresponding to the elapsed time information 74D indicating the elapsed time that is equal to or longer than the second threshold value. In addition, for example, in a case in which the movement speed specified from the movement speed information 74E (see FIG. 8) is equal to or higher than a third threshold value (for example, 1 meter/second), the virtual viewpoint image generation unit 28C may change the visual line direction with reference to the positional information 39 such that the advertisement sign image 77 is included in the virtual viewpoint image 76 at a position of the viewpoint corresponding to the movement speed that is equal to or higher than the third threshold value. In addition, for example, in a case in which a change amount per unit time of the angle of view specified from the angle-of-view information 74F (see FIG. 8) is equal to or larger than a fourth threshold value (for example, 30 degrees), the virtual viewpoint image generation unit 28C may change the visual line direction with reference to the positional information 39 such that the advertisement sign image 77 is included in the virtual viewpoint image 76 at a position of the viewpoint at which the change amount per unit time is equal to or larger than the fourth threshold value)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have incorporated calculating a moving speed of the virtual camera based on positions of the virtual camera by virtual camera parameters corresponding parameters corresponding to different frames, and maintaining, without change, an indication of the one or more virtual object groups determined as the one or more recommendation targets in the user interface in a case where the moving speed of the virtual camera is equal to or greater than a threshold value as suggested in Miyata into Ito in view of Yalniz. Doing so would be desirable because the system of Miyata would improve the system of Ito by enabling the user to easily adjust desired aspects of the viewpoint, such as allowing the user to flexibly set a movement speed of the position of the viewpoint (see Miyata [0113]). The system of Miyata would enable the interface to be unchanged by still successfully advertisements when movements speeds are above a threshold (see Miyata [0186]), thereby increasing the usefulness and useability of the system. The system of Miyata would improve the advertising display of Ito by additionally enabling enhancement and emphasis of the advertisement even when the user adjusts the visual line direction, such as by changing movement speed (see Miyata [0186-0187])
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to claims 1 and 3-13, the addition of claims 14-19, and the cancellation of claim 2. The correction to the title is approved and the objection to the title is respectfully withdrawn. The previous objections to claims 10 and 11 are respectfully withdrawn. Claims 1 and 3-19 stand rejected under 35 U.S.C. 112(b), as necessitated by the amendments.
Regarding independent claim 1, the Applicant alleges that Ito as described in the previous Office action, does not explicitly teach amended claim 1. Examiner has therefore rejected independent claim 1 under 35 U.S.C § 103 as unpatentable over Ito in view of Yalniz.
Similar arguments have been presented for claims 12 and 13 and thus, Applicant’s arguments are not persuasive for the same reasons.
Applicant states that the dependent claims recite all the limitations of the independent claims, and thus, are allowable in view of the remarks set forth regarding the independent claims. However, as discussed above, Ito in view of Yalniz is considered to teach the independent claims, and consequently, the dependent claims are rejected.
Regarding dependent claim 9, the Applicant alleges that Ito in view of Miyata as described in the previous Office action, does not explicitly teach amended claim 9. Examiner has therefore rejected claim 9 under 35 U.S.C § 103 as unpatentable over Ito in view of Yalniz in further view of Miyata. With respect to Applicant’s argument that Miyata does not maintain the display of the recommendation target on the UI without changing it, Examiner notes the claim appears to only require an “indication” to be maintained (see also interpretation of claim 9 above with respect to 112(b)). The claims do not place any limitations on what the indication must comprise. As discussed above, Miyata discloses a virtual viewpoint image generation unit 28C acquires the virtual viewpoint image 76 including an advertisement sign image 77 (see FIG. 11) showing the advertisement sign 6 based on the viewpoint information 74 and the positional information 39 ([0120]). In a case in which the movement speed specified from the movement speed information 74E (see FIG. 8) is equal to or higher than a third threshold value (for example, 1 meter/second), the virtual viewpoint image generation unit 28C may change the visual line direction with reference to the positional information 39 such that the advertisement sign image 77 is included in the virtual viewpoint image 76 at a position of the viewpoint corresponding to the movement speed that is equal to or higher than a threshold value ([0186]). Thus, the combination of Ito in view of Yalniz in further view of Miyata is considered to teach claim 9.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Fang (US 20200111261 A1) Fig. 11 and [0061], [0081-0083].
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN T REPSHER III whose telephone number is (571)272-7487. The examiner can normally be reached Monday - Friday, 8AM-5PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN T REPSHER III/ Primary Examiner, Art Unit 2143
Prosecution Insights