DETAILED ACTION
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 3, 4, 7, 9, 10 – 14, 17, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ahmed et al. (Publication: US 2022/0180898 A1) in view of Tomaru (Publication: US 2021/0256760 A1) and Malia et al. (Publication: US 2021/0097768 A1).
Regarding claim 1, see rejection on claim 19.
Regarding claim 2, see rejection on claim 20.
Regarding claim 3, see rejection on claim 13.
Regarding claim 4, see rejection on claim 14.
Regarding claim 7, see rejection on claim 17.
Regarding claim 9, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 1.
Ahmed discloses further comprising generating a relative pixel depth map associated with a frame included in the input video sequence ([0029] , [0065] - Replacement media generation constraints 410 may include rules that can be used to match secondary content items with candidate insertion points of media content items. More particularly, the rules may include or be used to generate a “mapping” between compatible scene types (e.g., kitchen counter in comedy, billboard outside in drama) and attributes of compatible secondary content (e.g., product type, product category, brand). Constraints 410 can specify secondary content items (or associated attributes) that are permitted (or, conversely, not permitted) to be “placed” in primary content. For example, the primary content can include a specific media item, a specific clip/scene, or a specific candidate placement. As another example, the primary content may be a video title, clip, or candidate placement having specific attributes. Each secondary content item represents a three-dimensional object, 3D information indicating a depth of the first object, having a flat surface that is suitable for placement on a kitchen counter. In this example, secondary content items 128 include a beer bottle, a soda can, and an Amazon Echo. Replacement clips 130 are generated using original clip 122 and secondary content items 128. Each replacement clip 130 is the same length and corresponds to the same segment of time. Thus Mapping involves 3D information indicating a depth of the secondary content item in the clip, video sequence.).
Regarding claim 10, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 1.
Ahmed discloses further comprising generating an ordered list of combinations of virtual objects and planar surfaces based at least on the suitability metric ([0055] - Attributes of a candidate placement can further include a duration of time that the candidate placement is visible on screen. a candidate placement can include attributes of object(s) on which secondary content can be “placed” (e.g., categorization, relative size of the object relative to other content, pixel coordinates, etc.) and/or attribute(s) surface(s) on which the secondary content can be “placed” (e.g., whether the surface is vertical or horizontal, whether the surface is flat or curved, relative size of the surface, etc.).
[0016] - corresponding to surfaces of object(s) represented within the clip on which a virtual product (e.g. Amazon logo), object, or branding can be placed .).
Regarding claim 11, see rejection on claim 19.
Regarding claim 12, see rejection on claim 20.
Regarding claim 13, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 12.
Ahmed discloses wherein the suitability metric is further based on a size or a duration associated with the virtual object (
[0055] - Attributes of a candidate placement can further include a duration of time that the candidate placement is visible on screen. a candidate placement can include attributes of object(s) on which secondary content can be “placed” (e.g., categorization, relative size of the object relative to other content, pixel coordinates, etc.) and/or attribute(s) surface(s) on which the secondary content can be “placed” (e.g., whether the surface is vertical or horizontal, whether the surface is flat or curved, relative size of the surface, etc.).
[0016] - corresponding to surfaces of object(s) represented within the clip on which a virtual product (e.g. Amazon logo), object, or branding can be placed .).
Regarding claim 14, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 11.
Ahmed discloses wherein the one or more machine learning models include [[a rendering generator or a diffusion generator] -] - ([0086] - Machine learning algorithms used to ascertain media attributes may be automatically updated as new data is collected. This data can include additional attributes that are ascertained, for example, based on analysis of frames of video titles as they are added to video title catalog 702. In addition, the data can include human input that validates (or invalidates) the accuracy of attributes determined via the machine learning algorithms, as well as human input that modifies previously identified attributes of video titles or adds additional attributes to those previously identified.).
Tomaru discloses
include a rendering generator or a diffusion generator ([0123] - The generation of the image of the virtual object in the three-dimensional drawer 45 “rendering generator”can be implemented by performing rendering by using three-dimensional graphics.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Ahmed in view Tomaru with include a rendering generator or a diffusion generator as taught by Tomaru. The motivation for doing so the user can institutively perceive the position of an object.
Regarding claim 17, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 11.
Tomaru discloses to perform the step of generating a polygon defining a boundary of the planar surface ([0119] - As illustrated in FIGS. 8 and 9A, a pair of straight lines Lθa and Lθb forming the angles θa and θb with the Zv-axis in the Xv-Zv plane in the user viewpoint coordinate system are drawn, and boundary surfaces Sθa and Sθb are formed by trajectories obtained by moving the straight lines Lθa and Lθb in the Yv-axis direction. Although the straight lines Lθa and Lθb and boundary surfaces Sθa and Sθb extend infinitely, FIG. 9A illustrates only part of the straight lines Lθa and Lθb and boundary surfaces Sθa and Sθb. The boundary surfaces Sθa and Sθb can be said to be surfaces that define the area of the reference horizontal view angle.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Ahmed with select a virtual object included in an object library as taught by Tomaru. The motivation for doing so the user can institutively perceive the position of an object.
Regarding claim 19, Ahmed discloses a system comprising ([0038] , [0040] - FIG. 3, a computing environment in which a video content service 302 provides streaming content (e.g., video and/or audio) via network 304 to a variety of client devices (306-1 through 306-5).):
one or more memories storing instructions ([0038] - FIG. 3, memory stores instructions. ); and
one or more processors for executing the instructions to ([0038] - FIG. 3, the processor to execute instructions.) :
identify a planar surface depicted in an input video sequence ([0020], [0073] - surface of box is identified as a candidate placement for secondary content in the clip, frames of the movie “sequence”.
[0073] - the clip, frames of the movie, is retrieved from primary content information, “input video sequence, frames” .);
generate, for a combination of the planar surface and the virtual object, a suitability metric associated with the combination (
[0065] - rules that can be used to match secondary content items “a suitability metric” with candidate insertion points of media content items. More particularly, the rules may include or be used to generate a “mapping” between compatible scene types (e.g., kitchen counter in comedy, billboard outside in drama) and attributes of compatible secondary content (e.g., product type, product category, brand), “generate a suitability metric”.
[0026] - As shown in FIG. 1A, replacement clip is generated using original clip and a secondary content overlay “combination” including an Amazon logo based on the rules used to match. In this example, vertical surface 116 of Amazon box 118 represented in replacement clip contains an Amazon logo, which replaces the text previously shown on vertical surface of box in original clip. Since the actor is moving box in the scene, pixel coordinates of candidate placement may vary among frames in replacement clip, “for a combination of the planar surface and the virtual object; a suitability metric associated with the combination”.
[0016] - corresponding to surfaces of object(s) represented within the clip on which a virtual product (e.g. Amazon logo), object, or branding can be placed, “virtual object”
PNG
media_image1.png
338
612
media_image1.png
Greyscale
),
wherein the suitability metric is based at least on a plurality of per-frame determination of data(
[0020], [0073] - surface of box is identified as a candidate placement for secondary content in the clip, frames of the movie.
[0051], [0065] - rules that can be used to match secondary content items with candidate insertion points of media content items. More particularly, the rules may include or be used to generate a “mapping” between compatible scene types (e.g., kitchen counter in comedy, billboard outside in drama) and attributes of compatible secondary content (e.g., product type, product category, brand), “suitability metric”. For a given title, clips corresponding to candidate insertion points may be scored according to a set of scoring algorithms, where each candidate insertion point clip includes a representation of an virtual object or associated object surface on which secondary content can be virtually placed. For example, the clip can correspond to a scene in a movie. Candidate insertion points at which segments of a video content item can be replaced with a corresponding replacement clip may be selected based, at least in part, on the scores, “suitability metric is based at least on a semantic compatibility between the virtual object and the planar surface”.
[0016] - candidate replacement, e.g. Amazon logo, corresponding to surfaces of object(s) represented within the clip on which a virtual product (e.g. Amazon logo), object, or branding can be placed, “virtual object”); and
generate, via one or more machine learning models, a modified video sequence based on the suitability metric (
[0026] - As shown in FIG. 1A, replacement clip “video” is generated using original clip and a secondary content overlay including an Amazon logo, “generate”.
Machine learning techniques can be used to identify candidate insertion points , as well as attributes of an identified candidate insertion point. A candidate insertion point may be defined by a clip (e.g., scene) within the media content item, as well as candidate placement(s) “modified” corresponding to surfaces of physical object(s) represented within the clip on which a virtual product, (e.g. Amazon logo) or branding can be placed. Attributes of a candidate insertion point can include a start time of the pertinent clip, an end time and/or duration of the pertinent clip, and attributes of pertinent physical object(s) represented within the clip, “generate, via one or more machine learning models, a modified video sequence”.
[0065] - include rules that can be used to match secondary content items with candidate insertion points of media content items. More particularly, the rules may include or be used to generate a “mapping” between compatible scene types (e.g., kitchen counter in comedy, billboard outside in drama) and attributes of compatible secondary content (e.g., product type, product category, brand), (e.g. Amazon logo), “generate a modified video sequence based on the suitability metric”.
[0016] - The information that is collected and stored for a media content item may be analyzed to identify “candidate insertion points” at which secondary content may be “inserted” to achieve virtual product placements “modified” or branding. A candidate insertion point is a potential two-dimensional (2D) placement for a virtual product or branding. ),
wherein the modified video sequence depicts the virtual object placed on the planar surface ([0016] - A candidate insertion point may be defined by a clip “video” (e.g., scene) within the media content item, as well as candidate placement(s) “modified” corresponding to surfaces of physical object(s) represented within the clip on which a virtual product or branding can be placed, “modified video sequence depicts the virtual object placed on the planar surface”.
[0026] - As shown in FIG. 1A, replacement clip “video” is generated using original clip and a secondary content overlay including an Amazon logo.
[0016] - candidate replacement “modified”, e.g. Amazon logo, corresponding to surfaces of object(s) represented within the clip on which a virtual product (e.g. Amazon logo), object, or branding can be placed, “virtual object”
Attributes of a candidate insertion point can include a start time of the pertinent clip, an end time and/or duration of the pertinent clip, and attributes of pertinent physical object(s) represented within the clip. Attributes of an object represented within a clip can include a classification of the object (e.g., a desk, a bus, an empty wall, a billboard). Attributes of a candidate insertion point can also include attributes of the pertinent clip (e.g., scene) and/or media content item such as a setting or genre.
PNG
media_image2.png
314
612
media_image2.png
Greyscale
).
Ahmed does not however Tomaru discloses
select a virtual object included in an object library ([0094] - a virtual object selector to select one of the plurality of virtual objects “library” stored in the information storage.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Ahmed with select a virtual object included in an object library as taught by Tomaru. The motivation for doing so the user can institutively perceive the object.
Ahmed in view Tomaru do not disclose; however, Malia discloses
(i) an extent to which the virtual object will be occluded by one or more different objects and (ii) whether the inserted object will be in focus ([0304] - from the first view of the physical environment, is occluded by the one or more respective physical objects “occluded by one or more different objects”, changing (918) an appearance of (e.g., virtual stool 542 is shown in deemphasized state in FIGS. 5P-5Q) so as to deemphasize (e.g., for example by displaying as at least partially transparent, forgoing displaying, and/or displaying (e.g., only) an outline of) the one or more portions of the virtual object that overlap with the one or more representations of respective physical objects (e.g., to represent the portion of the virtual object being partially blocked (e.g., occluded) from view by the representation of the first physical object). In some embodiments, a portion of the virtual object that is occluded is deemphasized, while another portion of the virtual object is not occluded and thus not deemphasized (e.g., the texture is displayed). A visual change to a virtual object (e.g., deemphasizing) illustrates to the user that the virtual object is occluded by a physical object, which provides the user with depth information of the virtual object in the physical environment, “in focus”. ).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Ahmed in view Tomaru with (i) an extent to which the virtual object will be occluded by one or more different objects and (ii) whether the inserted object will be in focus as taught by Malia. The motivation for doing so the task is automat thus make the operation more efficiency as taught by Malia.
Regarding claim 20, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 19.
Ahmed discloses wherein the suitability metric is further based on a semantic compatibility between the virtual object and a scene depicted in the input video sequence ([0055] - Attributes of a candidate placement can further include a duration of time that the candidate placement is visible on screen. a candidate placement can include attributes of object(s) on which secondary content can be “placed” (e.g., categorization, relative size of the object relative to other content, pixel coordinates, etc.) and/or attribute(s) surface(s) on which the secondary content can be “placed” (e.g., whether the surface is vertical or horizontal, whether the surface is flat or curved, relative size of the surface, etc.).
[0016] - corresponding to surfaces of object(s) represented within the clip, “input video sequence” , on which a virtual product (e.g. Amazon logo), object, or branding can be placed, “virtual object” .).
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Ahmed et al. (Publication: US 2022/0180898 A1) in view of Tomaru (Publication: US 2021/0256760 A1), Malia et al. (Publication: US 2021/0097768 A1), and Border et al. (Publication: US 2012/0019928 A1). .
Regarding claim 8, see rejection on claim 18.
Regarding claim 18, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 11.
Ahmed in view of Tomaru, Malia do not however Border discloses
to perform the step of generating, for each of one or more pixels included in the planar surface, a normal vector describing an orientation of the pixel ([0026] , [0029] , [0038] - generate the norma map and the normal map 244 indicates surface normals in the rendered image 240. For example, the normal map 244 could include a respective vector for each pixel of the character indicating a surface normal direction.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Ahmed in view of Tomaru, Malia with to perform the step of generating, for each of one or more pixels included in the planar surface, a normal vector describing an orientation of the pixel as taught by Border. The motivation for doing so to effectively improve rendering.
Claims 5, 6, 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ahmed et al. (Publication: US 2022/0180898 A1) in view of Tomaru (Publication: US 2021/0256760 A1), Malia et al. (Publication: US 2021/0097768 A1), and Gusmao et al. (NPL: DeepPlacer: A custom integrated OpAmp placement tool using deep models, Applied Soft Computing, Volume 115, January 2022, 108188)
Regarding claim 5, see rejection on claim 15.
Regarding claim 6, see rejection on claim 16.
Regarding claim 15, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 11.
Ahmed discloses perform the step of iteratively modifying one or more input parameters associated with the one or more machine learning models based on a function ([0016] , [0086] - Machine learning algorithms used to ascertain media attributes may be automatically updated as new data is collected, “modifying one or more input parameter”. Computer vision and machine learning techniques can be used to identify candidate insertion points, as well as attributes of an identified candidate insertion point. A candidate insertion point may be defined by a clip (e.g., scene) within the media content item, as well as candidate placement(s) corresponding to surfaces of physical object(s) represented within the clip on which a virtual product or branding can be placed. Attributes of a candidate insertion point can include a start time of the pertinent clip, an end time and/or duration of the pertinent clip, and attributes of pertinent physical object(s) represented within the clip. Attributes of an object represented within a clip can include a classification of the object (e.g., a desk, a bus, an empty wall, a billboard). Attributes of a candidate insertion point can also include attributes of the pertinent clip (e.g., scene) and/or media content item such as a setting or genre.
[0089] - the candidate placement identification process may be applied, repeatedly, “iteratively”. ).
Ahmed in view of Tomaru, Malia do not however Border discloses
Perform based on a generation loss function (Page 1 – innovative formulation of a generation loss function).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Ahmed in view of Tomaru, Malia with Perform based on a generation loss function as taught by Border. The motivation for doing so to effectively improve the design by automation.
Regarding claim 16, Ahmed in view of Tomaru, Malia disclose all the limitation of claim 11.
Ahmed discloses to perform the step of iteratively modifying one or more placement parameters based on a placement function ([0106] , Fig. 9 - , replacement clips may be selected for more than one candidate insertion point with an emphasis on repeated exposure to the same product or brand. If there are further candidate insertion points for the video (918), the replacement clip selection process may repeat “iteratively” for subsequent candidate insertion points (910). Step 918 in Fig. 9 is “placement parameters based on a placement function”).
Ahmed in view of Tomaru, Malia do not however Border discloses
Perform based on a generation loss function (Page 1 – innovative formulation of a generation loss function).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Ahmed in view of Tomaru, Malia with Perform based on a generation loss function as taught by Border. The motivation for doing so to effectively improve the design by automation.
Response to Arguments
Claim Rejection Under 35 U.S.C. 103
Applicant asserts “Claim 19 as currently amended recites that the suitability metric is based at least on a plurality of per-frame determinations of (i) an extent to which the virtual object will be occluded by one or more different objects and (ii) whether the inserted object will be in focus._Neither the Ahmed reference nor the Tomaru reference cited by the Examiner discloses, teaches, or suggests this limitation. A careful review of the other references cited by the Examiner shows that these references also fail to disclose, teach, or suggest the above limitation of amended claim 19.”
The argument has been fully considered and is persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Malia reference.
Regarding claims 2 – 10, 12 – 18, and 20 the Applicant asserts that they are not obvious over based on their dependency from independent claims 1, 11, and 19 respectively. The examiner cannot concur with the Applicant respectfully from same reason noted in the examiner’s response to argument asserted from claims 1, 11, and 19 respectively.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ming Wu whose telephone number is (571) 270-0724. The examiner can normally be reached on Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ming Wu/
Primary Examiner, Art Unit 2618
Prosecution Insights