DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 4, 6, 7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2, 4, 6, 7 recites “preferably” limitation clause. “preferably” indicates desirable or optional feature, but not necessarily required feature, which deems the scope of the related limitations indefinite. Clarification is required. For compact prosecution, Examiner interprets the related feature as optional.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chai et al. (US 2025/0391108 A1) in view of Ryland et al. (US 2024/0256866 A1).
Regarding claim 1, Chai teaches:
A method for an automated generation of synthetic scenes, (Abstract: “A 3D-aware generative model for high-quality and controllable scene synthesis uses an abstract object-level representation”) comprising the following steps:
- providing scene data representing a plurality of template scenes, (FIG. 3, [0039], “the user provides a layout prior 110”)
- providing object data specifying various objects for insertion into the synthetic scene, (FIG. 3, [0040], “generate an object for each 3D bounding box 132 at 310.”)
- providing at least one scene parameter for the template scenes, which describes at least one characteristic of the template scenes, ([0029], “Similar to NeRF, a pinhole camera model may be used to perform ray casting. For each object, the points on the rays can be sampled at adaptive depths rather than fixed ones since the bounding box provides clues about where the object locates. Specifically, the cast rays R={r.sub.j|j ∈[1, S.sup.2]} in a resolution S are transformed into the canonical object coordinate system. Then, a Ray-AABB (axis-aligned bounding box) intersection algorithm may be applied to calculate the adaptive near and far depth (d.sub.j,l,n, d.sub.j,l,f) of the intersected segment between the ray r.sub.j and the l-th box B.sub.l. After that, N.sub.d points are sampled equidistantly in the interval [d.sub.j,l,n, d.sub.j,l,f]. An intersection matrix M is maintained of size N×S.sup.2, whose elements indicate if this ray intersects with the box. With M, valid points are selected to infer, which can greatly reduce the rendering cost. Different background sampling strategies are adopted depending on the dataset. In general, fixed depth sampling is performed for bounded backgrounds in indoor scenes and the inverse parametrization of NeRF++ is inherited for complex and unbounded outdoor scenes, which uniformly samples background points in an inverse depth range..”)
- generating intermediate representations for the synthetic scenes, [0037], “receives a layout prior 110 of scenes in the input images that is used by the object generator 120 to generate an object for each 3D bounding box 132. At 230, the background of the scenes is separately generated. The volume renderer 152 of the neural rendering pipeline 150 generates a low resolution version of the scene from the objects for each 3D bounding box and the background of the scenes at 240.”) wherein, regarding the intermediate representations, at least one selected object is in each case inserted into a selected template scene, ([0021], “As shown, conditioned by the layout prior 110, the object generator 120 generates spatially disentangled generative radiance fields 130 of individual objects. For example, bounding boxes 132 may be placed around respective vehicles in a scene as shown. The scene layout is defined and the coordinates of the bounding boxes are identified to determine the absolute positions in the scene.”) wherein the insertion of the at least one selected object is parameterized based on the provided scene parameter in order to account for physical plausibility in the respective intermediate representation,([0022], “layout of the scene is provided as an explicit layout prior 110 to disentangle objects. Based on the layout prior 110, spatially disentangled radiance fields 130 and a neural rendering pipeline 150 achieve controllable 3D-aware scene generation.” In order to generate a 3D-aware scene, the depth information is calculated and considered in the intermediate image generation process. [0031], “In the methods described herein, objects are always assumed to be in front of the background. So objects and background can be rendered independently first and composited thereafter. For a ray r.sub.j intersecting with n.sub.j(n.sub.j≥1) boxes, its sample points X.sub.j={x.sub.j,k|k ∈[1, n.sub.jN.sub.d]} can be easily obtained from the depth range and the intersection matrix M. Since rendering should consider inter-object occlusions, the points X are sorted by depth, resulting in an ordered point set X.sub.j.sup.s={x.sub.j,sk|s.sub.k ∈[1, n.sub.jN.sub.d], d.sub.j,sk≤d.sub.j,sk+1}, where d.sub.j,sk denotes the depth of point x.sub.j,sk. With color c(x.sub.j,s.sub.k) and density σ(x.sub.j,s.sub.k) of the ordered set inferred with G.sub.obj(.Math.) by Equation (5), the corresponding pixel f(r.sub.j) may be calculated as:”)
However, Chai does not explicitly, but Ryland teaches:
- determining conditioning data from the generated intermediate representations in order to also account for physical plausibility in the synthetic scenes,- initiating the generation of the synthetic scenes based on the determined conditioning data. ([0013], “These raw synthetic images can be processed by a generative NN, such as a cycle generative adversarial network (CycleGAN). The generative NN can remove typical image artifacts associated with the fake objects and the real objects over real backgrounds. For example, the generative network adds and adjusts complex interactions with lighting, color matching with similar objects, decals and weathering, and edge clarity for the fake objects, among other more subtle tells of fakery. The generative network also adds details associated with sensor characteristics like dynamic range, signal-to-noise ratio (SNR), and blur. These sensor characteristics are effects that can be compensated by the generative model. This processing by the generative model (sometimes called a “generative model”) provides tailored forgeries based on real-world sensing data that is more convincing to a human and machine analyst than an image generated by a 3D editor.”)
Chai teaches steps of generating a synthetic scene. Ryland also teaches steps of generating a synthetic scene. In Ryland’s method, the computer-generated intermediate image is further enhanced by considering the data from intermediate image, so as to generate a final real scene image.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Chai with the image condition aware image enhance step of Ryland to generate a more real scene image.
Regarding claim 2, Chai in view of Ryland teaches:
The method according to claim 1, characterized in that the synthetic scenes are generated by a generative model, preferably by a diffusion model which generates the synthetic scenes based on a scene definition, preferably a text prompt, wherein the generation is conditioned by the conditioning data, wherein the conditioning data are preferably used for this purpose during training of the model in order to influence the outputs of the model. (Ryland [0013], “These raw synthetic images can be processed by a generative NN, such as a cycle generative adversarial network (CycleGAN). The generative NN can remove typical image artifacts associated with the fake objects and the real objects over real backgrounds. For example, the generative network adds and adjusts complex interactions with lighting, color matching with similar objects, decals and weathering, and edge clarity for the fake objects, among other more subtle tells of fakery. The generative network also adds details associated with sensor characteristics like dynamic range, signal-to-noise ratio (SNR), and blur. These sensor characteristics are effects that can be compensated by the generative model. This processing by the generative model (sometimes called a “generative model”) provides tailored forgeries based on real-world sensing data that is more convincing to a human and machine analyst than an image generated by a 3D editor.” The combination of claim 1 is incorporated here.)
Regarding claim 3, Chai in view of Ryland teaches:
The method according to claim 1, characterized in that the provided scene parameter comprises at least depth map, and/or semantic labelmap, and/or Canny edge representation for use in automatically determining a position, and/or orientation, and/or size of the objects for insertion into the template scenes. (Chai, [0029], “Similar to NeRF, a pinhole camera model may be used to perform ray casting. For each object, the points on the rays can be sampled at adaptive depths rather than fixed ones since the bounding box provides clues about where the object locates. Specifically, the cast rays R={r.sub.j|j ∈[1, S.sup.2]} in a resolution S are transformed into the canonical object coordinate system. Then, a Ray-AABB (axis-aligned bounding box) intersection algorithm may be applied to calculate the adaptive near and far depth (d.sub.j,l,n, d.sub.j,l,f) of the intersected segment between the ray r.sub.j and the l-th box B.sub.l. After that, N.sub.d points are sampled equidistantly in the interval [d.sub.j,l,n, d.sub.j,l,f]. An intersection matrix M is maintained of size N×S.sup.2, whose elements indicate if this ray intersects with the box. With M, valid points are selected to infer, which can greatly reduce the rendering cost. Different background sampling strategies are adopted depending on the dataset. In general, fixed depth sampling is performed for bounded backgrounds in indoor scenes and the inverse parametrization of NeRF++ is inherited for complex and unbounded outdoor scenes, which uniformly samples background points in an inverse depth range..”)
Regarding claim 4, Chai in view of Ryland teaches:
The method according to claim 1, characterized in that the physical plausibility for the intermediate representation is taken into account and maintained at least in that the insertion of the at least one selected object is parameterized with respect to a position of the object using the provided scene parameter, preferably in order to determine a spatial placement of the object in the template scene as a function of a driving situation and/or an environment of the template scenes. (Chai, [0029], “Similar to NeRF, a pinhole camera model may be used to perform ray casting. For each object, the points on the rays can be sampled at adaptive depths rather than fixed ones since the bounding box provides clues about where the object locates. Specifically, the cast rays R={r.sub.j|j ∈[1, S.sup.2]} in a resolution S are transformed into the canonical object coordinate system. Then, a Ray-AABB (axis-aligned bounding box) intersection algorithm may be applied to calculate the adaptive near and far depth (d.sub.j,l,n, d.sub.j,l,f) of the intersected segment between the ray r.sub.j and the l-th box B.sub.l. After that, N.sub.d points are sampled equidistantly in the interval [d.sub.j,l,n, d.sub.j,l,f]. An intersection matrix M is maintained of size N×S.sup.2, whose elements indicate if this ray intersects with the box. With M, valid points are selected to infer, which can greatly reduce the rendering cost. Different background sampling strategies are adopted depending on the dataset. In general, fixed depth sampling is performed for bounded backgrounds in indoor scenes and the inverse parametrization of NeRF++ is inherited for complex and unbounded outdoor scenes, which uniformly samples background points in an inverse depth range..”)
Regarding claim 5, Chai in view of Ryland teaches:
The method according to claim 1, characterized in that the object data comprise at least one dataset having the various objects, wherein the objects comprise at least tires and motorcycles, wherein the at least one dataset provides each of the objects in two-dimensional form as a 2D image and/or in three-dimensional form as a 3D model, wherein the 3D models differ from one another with respect to their parameterization of a dimension, and/or an angle, and/or a pose, and/or an orientation. (Chai FIG. 3, [0040], “generate an object for each 3D bounding box 132 at 310.” FIG. 1 shows that objects are cars, which have tires. FIG. 1 also shows different orientation.)
Regarding claim 6, Chai in view of Ryland teaches:
Chai teaches:
The method according to claim 1, characterized in that the template scenes comprise multiple artificial and/or real driving scenes for autonomous driving, wherein the synthetic scenes,(Chai, FIG. 1, an generated driving scene.) …such that information about the inserted object and preferably about a layout of the respective intermediate representation is taken into account, wherein training data are provided in order to train a machine learning model for autonomous driving based on the generated synthetic scenes. ([0041], “WAYMO® is a large-scale autonomous driving dataset with 1K video sequences of outdoor scenes.” FIG. 1 shows the objects are considered in the scene generation process. FIG. 2 shows the training process.)
On the other hand, Ryland teaches:
the synthetic scenes … which are generated in reference to the determined conditioning data,(Chai, FIG. 1, an generated driving scene. The synthetic image can be generated based on conditioning data as shown in Claim 1 by Ryland.) in particular by an influence on weighting parameters of a generative model, (Ryland, FIG. 6, the generative model has weighted parameters in the neural network.)
Chai teaches steps of generating a synthetic scene. Ryland also teaches steps of generating a synthetic scene. In Ryland’s method, the computer-generated intermediate image is further enhanced by considering the data from intermediate image, so as to generate a final real scene image.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Chai with the image condition aware image enhance step of Ryland to generate a more real scene image.
Regarding claim 7, Chai in view of Ryland teaches:
The method according to claim 1, characterized in that the determination of the conditioning data based on the generated intermediate representations comprises at least the following step: - extracting at least one scene parameter from the respective intermediate representation, preferably a Canny edge representation, which comprises edges from the template scene and the inserted object. (Ryland, [0013], “The generative NN can remove typical image artifacts associated with the fake objects and the real objects over real backgrounds. For example, the generative network adds and adjusts complex interactions with lighting, color matching with similar objects, decals and weathering, and edge clarity for the fake objects, among other more subtle tells of fakery.” The combination of claim 1 is incorporated here.)
Claims 9 and 10 recite similar limitations of claim 1, in a form of device and medium respectively, thus are rejected accordingly.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YANNA WU whose telephone number is (571)270-0725. The examiner can normally be reached Monday-Thursday 8:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YANNA WU/Primary Examiner, Art Unit 2615
Prosecution Insights