DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 1, 3, and 4 are objected to because of the following informalities: Claim 1 recites “identify an external object from the obtained real space image; determine an object of interest among the identified external object”, Claim 3 recites “determine, as the object of interest, an external object with an object type corresponding to the focus mode of the electronic device among the identified external object”, and Claim 4 recites “determine the object of interest among the identified external object based on the focus mode of the electronic device and an input of a user”. Since identifying an object of interest from a single identified external object is deemed superfluous in its direct interpretation, an assumption of the understanding of the Claim is that there are multiple identified external objects from which object(s) of interest are identified from. Appropriate correction is required to reflect the assumed interpretation.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3-5, 10, 13, 15-16, and 19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Assouline (US 20220327608 A1).
Regarding Claim 1, Assouline teaches an electronic device comprising:
A camera configured to capture a real space (Paragraph [0135]: “The machine 1100 may include … input/output (I/O) components 1138”; Paragraph [00138]: “In further examples, the I/O components 1138 may include biometric components 1128, motion components 1130, environmental components 1132, or position components 1134, among a wide array of other components”; Paragraph [0139]: “The environmental components 1132 include, for example, one or cameras”);
Communication circuitry (Paragraph [0143]: “The I/O components 1138 further include communication components 1136 operable to couple the machine 1100 to a network 1120 or devices 1122 via respective coupling or connections. For example, the communication components 1136 may include a network interface component or another suitable device to interface with the network 1120”);
At least one processor (Paragraph [0135]: “The machine 1100 may include processors 1102”); and
Memory for storing instructions that, when executed individually and/or collectively by the at least one processor cause the electric device (Paragraph [0135]: “The machine 1100 may include … memory 1104”; Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
Obtain a real space image using the camera (Paragraph [0097]: “The object detection module 512 receives a monocular image (or video) depicting a room in a home 501. This image can be received as part of a real-time video stream, a previously captured video stream or a new image captured by a camera of the client device 102”; Paragraph [0134]: “The machine 1100, for example, may comprise the client device 102 or any one of a number of server devices forming part of the messaging server system 108”);
Identify an external object from the obtained real space image (Paragraph [0097]: “The object detection module 512 applies one or more machine learning techniques to identify real-world physical objects that appear in the image depicting a room in a home 501”);
Determine an object of interest among the identified external object based on the configuration for virtual space (Paragraph [0114]: “Specifically, as shown in FIG. 6, the AR recommendation system 224 receives an image or video 600 that depicts a room in a home. The AR recommendation system 224 applies one or more machine learning techniques to detect and recognize one or more real-world objects that are depicted in the image or video 600. For example, the AR recommendation system 224 detects and recognizes a speaker 630 and a television 610 among other objects (e.g., a sofa, a rug, a flower pot, a coffee table, and so forth)”); and
Arrange a virtual object corresponding to the determined object of interest in the virtual space (Paragraph [0120]: “In some cases, the AR recommendation system 224 determines that an office chair (a piece of furniture) exists in the image or video 700. The AR recommendation system 224 obtains the 3D mesh representation or reconstruction of the room and determines that the existing office chair fails to satisfy one or more fit parameters for the room. For example, the AR recommendation system 224 determines that the existing office chair is too large or too small for the room. In such cases, the AR recommendation system 224 identifies an office chair (an alternate or different furniture item) that is available for purchase and that satisfies one or more fit parameters based on the 3D mesh representation or reconstruction of the room. Namely, the AR recommendation system 224 identifies an office chair furniture item that fits within the available physical space and dimensions of the real-world office table depicted in the image or video 700. The AR recommendation system 224 searches for and generates an AR representation 730 corresponding to the identified office chair furniture item that is available for purchase. The AR recommendation system 224 replaces the existing office chair in the image or video 700 with the AR representation 730 of the office chair furniture item. In some cases, the AR recommendation system 224 positions the AR representation 730 of the office chair in the same place as the existing office chair after the real-world office chair is removed or deleted from the image or video 700. This allows the user to see how a different office chair (piece of furniture) would look in the user's office room before purchasing the office chair”).
Regarding Claim 3, the electronic device of Claim 1 is rejected over Assouline.
Assouline teaches the device of Claim 1, wherein the configuration for the virtual space comprising a focus mode of the electronic device (Paragraph [0051]: “For example, the AR recommendation system 224 can classify the room as a kitchen, a bedroom, a nursery, a toddler room, a teenager room, an office, a living room, a den, a formal living room, a patio, a deck, a balcony, a bathroom, or any other suitable home-based room classification. Once classified, the AR recommendation system 224 identifies one or more items (such as physical products or electronically consumable content items) related to the room classification”. Notes: the focus mode, in its broadest reasonable interpretation in light of the specification, identifies and suggests objects of interest based on the context of the real/virtual space (ex. living room, kitchen)), and
Wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
Determine, as the object of interest, an external object with an object type corresponding to the focus mode of the electronic device among the identified external object (Paragraph [0051]: “Once classified, the AR recommendation system 224 identifies one or more items (such as physical products or electronically consumable content items) related to the room classification”).
Regarding Claim 4, the electronic device of Claim 1 is rejected over Assouline.
Assouline teaches the electronic device of Claim 1, wherein instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
Determine the object of interest among the identified external object based on the focus mode of the electronic device and an input of a user (Paragraph [0051]: “The AR recommendation system 224 retrieves AR representations of the identified items and incorporates (displays at specified positions) the AR representations within the image or video. The AR representations can be interactive, such that upon receiving a user selection or input that selects the particular AR representation, an electronic commerce (e-commerce) purchase transaction is performed to obtain access to or receive the corresponding item”. Notes: the user is able to select an object of interest).
Regarding Claim 5, the electronic device of Claim 1 is rejected over Assouline.
Assouline teaches the electronic device of Claim 1, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
Load stored three-dimensional (3D) data for the same object type as the determined object of interest from a database (Paragraph [0106]: “The AR item selection module 519 can then search for and retrieve an AR representation of the bed furniture item from a list of AR representations of beds and can instruct the image modification module 518 to incorporate the AR representation of the bed furniture item into the image captured by the client device 102”; Paragraph [0111]: “The expected object module 516 can also process the 3D mesh representation of the room to compute an amount of available physical space remaining in the room depicted in the image. In response, the expected object module 516 can determine that only the coffee table furniture item can physically fit within the dimensions of the room and that the room cannot physically fit all three missing furniture items”. Notes: The stored AR representation of furniture items, which are stored as lists of categories such as beds, have their dimensions checked before placing it in the AR space, and are hence composed of three-dimensional (3D) data), and
Generate the virtual object of interest based on the loaded 3D data (Paragraph [0051]: “The AR recommendation system 224 retrieves AR representations of the identified items and incorporates (displays at specified positions) the AR representations within the image or video”).
Regarding Claim 10, the electronic device of Claim 1 is rejected over Assouline.
Assouline teaches the electronic device of Claim 1, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
Determine whether an asset of the determined object of interest is changed in a real space image of a next of a next frame (Paragraph [0115]: “In some cases, the AR recommendation system 224 displays the video item corresponding to the AR element or representation 620 on the television 610 in response to receiving the user selection of the AR element or representation 620”; Paragraph [0067]: “In some examples, when a particular modification is selected along with content to be transformed, elements to be transformed are identified by the computing device, and then detected and tracked if they are present in the frames of the video”. Notes: An asset, in its broadest reasonable interpretation, is something that defines a virtual object (ex. color, material, or even the object itself). The device determines whether an asset of the object of interest is changed inherently when it displays the new AR representation for a virtual object);
Initialize, when the asset of the determined object of interest is changed, the asset of the determined object of interest (Paragraph [0120]: “Namely, the AR recommendation system 224 identifies an office chair furniture item that fits within the available physical space and dimensions of the real-world office table depicted in the image or video 700. The AR recommendation system 224 searches for and generates an AR representation 730 corresponding to the identified office chair furniture item that is available for purchase. The AR recommendation system 224 replaces the existing office chair in the image or video 700 with the AR representation 730 of the office chair furniture item. In some cases, the AR recommendation system 224 positions the AR representation 730 of the office chair in the same place as the existing office chair after the real-world office chair is removed or deleted from the image or video 700”. Notes: An asset, in its broadest reasonable interpretation, is something that defines a virtual object (ex. color, material). An asset is inherently initialized if the asset is changed for display); and
Determine, when the asset of the determined object of interest is not changed, whether a physical change has occurred on the determined object of interest to give a motion effect to the virtual object corresponding to the determined object of interest (Paragraph [0065]: “tracking of such objects as they leave, enter, and move around the field of view in video frames, and the modification or transformation of such objects as they are tracked. Some examples may involve generating a three-dimensional mesh model of the object or objects, and using transformations and animated textures of the model within the video to achieve the transformation.”. Notes: An asset, in its broadest reasonable interpretation, is something that defines a virtual object (ex. color, material). A motion effect, in its broadest reasonable interpretation, is any effect pertaining to motion, which includes any modification of objects that move in the video frames, which includes virtual objects corresponding to objects of interest (three-dimensional mesh model of the object or objects).
Claim 13, being similar in scope to Claim 1, is rejected under the same rationale.
Claim 15, being similar in scope to Claims 3 and 4, are rejected under the same rationale.
Claim 16, being similar in scope to Claim 5, is rejected under the same rationale.
Claim 19, being similar in scope to Claim 10, is rejected under the same rationale, where “regenerate” reads on “initialize”, and “resource” reads on “asset”.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 8-9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Assouline.
Regarding Claim 8, the electronic device of Claim 1 is rejected over Assouline.
Assouline teaches the electronic device of Claim 1, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
Arrange a plurality of virtual objects corresponding to a plurality of objects of interest in the virtual space (Paragraph [0109]: “The expected object module 516 provides the identifier of the sofa and coffee table furniture items to the AR item selection module 519. The AR item selection module 519 can then search for and retrieve AR representations of the sofa and coffee table furniture items from a list of AR representations and can instruct the image modification module 518 to incorporate the AR representations of the sofa and coffee table furniture items into the image captured by the client device 102”).
Assouline does not explicitly teach arranging the plurality of virtual objects corresponding to a plurality of objects of interest so as not to overlap each other in the virtual space.
However, Assouline implicitly teaches checking the dimensions of virtual objects when determining whether to place it in the virtual space such that they do not overlap (Paragraph [0111]: “The expected object module 516 can also process the 3D mesh representation of the room to compute an amount of available physical space remaining in the room depicted in the image. In response, the expected object module 516 can determine that only the coffee table furniture item can physically fit within the dimensions of the room and that the room cannot physically fit all three missing furniture items. Namely, the room can only fit the coffee table furniture item and not the sofa and the rug furniture items”. Notes: Overlap, in its broadest reasonable interpretation with regards to virtual objects in a virtual space, would occur when two virtual objects that would not physically fit in the real space together are rendered in the virtual space. Therefore, because Assouline teaches checking whether the virtual objects of objects of interest fit in a virtual space, it inherently arranges a plurality of virtual objects so as not to overlap with one another in the virtual space).
It would be obvious to a person having ordinary skill in the art that while Assouline does not explicitly teach whether virtual objects overlap, it does implicitly teach it by checking the dimensions of virtual objects that could be added to the virtual space, and seeing if the dimensions of the virtual space would allow the arrangement of the virtual objects. Furthermore, while Assouline does not teach that the virtual objects having their dimensions checked correspond with objects of interest in a real space, the virtual objects of Assouline may correspond to objects of interest in a real space; the process of checking the dimensions of the virtual objects does not depend on whether the virtual objects do or do not correspond with a object of interest in a real space. As a result, Assouline implicitly teaches arranging virtual objects that correspond with objects of interest such that they do not overlap in a virtual space; the motivation for doing so is well established in the art, as interior design involving furniture often involves virtually decorating environments in accordance with the limitations of the real space.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention that the device of Assouline effectively arranges a plurality of virtual objects corresponding to a plurality of objects of interest so as not to overlap with one another in a virtual space.
Claim 18, being similar in scope to Claim 8, is rejected under the same rationale.
Regarding Claim 9, the electronic device of Claim 1 is rejected over Assouline.
Assouline teaches the electronic device of Claim 1, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
Arrange an individual virtual object close to a position in the virtual space corresponding to a position of an object of interest corresponding to the individual virtual object in the real space (Paragraph [0120]: “In some cases, the AR recommendation system 224 determines that an office chair (a piece of furniture) exists in the image or video 700. The AR recommendation system 224 obtains the 3D mesh representation or reconstruction of the room and determines that the existing office chair fails to satisfy one or more fit parameters for the room. For example, the AR recommendation system 224 determines that the existing office chair is too large or too small for the room. In such cases, the AR recommendation system 224 identifies an office chair (an alternate or different furniture item) that is available for purchase and that satisfies one or more fit parameters based on the 3D mesh representation or reconstruction of the room. Namely, the AR recommendation system 224 identifies an office chair furniture item that fits within the available physical space and dimensions of the real-world office table depicted in the image or video 700. The AR recommendation system 224 searches for and generates an AR representation 730 corresponding to the identified office chair furniture item that is available for purchase. The AR recommendation system 224 replaces the existing office chair in the image or video 700 with the AR representation 730 of the office chair furniture item. In some cases, the AR recommendation system 224 positions the AR representation 730 of the office chair in the same place as the existing office chair after the real-world office chair is removed or deleted from the image or video 700. This allows the user to see how a different office chair (piece of furniture) would look in the user's office room before purchasing the office chair”), and
Determining a priority of objects of interest corresponding to a plurality of virtual objects corresponding to a plurality of objects of interest (Paragraph [0109]: “In some implementations, the expected object module 516 can identify multiple objects that are in the expected list of objects for the room classification and which are missing from the list of objects provided by the object detection module 512 or room classification module 514. In such circumstances, the expected object module 516 can assign a rank or score to each of the missing objects and can select a set of two or three expected objects associated with higher scores or ranks than other non-selected expected objects that are missing. The score can be assigned based on an importance level of the expected object that is stored in association with each expected object in the list stored in the data structures 300. For example, the expected object module 516 can determine that the room classification is a living room and that a sofa, coffee table, and rug furniture items are missing from the list of objects provided by the object detection module 512 or room classification module 514. In such cases, the expected object module 516 can determine that the sofa and coffee table furniture items have higher scores than the rug furniture item and can in response select only the sofa and coffee table as furniture items to recommend to a user to purchase for inclusion in the room depicted in the image captured by the client device 102 and can exclude the rug furniture item”. Notes: the broadest reasonable interpretation of priority in context with the claim language and specification is a priority of generation of a virtual object with regards to the context of the room (ex. a bed has a high priority for a bedroom). The ranking of Assouline establishes a priority of a list of items related to a particular environment)
Assouline does not teach using the order of priority of objects of interest corresponding to a plurality of virtual objects corresponding to a plurality of objects of interest for arranging individual virtual objects.
However, Assouline does teach using the order of priority of objects of interest corresponding to a plurality of virtual objects corresponding to a plurality of objects of interest for arranging virtual objects that are anticipated in the space according to classification, but are not present in the real space. It would be obvious to a person having ordinary skill in the art that in a case where the virtual objects that correspond to objects of interest may overlap, more important objects may be prioritized for display as a virtual object; the motivation is provided in Assouline (Paragraph [0109]: “For example, the expected object module 516 can determine that the room classification is a living room and that a sofa, coffee table, and rug furniture items are missing from the list of objects provided by the object detection module 512 or room classification module 514. In such cases, the expected object module 516 can determine that the sofa and coffee table furniture items have higher scores than the rug furniture item and can in response select only the sofa and coffee table as furniture items to recommend to a user to purchase for inclusion in the room depicted in the image captured by the client device 102 and can exclude the rug furniture item”. Assouline teaches a priority for objects of interest, arranging virtual objects corresponding to objects of interest in a real space, as well as the motivation for using the priority of objects of interest for generating virtual objects.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to apply the priority of Assouline for the purpose of arranging virtual objects corresponding to objects of interest in a real space; doing so would yield the predictable result of generating relevant virtual objects that correspond to an object of interest in a real space based on the context of the space.
Claims 2 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Assouline, in view of Petrangeli (US 20210304706 A1) and Github (Small object detection – what is the minimum bounding box size, 2021).
Regarding Claim 2, the electronic device of Claim 1 is rejected over Assouline.
Assouline teaches the electronic device of Claim 1, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
identify external objects (Paragraph [0097]: “The object detection module 512 applies one or more machine learning techniques to identify real-world physical objects that appear in the image depicting a room in a home 501”).
Assouline does not teach reducing a minimum or small size of an external object as a reality level set in the electronic device increases.
However, Petrangeli teaches a reality level that can be set to increasing levels (Paragraph [0019]: “Each identified augmented reality object can be associated with a plurality of versions, each version corresponding to a plurality of augmented reality levels of detail. For example, an augmented reality object may have three corresponding versions—one at a low level of detail, one at a medium level of detail, and one at a high level of detail. Each level of detail may be associated with different degrees of realism such that the version of the augmented reality element at the high level of detail is more photo-realistic than the version at the low level of detail”. Notes: The broadest reasonable interpretation of a reality level with regards to a scene involves adjustments to any attributes of an image/scene that can affect how real the scene looks).
Assouline and Petrangeli are considered analogous in the art with respect to displaying virtual entities in augmented reality. A common motivation in the art is to implement an ability to adjust how real the scene is by adjusting how much augmented content is displayed in conjunction with the real video; this is demonstrated by Petrangeli. A person having ordinary skill in the art would appreciate that increasing a “reality level”, as established by Assouline as modified, encompasses any changes to the virtual space depicting the real space that result in a more realistic virtual space; this includes how accurately a virtual space reflects a real space in terms of identified external objects that are represented as virtual objects in the virtual space.
Therefore, it would have been obvious to a person having ordinary skill in the art to combine the electronic device capable of displaying virtual objects of Assouline with the reality level feature of Petrangeli; Doing so would yield the predictable result of an electronic device capable of adjusting how real a virtual scene looks with regards to how many identified external objects have a corresponding virtual object.
Additionally, Github teaches reducing the size of external objects identifiable in a real space (post by ultralytics: “What is the smallest bounding box size for Yolo5 training? I'll need to detect medium and small objects (think cars at distance or ships at distance) that can be just a couple of pixels in size. It'll be just one class of objects that I'll need to detect”. Notes: Yolo5 is a well-known object detection model, with user adjustable bounding box allowing the detection of objects of a specific size, where reducing the dimensions of the bounding box results in smaller objects being detected).
Assouline as modified and Github are considered analogous in the art with respect to object detection. A common motivation in the art is to identify objects of a certain size depending on the task; this is evident through Github (ultralytics: “I'll need to detect medium and small objects (think cars at distance or ships at distance) that can be just a couple of pixels in size”).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the electronic device capable of identifying external objects of a real space and adjusting the reality level of the related virtual space with regards to identified external objects and their corresponding virtual objects of Assouline as modified with the ability to reduce a minimum or small size of an external object identifiable in a real space of Github; Doing so would yield the predictable result of an electronic device capable of identifying more external objects in a real scene and generating more corresponding virtual objects, leading to a more realistic virtual space with regards to its associated real space.
Claim 14, being similar in scope to Claim 2, is rejected under the same rationale.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Assouline as modified, in view of Samyam (Play or Stream a Video with 3D audio in Unity, Youtube, 2022).
Regarding Claim 6, the electronic device of Claim 1 is rejected over Assouline as modified.
Assouline as modified teaches the electronic device of Claim 1, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Assouline, Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Assouline, Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
establish communication with the determined object of interest (Paragraph [0117]: “As shown in FIG. 7, the AR recommendation system 224 receives an image or video 700 that depicts a room in a home. The AR recommendation system 224 applies one or more machine learning techniques to detect and recognize one or more real-world objects that are depicted in the image or video 700. For example, the AR recommendation system 224 detects and recognizes a desk and a computer monitor 710 among other objects”; Paragraph [0143]: “The I/O components 1138 further include communication components 1136 operable to couple the machine 1100 to a network 1120 or devices 1122 via respective coupling or connections. For example, the communication components 1136 may include a network interface component or another suitable device to interface with the network 1120”. Notes: Assouline establishes that an object of interest can be identified as a computer monitor among other objects, which extends to a computer in its broadest reasonable interpretation. Establishing connection between an electronic device capable of communication via physical or non-physical connection is an inherent capability to possessing communication capabilities, and encompasses basic file sharing in its broadest reasonable interpretation).
Assouline as modified does not teach displaying the same screen as a screen on a display of the determined object of interest in an area of the virtual object corresponding to the display of the determined object of interest.
However, Samyam teaches displaying a video or image obtained from a different source on a virtual screen resembling an object of interest in a real space (5:30-5:50: Demonstrates displaying a video file on a virtual object corresponding to an object of interest with a screen; 3:11: Demonstrates that a video file is an input resulting in displaying the video file on the screen of the virtual object. Notes: while Samyam utilizes a video file, displaying static content such as an image is also possible. Videos and images originating from an object of interest corresponding to a device capable of displaying a video or image can be used as input as demonstrated at 3:11).
Assouline as modified and Samyam are considered analogous in the art with respect to working with virtual spaces. Replicating screens between devices is well known in the art. A common motivation in the art for replicating a screen on an object of interest in a real space to a virtual object screen is allowing the viewing of videos or images within a virtual environment for immersion purposes, such as in games or in simulations.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the electronic device capable of establishing communication with an object of interest with a screen for display of Assouline as modified with the ability to replicate the screen of an object of interest to a virtual object with a screen of Samyam; Doing so would yield the predictable result of an electronic device capable of identifying an object of interest with a screen in a real space, and displaying a virtual object corresponding to the object of interest in a virtual space with the same display of the object of interest.
Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Assouline as modified, in view of Gomez (Selective Style Transfer for Text, 2019).
Regarding Claim 7, the electronic device of Claim 1 is rejected over Assouline as modified.
Assouline as modified teaches the electronic device of Claim 1, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Assouline, Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Assouline, Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
display a virtual object corresponding to a determined object of interest.
Assouline as modified does not teach recognizing a character on a surface of the determined object of interest, and displaying the character in an area of the virtual object corresponding to the surface of the determined object of interest.
However, Gomez (Selective Style Transfer for Text, 2019) teaches recognizing a character on a surface of a determined object of interest, and displaying the character in an area of the virtual object corresponding to the surface of the determined object of interest (Figure 3 and Figure 5 demonstrate how text on an object of interest (external object in a real space) is kept the same, while the virtual object (modified object in a modified image depicting the real space) is visually different from the object of interest).
Assouline and Gomez are considered analogous in the art with regards to depicting alternate representations of an object of interest. A common motivation in the art is to modify the appearance of objects. Objects may have characters on them, and preserving the text while changing the appearance of the object is an established motivation, as demonstrated by Gomez. While Gomez does not explicitly teach displaying characters on a virtual object, virtual objects by definition possess textures. One ordinarily skilled in the art would appreciate that textures of a virtual object include images, and hence displaying images modified by the method of Gomez for the purpose of changing the appearance but keeping the text of an object as a texture would be similarly obvious.
Therefore, it would be obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the electronic device capable of displaying virtual objects corresponding to objects of interest of Assouline with the method of changing the appearance of an object while maintaining characters on its surface of Gomez; Doing so would yield the predictable result of an electronic device capable of displaying virtual objects that have characters on their surfaces that are also visible on the objects of interest, leading to a virtual object that more accurately represented object of interest.
Claim 17, being similar in scope to Claim 7, is rejected under the same rationale.
Claims 11-12 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Assouline as modified, in view of Joshi (US 20220084301 A1).
Regarding Claim 11, the electronic device of Claim 1 is rejected over Assouline.
Assouline teaches the electronic device of Claim 1, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Assouline, Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Assouline, Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
Enter a pet mode, and arrange a virtual object corresponding to a pet identified as an external object in the virtual space (Assouline, Paragraph [0066]: “Additionally, any objects can be processed using a computer animation model, such as a human's face and parts of a human body, animals, or non-living things such as chairs, cars, or other objects”; Assouline, Paragraph [0120]: “Namely, the AR recommendation system 224 identifies an office chair furniture item that fits within the available physical space and dimensions of the real-world office table depicted in the image or video 700. The AR recommendation system 224 searches for and generates an AR representation 730 corresponding to the identified office chair furniture item that is available for purchase. The AR recommendation system 224 replaces the existing office chair in the image or video 700 with the AR representation 730 of the office chair furniture item. In some cases, the AR recommendation system 224 positions the AR representation 730 of the office chair in the same place as the existing office chair after the real-world office chair is removed or deleted from the image or video 700”. Notes: animals can be processed as objects by the device of Assouline as modified, and hence are regarded as eligible for being represented by a virtual object. The broadest reasonable interpretation of a pet mode is a mode in which pets can be identified, which the presented mode of operation of Assouline as modified is capable of).
Assouline as modified does not teach comparing a size of the real space and a size of the virtual space.
However, Joshi teaches comparing a size of the real space and a size of the virtual space (Paragraph [0157]: “However, the main difficulty of using natural walking as a locomotion technique in VE is the requirement that the size of the PTS be comparable in size with the VE, which is often not the case; especially for simulations involving large-scale environments. Today this is still an active area of research with a particular focus on locomotion techniques which do not carry, in any degree, the spatial constraints imposed by the physical space over to the theoretically boundless virtual space of the VE”. Notes: VE is virtual environment (virtual space), and PTS is physical tracked space (real space)).
Assouline as modified and Joshi are considered analogous in the art with respect to virtual spaces corresponding to a real space. Joshi establishes that comparing the size of a virtual space and its corresponding real space is established in the art.
Furthermore, it would have been obvious that if an object isn’t in the real space corresponding with the virtual space, it won’t be rendered or displayed as a virtual object. In the reverse, it is also obvious that anything in a real space that is observable in the virtual space would be rendered or displayed in the virtual space; this follows from the fact that if an object exists in the real space where the real space is defined as being smaller than its virtual space counterpart, it must also exist in the virtual space, by virtue of the real space being encompassed by the virtual space by size. Lastly, whether a virtual object is displayed in the virtual space is an inherent result of whether the virtual space is bigger than the real space or the reverse.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the electronic device capable of displaying a virtual object representative of a pet corresponding to an identified external object of Assouline as modified with the established practice of comparing the size of the virtual space with the real space of Joshi and the inherent result of whether a virtual object corresponding to an object of interest in the real space is displayed in the virtual space; Doing so would yield the predictable result of arranging or not arranging a virtual object within the virtual space, and more accurately depict the virtual space in relation to the real space as defined.
Claim 20, being similar in scope to Claim 11, is rejected under the same rationale.
Regarding Claim 12, the electronic device of Claim 11 is rejected over Assouline as modified.
Assouline as modified teaches the electronic device of Claim 11, wherein the instructions, when executed individually and/or collectively by the at least one processor, cause the electronic device (Assouline, Paragraph [0135]: “The machine 1100 may include processors 1102, memory 1104”; Assouline, Paragraph [0136]: “The main memory 1104, the static memory 1114, and the storage unit 1116 store the instructions 1108 embodying any one or more of the methodologies or functions described herein”) to:
arrange, when the size of the real space is larger than the size of the virtual space, the virtual object corresponding to the pet in the virtual space only when the pet is positioned in an area of the real space corresponding to the virtual space (it is obvious in the art that if an object isn’t in the real space corresponding with the virtual space, it won’t be rendered or displayed as a virtual object); and
arrange, when the size of the real space is smaller than the size of the virtual space, the virtual object corresponding to the pet in the virtual space regardless of the position of the pet (it is also obvious that anything in a real space that is observable in the virtual space would be rendered or displayed in the virtual space; this follows from the fact that if an object exists in the real space where the real space is defined as being smaller than its virtual space counterpart, it must also exist in the virtual space, by virtue of the real space being encompassed by the virtual space by size).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAYMOND CHUN LAM LI whose telephone number is (571)272-5124. The examiner can normally be reached M-F 8:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAYMOND CHUN LAM LI/ Examiner, Art Unit 2614
/KENT W CHANG/ Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights