Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 10 and 19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
As per claims 10 and 19, it is unclear to the limitations, display, based on a real IoT device being added to be displayed within the image due to a change in a viewpoint of the camera, a control item corresponding to the added real IoT device; and remove, based on a real IoT device displayed within the image being excluded therefrom due to a change in a viewpoint of the camera, a control item corresponding to the excluded real IoT device.”
It is unclear because the limitations “a real IoT device displayed within the image being excluded therefrom due to a change in a viewpoint of the camera, a control item corresponding to the excluded real IoT device” contrasts the previous limitations adding a real IOT device based upon a change in a viewpoint of the camera. One part of the limitation added a real IOT device based on a viewpoint of a camera while the second part of the limitations recite excluding the real IOT device based on a viewpoint of a camera. These limitations contrast with each other. Examiner cannot determine the meets and bounds of the recited claim limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 8-9, 12-15, 17-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Minakuchi et al. – hereinafter Minakuchi (US 2021/0212162) in view of Chojnacka et al. – hereinafter Chojnacka (US 2020/0342668) / Mittleman et al. – hereinafter Mittleman (US 2021/0074073)
As per claim 1, Minakchi disclose an electronic device comprising: a display; a camera; a memory configured to store instructions; and at least one processor, comprising processing circuitry, operatively connected to the display, the camera, and the memory, ([0055]; or example, video data may be generated when the data generator 31 is a monitoring camera. Weather data may be generated when the data generator 31 is a weather sensor. The communication unit 32 provides an interface with a network. In this example, the communication unit 32 transmits IoT data generated by the data generator 31 to a virtual server 10.)
wherein at least one processor, individually and/or collectively, is configured to execute the instructions and to cause the electronic device to: activate the camera and display an image acquired in real time through the camera on the display; analyze the image to determine at least one space within the image; (0103]; The user device 2 y creates and makes available to the public empty/full information indicating whether there is an “empty space” in the parking lot 40. The empty/full information can be created by analyzing video data output from the IoT device 1)
Minakuchi fails to disclose identify at least one Internet of Things (IoT) device capable of being disposed in the at least one space; and display at least one virtual IoT device corresponding to the at least one identified IoT device on the at least one determined space of the image. Chojnacka discloses identify at least one Internet of Things (IoT) device capable of being disposed in the at least one space; and display at least one virtual IoT device corresponding to the at least one identified IoT device on the at least one determined space of the image. ([0020]; empty spacing , for example , spacing available for placement of virtual objects , may be taken into account by transmitting beams through the detected real object ( s ) in a point cloud , to detect whether or not a space is occupied by a real object . In particular, the
detection of unoccupied space may be used to determine spacing available for the selection and suggestion / placement of contextual virtual object(s), based on the semantic understanding of the physical real world environment.)
The combined teachings of Minakuchi / Chojnacka fail to disclose that the virtual object inserted into the mixed reality environment is an IOT virtual device.
Mittleman discloses that the virtual object inserted into the mixed reality environment is an IOT virtual device. ([0006]; computing device may include a display and a camera. The operations may also include rendering a view of a virtual object that represents a field - of - view of the smart home device , where the view of the virtual object may be rendered based on a position corresponding to a position of the mobile computing device)
It would have been obvious before the earliest effective filing date of the invention for the combined teachings of Minakuchi / Chojnacka to be further modified so that the IOT smart home device is inserted in the empty space. This would have been beneficial to optimize placement of the IOT devices in the room to effectively capture events in a surrounding area.
As per claim 2, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 1. wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: identify at least one real object included in the image through analysis of the image; and identify an IoT device that is connectable with the identified real object. ([0020]; Empty spacing, for example, spacing available for placement of virtual objects, may be taken into account by transmitting beams through the detected real object(s) in a point cloud, to detect whether or not a space is occupied by a real object)
As per claim 3, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 1. Choinacka discloses wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: display a virtual operation state corresponding to an operation state of the IoT device during operation within the image. ([0042]; A virtual object representing a first entree 560V has been placed in the camera view 500H of the physical environment 100B shown in FIG. 5H. In this example, the first entree 560V has been placed in a space on the table 510 which was determined to be occupied by the first plate 520. However, based on the semantic understanding of the physical environment 100B, and the real objects occupying the physical environment 100B, the system may determine that the surface defined by the plate 520 is capable of receiving suggested items, and in particular, food items. FIGS. 5I and 5J illustrate camera views 500I and 500J, respectively, of the physical environment 100B, in which different virtual objects, representing different entrees 565V and 570V, are presented for consideration by the user.; [0043] In some implementations, the system may take into account other factors such as, for example, geographic location, time of day, user preferences and the like, in determining appropriate virtual objects for placement in the scene of the virtual environment. For example, in placing virtual objects representing the cup of coffee 550V, the beverage glass 555V and the pancakes 560V in the camera view 500H (and/or the bagel 565V and the waffle 570V in the camera views 500I and 500J, respectively), the system may take into consideration the time of day, indicating it is time for breakfast.)
As per claim 4, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 1. Chojnacka discloses wherein the at least one processor, individually and/or collectively, is configured to cause the electronic device to: display at least one control item corresponding to the virtual IoT device; and display the virtual operation state within the image, based on an input to the control item. ([0042]; A virtual object representing a first entree 560V has been placed in the camera view 500H of the physical environment 100B shown in FIG. 5H. In this example, the first entree 560V has been placed in a space on the table 510 which was determined to be occupied by the first plate 520. However, based on the semantic understanding of the physical environment 100B, and the real objects occupying the physical environment 100B, the system may determine that the surface defined by the plate 520 is capable of receiving suggested items, and in particular, food items. FIGS. 5I and 5J illustrate camera views 500I and 500J, respectively, of the physical environment 100B, in which different virtual objects, representing different entrees 565V and 570V, are presented for consideration by the user.; [0043] In some implementations, the system may take into account other factors such as, for example, geographic location, time of day, user preferences and the like, in determining appropriate virtual objects for placement in the scene of the virtual environment. For example, in placing virtual objects representing the cup of coffee 550V, the beverage glass 555V and the pancakes 560V in the camera view 500H (and/or the bagel 565V and the waffle 570V in the camera views 500I and 500J, respectively), the system may take into consideration the time of day, indicating it is time for breakfast.)
As per claim 5, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 4. Chojnacka discloses wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: display, based on displaying a plurality of virtual IoT devices, a plurality of control items corresponding to the plurality of virtual IoT devices, respectively. ([0042; A virtual object representing a first entree 560V has been placed in the camera view 500H of the physical environment 100B shown in FIG. 5H. In this example, the first entree 560V has been placed in a space on the table 510 which was determined to be occupied by the first plate 520. However, based on the semantic understanding of the physical environment 100B, and the real objects occupying the physical environment 100B, the system may determine that the surface defined by the plate 520 is capable of receiving suggested items, and in particular, food items. FIGS. 5I and 5J illustrate camera views 500I and 500J, respectively, of the physical environment 100B, in which different virtual objects, representing different entrees 565V and 570V, are presented for consideration by the user; [0043] In some implementations, the system may take into account other factors such as, for example, geographic location, time of day, user preferences and the like, in determining appropriate virtual objects for placement in the scene of the virtual environment. For example, in placing virtual objects representing the cup of coffee 550V, the beverage glass 555V and the pancakes 560V in the camera view 500H (and/or the bagel 565V and the waffle 570V in the camera views 500I and 500J, respectively), the system may take into consideration the time of day, indicating it is time for breakfast.)
As per claim 6, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 4. Choinacka discloses wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: display the image acquired through the camera on a first area of the display; and display the at least one control item on a second area of the display. ([0042; A virtual object representing a first entree 560V has been placed in the camera view 500H of the physical environment 100B shown in FIG. 5H. In this example, the first entree 560V has been placed in a space on the table 510 which was determined to be occupied by the first plate 520. However, based on the semantic understanding of the physical environment 100B, and the real objects occupying the physical environment 100B, the system may determine that the surface defined by the plate 520 is capable of receiving suggested items, and in particular, food items. FIGS. 5I and 5J illustrate camera views 500I and 500J, respectively, of the physical environment 100B, in which different virtual objects, representing different entrees 565V and 570V, are presented for consideration by the user; [0043] In some implementations, the system may take into account other factors such as, for example, geographic location, time of day, user preferences and the like, in determining appropriate virtual objects for placement in the scene of the virtual environment. For example, in placing virtual objects representing the cup of coffee 550V, the beverage glass 555V and the pancakes 560V in the camera view 500H (and/or the bagel 565V and the waffle 570V in the camera views 500I and 500J, respectively), the system may take into consideration the time of day, indicating it is time for breakfast.)
As per claim 8, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 1. Choinacka discloses further comprising a communication module comprising communication circuitry configured to support short-range wireless communication, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: recognize at least one adjacent real IoT device using the communication module; and identify the recognized real IoT device from the image. ([0047]; Physical objects detected, for example, through a scan of the physical environment (block 720) may be analyzed for recognition/identification (block 730). Based on the physical objects identified in the physical environment, a semantic understanding of the physical environment may be developed (block 740). Based on the semantic understanding developed in the analysis of the physical objects detected in the physical environment, appropriate contextual objects may be selected for the physical environment, and virtual representations of the suggested objects may be placed in the AR/MR scene of the physical environment (block 750). The process may continue until the augmented reality/mixed reality experience has been terminated (block 760).; [0060]; In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module 2070 may provide additional navigation- and location-related wireless data to device 2050, which may be used as appropriate by applications running on device 2050.)
As per claim 9, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 1. Choinacka discloses wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: display at least one control item corresponding to the real IoT device; and transmit a control signal instructing operation of the real IoT device, based on an input to the control item. ([0025]; As the images of the physical environment 100A are streamed, and the image frames are fed into the recognition algorithm or model, 3D pose and location information related to real objects in the physical environment 100A may be detected. Images of the real objects may be correlated against images stored in an image database, to identify the detected real objects. Identification of the real objects in this manner may provide for development of a semantic understanding of the physical, real world environment.; [0057]; The control interface 2058 may receive commands from a user and convert them for submission to the processor 2052. In addition, an external interface 2062 may be provide in communication with processor 2052, so as to enable near area communication of device 2050 with other devices. External interface 2062 may provide, for example, for wired communication in some implementations, or for wireless comm7unication in other implementations, and multiple interfaces may also be used) As per claims 12 and 20, please see the discussion under claim 1 as similar logic applies.
As per claim 13, please see the discussion under claim 3 as similar logic applies.
As per claim 14, please see the discussion under claim 4 as similar logic applies.
As per claim 15, please see the discussion under claim 5 as similar logic applies.
As per claim 17, please see the discussion under claim 8 as similar logic applies.
As per claim 18, please see the discussion under claim 9 as similar logic applies.
Claims 7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Minakuchi (US 2021/0212162) / Chojnacka (US 2020/0342668) / Mittleman (US 2021/0074073) further in view of Chen et al. – hereinafter Chen (US 2016/0065653)
As per claim 7, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 1. The combined teachings of Minakucki / Choinacka / Mittleman fail to disclose wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: display an affordance item corresponding to the virtual IoT device; and provide a purchase screen for purchasing an IoT device corresponding to the virtual IoT device, based on an input to the affordance item.
Chen discloses wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: display an affordance item corresponding to the virtual IoT device; and provide a purchase screen for purchasing an IoT device corresponding to the virtual IoT device, based on an input to the affordance item. ([0098] The marketplace may provide one or more lists of IOT devices and/or virtual appliances (e.g., gateway modules and cloud services) and may enable purchase of the physical IOT devices, solution templates, the virtual appliances, or some combination thereof. Additionally or alternatively, the marketplace may interface with or provide links to one or more commercial sites on which a user may purchase the IOT devices and/or the virtual appliances.;[0099] In some embodiments, the marketplace module 210 may provide the lists of the IOT devices, solution templates, and virtual appliances prior to the configuration module 206 configuring the network connections. In some embodiments, the marketplace module 210 may provide the lists from which the IOT device selection 230 and/or the template selection 232 may be made. For example, the IOT device selection 230 may include a selection of one or more IOT devices 106 included in the lists. In these and other embodiments, the network connections may accordingly be configured to include the specific IOT devices 106 selected from a list provided by the marketplace module 210.)
It would have been obvious before the earliest effective filing date of the invention for the combined teachings of Minakuchi / Choinacka / Mittleman to be further modified so that the purchase of IOT device based upon the selection of the IOT device of the commercial web sites. This would have been yielded predictable results of enabling IOT devices to communicate with each other.
As per claim 16, please see the discussion under claim 7 as similar logic applies.
Claims 10-11 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Minakuchi (US 2021/0212162) / Chojnacka (US 2020/0342668) / Mittleman (US 2021/0074073) further in view of Byl et al. – hereinafter Byl (US 2018/0350145)
As per claim 10, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 8. The combined teachings of Minakuchi / Choinacka / Mittleman fail to disclose wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: display, based on a real IoT device being added to be displayed within the image due to a change in a viewpoint of the camera, a control item corresponding to the added real IoT device; and remove, based on a real IoT device displayed within the image being excluded therefrom due to a change in a viewpoint of the camera, a control item corresponding to the excluded real IoT device.( [0036] Referring now to FIG. 3, a flowchart of a method for rendering virtual objects within real physical space and interacting with these virtual objects is illustrated, in accordance with an embodiment. The real physical space, for example, may include, but is not limited to an open space for installation of permanent or temporary structures (for example, a home, an office, a building, an exhibition, a ceremony, a conference, etc.), a vehicle (for example, a car, a plane, a recreational vehicle, etc.), or a bare shell in a building.; [0050] Thus, to refine overlaying of the virtual object 602, user 104 may interact with the virtual object, via augmented reality device 106, at step 308, to exactly match the predefined point of the virtual object 602 with the coordinates of the anchoring point and exactly align the predefined facet of the virtual object with the anchoring vector. User 104's interaction with the virtual object includes either moving the virtual object 602 or altering its dimensions.) It would have been obvious before the earliest effective filing date of the invention for the combined teachings of Minakuchi / Choinacka / Mittleman to be modified to display, based on the real IOT devices, a control item based on camera viewpoint change in that the moves the virtual object based on the coordinates of the anchoring points. This would have enabled a user to have an interactive experience within a real-world room or structure, or any indoor or outdoor area of interest. (Byl, [0002])
As per claim 11, Minakuchi / Choinacka / Mittleman disclose the electronic device of claim 10. Byl discloses wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: identify at least one real object included in the image through analysis of the image; and determine at least one IoT device capable of being disposed in the at least one space, based on at least one of a size, distribution, or density of the identified real object. ([0036] Referring now to FIG. 3, a flowchart of a method for rendering virtual objects within real physical space and interacting with these virtual objects is illustrated, in accordance with an embodiment. The real physical space, for example, may include, but is not limited to an open space for installation of permanent or temporary structures (for example, a home, an office, a building, an exhibition, a ceremony, a conference, etc.), a vehicle (for example, a car, a plane, a recreational vehicle, etc.), or a bare shell in a building. In order to accurately overlay virtual objects or models within the real physical space, augmented reality device 106 can capture spatial information associated with the real physical space using optical sensors 206 and additional sensors 208, at step 302. [0049]; In other words, when the virtual object is overlaid after defining the anchoring point and the anchoring vector of a physical space, the predefined anchoring point and vector in the virtual object may not exactly coincide with the desired location viewed in the physical space due to the anchoring point and vector in the real physical space being misplaced. This imprecise initial placement is also depicted in FIG. 6, where 3D hologram 602 of the virtual objects 602 can be seen slightly displaced from the desired area of placement within the room 408.)
As per claim 19, please see the discussion under claim 10 as similar logic applies.
Conclusion
The prior art made of record and not relied upon is considered pertinent toapplicant's disclosure. See PTO-892 form.
Any inquiry concerning this communication or earlier communications from theexaminer should be directed to Chirag R Patel whose telephone number is (571)272-7966. The examiner can normally be reached on Monday to Friday from 9:00AM to 6:00PM. If attempts to reach the examiner by telephone are unsuccessful, theexaminer's supervisor, Glenton Burgess, can be reached on 571-272-3949. The fax phone number for the organization where this application or proceedingis assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status informationfor published applications may be obtained from either Private PAIR or PublicPAIR. Status information for unpublished applications is available throughPrivate PAIR only. For more information about the PAIR system, seehttp://pairdirect.uspto.gov. Should you have questions on access to the PrivatePAIR system, contact the Electronic Business Center (EBC) at 866-217-9197(toll free).
/Chirag R Patel/
Primary Examiner, Art Unit 2454
Prosecution Insights