DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1, 3-7, 9, 11-15, 17 and 19-20 are pending for examination.
Claims 1, 9 and 17 are independent Claims.
Claims 1, 3-7, 9, 11-15, 17 and 19-20 are rejected under 35 U.S.C. §103.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-7, 9, 11-15, 17 and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Smith et al. (U.S. 2012/0284426 hereinafter Smith) in view of Bui et al. (U.S. 2020/0042286 hereinafter Bui) in further view of Fein et al. (U.S. 2014/0282162 hereinafter Fein).
As Claim 1, Smith teaches a method of generating an interactive AR experience using a portable electronic device comprising a microphone (Smith (¶0031 line 3), audio recording), a camera (Smith (¶0029 line 3-6), camera), a display (Smith (¶0010 last 2 lines), displays), and a memory (Smith (¶0045 line 1-3), memory), the method comprising:
capturing frames of video data of a physical environment using the camera (Smith (¶0052 line 8-15), recipient playbacks the video in order to view the scene with additional subsequent scene);
which represents a portion of the interactive AR experience (Smith (¶0056 line 3-12), user zooms in and annotates a virtual object with record audio and/or touch input);
Smith may not explicitly disclose:
presenting on the display a question and one or more answer buttons, wherein the question is associated with a scene
receiving a user input comprising an answer to the question, wherein the answer is selected from among the one or more answer buttons;
Bui teaches:
presenting on the display a question and one or more answer buttons, wherein the question is associated with a scene (Bui (¶0136 last 8 lines, fig. 4 item 418), system displays question 418 with option for the answers)
receiving a user input comprising an answer to the question, wherein the answer is selected from among the one or more answer buttons (Bui (¶0136 last 8 lines, fig. 4 item 418), user wishes to describe another edit of first image);
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the voice input of Smith instead be voice input taught by Bui, with a reasonable expectation of success. The motivation would be to offer user convenience by providing an efficient user interface for image editing (Bui (¶0004)).
Smith in view of Bui may not explicitly disclose:
establishing in response to the user input a virtual object position relative to the physical environment;
presenting on the display a virtual object at the virtual object position, wherein the virtual object is defined in size relative to the virtual object position;
determining, using the localizing system, a portable electronic device location relative to the virtual object position;
detecting in the frames of video data a physical hand location relative to the physical environment;
detecting, in accordance with the portable device location, an intersection between the physical hand location and the virtual object;
executing a touch response relative to the virtual object in accordance with the intersection, wherein the touch response comprises an action selected from a touch group consisting of presenting on the display an object name associated with the virtual object, presenting on the display a highlight near the virtual object, playing a sound through a speaker coupled to the portable electronic device, and generating a tactile movement of the portable electronic device; and
storing in the memory the virtual object, the virtual object position, the intersection, and the touch response associated with the scene
Fein teaches:
establishing in response to the user input a virtual object position relative to the physical environment (Fein (¶0078 line 1-5, fig. 4B), “FIG. 4b depicts a user's hand pointing to a book on one of the shelves; this gesture is detected by e.g., augmented reality system 322 and/or image processing module 312, which may capture text printed on the spine of the book in the vicinity of, or touched by (a virtual object position), the user's index finger”);
presenting on the display a virtual object at the virtual object position, wherein the virtual object is defined in size relative to the virtual object position (Fein (¶0080, fig. 4c), “FIG. 4c depicts that a single book on the bookshelf is highlighted (virtual object at the virtual object position), corresponding to the one pointed to by the user's index finger.”);
determining, using the localizing system (Fein (¶0078 line 6-13), “context evaluation module 308”), a portable electronic device location relative to the virtual object position (Fein (¶0077, fig. 4a), “an augmented reality device (a smartphone showing on its display a bookshelf containing books in the camera's field of view”);
detecting in the frames of video data a physical hand location relative to the physical environment (Fein (¶0078 line 1-5, fig. 4B), “FIG. 4b depicts a user's hand pointing to a book on one of the shelves; this gesture is detected by e.g., augmented reality system 322 and/or image processing module 312, which may capture text printed on the spine of the book in the vicinity of, or touched by, the user's index finger”);
detecting, in accordance with the portable device location, an intersection between the physical hand location and the virtual object ((Fein (¶0078 line 1-5, fig. 4B), “FIG. 4b depicts a user's hand pointing to a book on one of the shelves; this gesture is detected by e.g., augmented reality system 322 and/or image processing module 312, which may capture text printed on the spine of the book in the vicinity of, or touched by (an intersection between the physical hand location and the virtual object), the user's index finger”));
executing a touch response relative to the virtual object in accordance with the intersection, wherein the touch response comprises an action selected from a touch group consisting of presenting on the display an object name associated with the virtual object, presenting on the display a highlight near the virtual object (Fein (¶0080, fig. 4c), “FIG. 4c depicts that a single book on the bookshelf is highlighted, corresponding to the one pointed to by the user's index finger.”), playing a sound through a speaker coupled to the portable electronic device, and generating a tactile movement of the portable electronic device; and
storing in the memory the virtual object, the virtual object position, the intersection, and the touch response associated with the scene (Fein (¶0080, fig. 4c), “FIG. 4c depicts (the touch response) that a single book (virtual object position) on the bookshelf is highlighted (virtual object), corresponding to the one pointed to by the user's index finger (the intersection).”)
Smith in view of Lin teaches a system and method to manipulate virtual object. Fein teaches a system to generate a virtual object at an intersection of user’s finger and real object. The virtual object can be further manipulated using gesture. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the touch user interface of Smith in view of Lin with an augmented user interface taught by Fein because user touch interface and augmented user interface are exchangeable. The motivation would be to offer user convenience by “allows for powerful manipulation of graphical objects and environment” (Fein (¶0049 line 3-6)).
As Claim 3, besides Claim 1, Smith in view of Bui in further view of Fein teaches wherein the method further comprises:
executing a reaction associated with the intersection, wherein the reaction comprises an effect selected from a group consisting of interacting with the virtual object (Fein (¶0080, fig. 4c), “FIG. 4c depicts that a single book on the bookshelf is highlighted, corresponding to the one pointed to by the user's index finger.”), deleting the virtual object, starting to record a subsequent scene associated with the interactive AR experience, , presenting on the display a subsequent virtual object, and connecting the portable electronic device to an IoT product in the physical environment.
As Claim 4, besides Claim 1, Smith in view of Bui in further view of Fein teaches wherein the virtual object comprises an IoT product in the physical environment,
wherein detecting the intersection comprises detecting, based on the portable electronic device location, an intersection between the physical hand location and an IoT product location (Fein (¶0080, fig. 4c), “FIG. 4c depicts that a single book on the bookshelf is highlighted, corresponding to the one pointed to by the user's index finger.”), and
wherein executing the touch response comprises sending a control signal to the IoT product comprising instructions to perform an IoT action corresponding to the user behavior (Fein (¶0078 line 6-13, fig. 4b), “device context evaluation module 308 may detect that the device is running a program having virtual shopping cart functionality associated with a specific bookstore”).
As Claim 5, besides Claim 1, Smith in view of Bui in further view of Fein teaches further comprising:
presenting on the display a play button (Smith (¶0036 line 2-6), user’s annotation data (Datapod) is stored and presented as Youtube video); and
retrieving from the memory the virtual object, the intersection, and the touch response associated with the scene (Fein (¶0080, fig. 4c), “FIG. 4c depicts (the touch response) that a single book (virtual object position) on the bookshelf is highlighted (virtual object), corresponding to the one pointed to by the user's index finger (the intersection).”); and
presenting the scene on the display (Smith (¶0036 line 2-6), user’s annotation data (Datapod) is stored and presented as Youtube video).
As Claim 6, besides Claim 1, Smith in view of Bui in further view of Fein teaches:
wherein the answer comprises an invitation to speak, and wherein the method further comprises:
receiving through the microphone a spoken word (Bui (¶0134), user input trigger such as “add cape to this guy”);
identifying a trigger word based on the spoken word (Bui (¶0134), user input trigger such as “add cape to this guy”);
executing a word response based on the trigger word comprising an action selected from a word group consisting of presenting on the display a transcription of the identified trigger word, presenting a highlight on the display, playing a sound through a speaker coupled to the portable electronic device, playing a message through the speaker, and generating a tactile movement of the portable electronic device (Bui (¶0134), user’s voice command is transferred to a second image); and
storing in the memory the virtual object (Bui (¶0134), user input trigger such as “add cape to this guy”) and the trigger word associated with the scene (Bui (¶0134), user’s voice command (trigger word) is transferred to the generation of a second image (trigger word associated with the scene)).
As Claim 7, besides Claim 6, Smith in view of Bui in further view of Fein teaches wherein receiving the spoken word further comprises:
presenting on the display a graphic response to the spoken word (Bui (¶0134), user input trigger such as “add cape to this guy”); and
presenting the trigger word on the display (Bui (¶0134), user’s voice command is transferred to the generation of a second image);
presenting on the display a next button operative to accept the trigger word (Bui (¶0136 last 5 lines), user can select affirmative option of menu option 418).
As Claim 9 and 17, the Claims are rejected for the same reasons as Claim 1.
As Claim 11 and 19, the Claims are rejected for the same reasons as Claim 3.
As Claim 12 and 20, the Claims are rejected for the same reasons as Claim 4.
As Claim 13, the Claim is rejected for the same reasons as Claim 5.
As Claim 14, Claim 14 is rejected for the same reasons as Claim 6.
As Claim 15, Claim 15 is rejected for the same reasons as Claim 7.
Response to Arguments
Claim Rejections – 35 U.S.C. §103:
As Claims 1, 9 and 17, Applicants argue that cited references do not disclose “establishing in response to the user input a virtual object position …” (last paragraph of page 11 in the remarks).
PNG
media_image1.png
145
661
media_image1.png
Greyscale
Applicants’ arguments are moot because new reference Fein teaches the limitation(s);
Other independent/dependent Claims are not allowable for the same reasons above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NHAT HUY T NGUYEN whose telephone number is (571)270-7333. The examiner can normally be reached M-F: 12:00-8:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 571-270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NHAT HUY T NGUYEN/Primary Examiner, Art Unit 2147
Prosecution Insights