DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1-2, 4-8, 10-12, 14-18, and 20 are rejected under 35 U.S.C. 103.
Claims 3, 9, 13, and 19 are objected to for depending from a rejected base claim.
Claims 7-10 and 17-19 are objected to for minor informalities.
Claim Objections
Claims 7-10 and 17-19 objected to because of the following informalities:
In line 1 of claim 7, “a suggested query” and “an entity” should read “the suggested query” and “the entity”. Claim 17 requires similar correction. Claims 9-10 and 18-19 are objected to due to their dependencies.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4-5, 11, 14-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over VAN OS (US 2015/0382047 A1) in view of BURON (US 2007/0198495 A1).
Regarding Claim 1, VAN OS teaches a method performed by one or more processors of a user device, the method comprising… obtaining context data indicating: displayed content that is displayed on a screen of the user device… and previous information relating to previous user interactions with the user device; (“virtual assistant query suggestions can be determined based on displayed media content or a viewing history of media content (e.g., a movie, television show, sporting event, recently viewed show, recently viewed menu, recently viewed scene of a movie, recent scene of a playing television episode, etc.)… Metadata associated with displayed content (e.g., descriptive details of the media content) can also be used to determine query suggestions… For example, metadata associated with video 480 can include the character names of characters 1910, 1912, and 1914 along with the actresses who play those characters.” ¶ 188. Displayed content and metadata associated with the displayed content is obtained. Also see ¶ 189, which discusses performing facial recognition on the screen data, and ¶ 196, which discusses the screen data being a displayed notification and analyzing the text content within the notification. These are other methods of obtaining “displayed content that is displayed on a screen of the user device”. The viewing history information that is obtained also reads on “previous information relating to previous user interactions with the user device”.)
…generating, based on the context data including the displayed content, the off-screen information, and the previous information, a suggested query that refers to an entity in the displayed content; (¶ 189-191. Query suggestions are generated based on the terms that are determined to be relevant to a selected entity in the displayed content. In the example of an actor that is identified from the screen data, a query suggestion may be “What else is this actor in?”. A user can select a query suggestion by speaking the query, as discussed in Paragraph 193. See Figures 26 and 29 for examples of suggested queries.)
and providing the suggested query (“Suggestions interface 2650 can be displayed over a moving image, such as video 480, or over any other background content (e.g., a menu, a still image, a paused video, etc.)… virtual assistant query suggestions can be determined based on displayed media content or a viewing history of media content… FIG. 26 illustrates content-based suggestions 2652, which can be determined based on displayed video 480 shown in the background with characters 1910, 1912, and 1914 appearing on display 112.” Paragraphs 187-188. Suggested voice requests are visually displayed for a user after they invoke the virtual assistant for providing the suggested voice requests while the content is still displayed.)
receiving user input that invokes digital assistant functionality and providing the suggested query in response to receiving the user input that invokes the digital assistant functionality and before the user provides query terms following invocation of the digital assistant functionality. (“Input requesting query suggestions can be received, for example, from user device 102 or remote control 106. In some examples, the input can include a button press, a double click of a button, a menu selection, a voice command (e.g., show me some suggestions, what can you do for me, what are some options, etc.), or the like received at user device 102 or remote control 106. For instance, a user can double click a physical button on remote control 106 to request query suggestions, or can double click a physical or virtual button on user device 102 when viewing an interface associated with television set-top box 104 to request query suggestions.” Paragraph 0186. A user inputs a command for invoking a digital assistant functionality while content is currently displayed on an electronic device, such as the television show depicted in Figure 26. Also see Paragraph 199, which gives the example of a notification being displayed followed by a user requesting query suggestions, which results in query suggestions being displayed based on the content displayed in the notification. The query suggestions are provided in response to the input and before the user provides any further input, such as query terms.)
VAN OS does not teach the context data including off-screen information that comprises content that is not currently visible on the screen of the user device, but would be shown in a left, right, up, or down swiping, or scrolling view of the screen of the user device,
However, BURON, which is directed to a map search interface, teaches off-screen information that comprises content that is not currently visible on the screen of the user device, but would be shown in a left, right, up, or down swiping, or scrolling view of the screen of the user device, (¶ 87-88, Fig. 11: Query results are ranked and provided to the user of a device based on whether they are on-screen or off-screen. The off-screen content would be displayed in response to panning (i.e. scrolling or swiping) gestures. Content closer to the current viewport is higher ranked.)
Before the effective filing date of the invention, it would have been obvious to one
of ordinary skill in the art to modify the generation of query suggestions based on contextual data, such as currently and previously displayed data, taught by VAN OS by including off-screen data in the determination of the query suggestions as taught by BURON. Since the references are similarly directed to search interfaces that consider the context of the display, the combination would have yielded predictable results and would have amounted to scoring or ranking the suggestions based on their location in or out of a viewport. As suggested by BURON (¶ 4-6), such an implementation would improve the flexibility of the search interface, improving the reporting of location search queries to the user.
Claim 11 is directed to a client device comprising: a screen; a memory storing instructions; and one or more processors operable to execute the instructions, stored in the memory and claim 20 is directed to one or more non-transitory computer-readable storage media encoded with a software program, the program comprising instructions that, when executed by one or more computing devices, cause the one or more computing devices to execute instructions. Claim 11 and Claim 20 otherwise recite the same limitations as claim 1 and are rejected for the same reasoning discussed above.
Regarding Claim 4, VAN OS in view of BURON further teaches wherein the suggested query is displayed on the screen of the user device. (VAN OS, ¶ 1886-189, Figs. 26, 29: The suggested queries (i.e. “content-based suggestions”) are displayed on a screen of the user device.)
Claim 14 recites the same limitations as claim 4 and is rejected for the same reasoning discussed above.
Regarding Claim 5, VAN OS in view of BURON further teaches wherein the suggested query requests the digital assistant functionality to perform an action. (VAN OS, “Answer interface 2862 can include informational answers and/or media results responsive to a selected query suggestion (or responsive to any other query). For example, in response to selected query suggestion 2756, assistant result 2860 can be determined and provided… Informational answers and media results (e.g., selectable video links) can also be presented in any of the other ways discussed herein, or results can be presented in various other ways (e.g., speaking answers aloud, playing content immediately, showing an animation, displaying an image, etc.).” Paragraph 195. The action being performed includes information retrieval, providing links, and automatically playing content.)
Claim 15 recites the same limitations as claim 5 and is rejected for the same reasoning discussed above.
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over VAN OS (US 2015/0382047 A1) in view of BURON (US 2007/0198495 A1) and further in view of LEMAY (US 10,241,752 B2).
Regarding Claim 2, VAN OS in view of BURON teaches all the limitations of claim 1, on which claim 2 depends.
VAN OS in view of BURON does not teach wherein generating the suggested query comprises: determining a pronoun for the entity; and generating a suggested query that includes the pronoun to refer to the entity referenced in the context data.
However, LEMAY, which teaches determining an entity to which a pronoun refers, teaches wherein generating the suggested query comprises: determining a pronoun for the entity; and generating a suggested query that includes the pronoun to refer to the entity referenced in the context data. (“If the user is currently using an application when virtual assistant 1002 is invoked, the state of that application can provide useful context information… Referring now to FIGS. 11 through 13, there is shown a set of screen shots depicting examples of the use of application context in a text messaging domain to derive a referent for a pronoun, according to one embodiment… when the user provides a speech input, the virtual assistant 1002 repeats the user's input as a text string within quotation marks (“Call him”). The virtual assistant 1002 then presents a text string output (with or without a simultaneous speech output) informing the user what action is about to be performed performed (e.g., “Calling John Appleseed's mobile phone: (408) 555-1212 . . . ”). Col. 18:1-35. A virtual assistant understands a pronoun in a voice command based on the contextual information, such as the application that is open and the information on the screen. In this example, the pronoun “him” is mapped to an entity on the screen, “John Appleseed”. It would have been obvious in combination with VAN OS for a voice suggestion given by a virtual assistant to include a pronoun that refers to the entity determined from the screen data.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to modify the voice suggestions based on currently displayed screen content taught by VAN OS in view of BURON by including suggestions with pronouns given the teachings of LEMAY. Since LEMAY is similarly directed to virtual assistants for handling voice requests, the combination would have yielded predictable results. Such a combination would have merely amounted to including a particular suggestion that has a pronoun as long as the virtual assistant is capable of entity resolution, such as the virtual assistant taught by LEMAY. Furthermore, VAN OS (¶ 127) at least suggests performing entity resolution on voice commands, such as those that use the terms “that” and “this”.
Claim 12 recites the same limitations as claim 2 and is rejected for the same reasoning discussed above.
Claims 6-8 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over VAN OS (US 2015/0382047 A1) in view of BURON (US 2007/0198495 A1) and further in view of NORDSTROM (US 8,930,393 B1).
Regarding Claim 6, VAN OS in view of BURON teaches all the limitations of claim 5, on which claim 6 depends.
VAN OS in view of BURON does not teach wherein the action is to call a contact whose contact information is referenced in the displayed content or to provide directions to a location referenced in the displayed content.
However, NORDSTROM, which is similarly directed to providing query suggestions, teaches wherein the action is to call a contact whose contact information is referenced in the displayed content or to provide directions to a location referenced in the displayed content. (“the server 130 loads a copy of the web page 224 using the URL provided by the client and then interprets the content of the web page (e.g., text and images) to identify any entity(s) and predicted or expressed (e.g., via user selection) user purpose(s) associated with the web page 224… Sushi ABC is also associated in the entity database 234 with the purposes of going to the restaurant, calling the restaurant, saving information on the restaurant, and offers associated with the restaurant… the entities and actions are displayed in step 350 in a suggestions list below the search input field 424 when the user directs a cursor to the search input field 424 while the content web page 224 is still loaded and displayed… "Directions from your location to Sushi ABC" 414, which, when selected by the user, takes the user to a web page in the web browser 222 displaying directions from the current location of the tablet computer 110 to the geographical address for Sushi ABC; "Call Sushi ABC" 416, which, when selected by the user, places a telephone call from the tablet computer 110 to the work telephone number for Sushi ABC” Col. 12:1-61. Currently displayed content is analyzed to determine an entity. Suggestions relevant to the entity are determined from a knowledge graph related to a classification, e.g. location or restaurant, of the entity. The suggested actions include calling and providing directions.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to modify the voice suggestions based on currently displayed screen content taught by VAN OS in view of BURON by including suggestions for calling or providing directions to a location based on the identified entity in the screen data as taught by NORDSTROM. Since NORDSTROM similarly identifies and classifies entities on a current screen in order to determine the suggestions, the combination would have yielded predictable results. Furthermore, as taught by NORDSTROM (Col. 11:46-58), certain actions would be associated with certain entities and providing suggestions would assist the user in performing tasks.
Claim 16 recites the same limitations as claim 6 and is rejected for the same reasoning discussed above.
Regarding Claim 7, VAN OS in view of BURON teaches all the limitations of claim 1, on which claim 7 depends.
VAN OS in view of BURON further teaches wherein generating a suggested query that refers to an entity in the displayed content comprises: identifying multiple entities referenced in the displayed content, (VAN OS, “an actor or actress appearing on display 112 can be identified (e.g., based on metadata and/or facial recognition), and query suggestions associated with that actor or actress can be provided” Paragraph 190. “category, genre, rating, awards, descriptions, or the like associated with displayed content can be used to formulate query suggestions. For example, video 480 can correspond to a television program described as a comedy having female lead characters. A query suggestion can be formulated from this information to identify other shows with similar characteristics (e.g., “Find me other comedies with female leads.”)” Paragraph 192. “a notification about an exciting end of a live sporting event can be displayed. Should the user then request query suggestions, suggestions interface 2650 can be displayed, including query suggestions to view the sporting event, inquire about team statistics, or find content related to the notification… Based on the particular terms of interest identified in the notification, various other query suggestions can likewise be determined and provided to the user” Paragraph 199. Multiple entities, which can include people identified on the screen, information associated with displayed content, and text within a displayed notification, are determined based on processing of screen data.)
VAN OS in view of BURON does not teach ranking each of the identified multiple entities based on prominence of each of the identified multiple entities in the displayed content and off-screen information, and selecting the entity, from the identified multiple entities, based on the ranking.
However, NORDSTROM, which is similarly directed to providing query suggestions, teaches ranking each of the identified multiple entities based on prominence of each of the identified multiple entities in the displayed content and off-screen information, and selecting the entity, from the identified multiple entities, based on the ranking. (“Various ways to identify a referent entity of user-selectable content will now be described. User viewable content can include one or many referents, each of which is an entity. In cases where there are multiple entities identified in user viewable content, each entity can be weighted (or "scored") based on a likelihood that it is a referent of the user viewable content, with the highest weighted entity being the most likely referent.” Col. 8:14-21. Multiple entities are weighted and the entity that is selected is based on the weights determined for the multiple entities. The weights are equivalent to a ranking since they are a score for each entity.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to modify the presentation of voice query suggestions to a user taught by VAN OS in view of BURON by determining the entity on the screen that is relevant to the user based on a ranking as taught by NORDSTROM. Since NORDSTROM similarly identifies and classifies entities on a current screen in order to determine the suggestions, the combination would have yielded predictable results. Such a combination would improve the user experience by providing only the most relevant suggestions to the user.
Claim 17 recites the same limitations as claim 7 and is rejected for the same reasoning discussed above.
Regarding Claim 8, VAN OS in view of BURON and NORDSTROM further teaches wherein ranking each of the identified multiple entities comprises: ranking each of the identified multiple entities based on the respective classifications for each of the identified multiple entities. (NORDSTROM, “. For example, the user viewable content "Miromesnil" can refer to two entities, the Paris metro stop Miromesnil and the restaurant named Miromesnil. The restaurant referent may be assigned a higher value (probability=0.9) than the metro stop referent (probability=0.1) because on contextual data indicating, among other things, that the user may have recently conducted a search for restaurants on the user's client 110.” Col. 8:21-28. Based on the classification of an entity and other conditions, the entity will receive a higher weight. For example, an entity corresponding to a restaurant is ranked higher than the entity referring to a metro stop.)
The same motivation to combine discussed in the rejection of claim 7 applies to claim 8.
Claim 18 recites the same limitations as claim 8 and is rejected for the same reasoning discussed above.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over VAN OS (US 2015/0382047 A1) in view of BURON (US 2007/0198495 A1) and further in view of NORDSTROM (US 8,930,393 B1) and LI (US 2012/0130978 A1).
Regarding Claim 10, VAN OS in view of BURON and NORDSTROM teaches all the limitations of claim 7, on which claim 10 depends.
VAN OS in view of BURON does not teach wherein a given entity of the identified multiple entities corresponds with a first graphical characteristic and another entity of the identified multiple entities corresponds with a second graphical characteristic, and the ranking of the identified multiple entities is based on the first graphical characteristic and the second graphical characteristic.
However, LI, which is directed to query suggestions based on information on a document, teaches wherein a given entity of the identified multiple entities corresponds with a first graphical characteristic and another entity of the identified multiple entities corresponds with a second graphical characteristic, and the ranking of the identified multiple entities is based on the first graphical characteristic and the second graphical characteristic. (“The scanner 210 determines the context of a query trigger in the resource 212 by identifying the display format of the query trigger in the resource. Example display formats include bold, underline, italicized, highlighted, footnoted, or a different size font than the rest of the text of the resource 212. The context can also include, for example, whether the query trigger appeared in the title or in a heading in the resource 212.” Paragraph 46. See the examples in Paragraphs 67-70: Multiple entities are determined having different graphical characteristics, such as boldness, appearing in a title section, or having different size fonts. Based on these graphical characteristics, the entities are ranked and then selected for use with suggested search queries, as shown in Figure 3B.)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to modify the ranked voice suggestions based on currently displayed screen content taught by VAN OS in view of BURON and NORDSTROM by selecting entities for the suggested voice requests based on the graphical characteristics of the content on the display, as taught by LI. Since the references are similarly directed to generating suggested queries, the combination would have yielded predictable results. As taught by LI (Paragraph 3), such an implementation would improve the user experience by reducing the repetitive process of refining query terms related to the subject matter described in content currently being viewed.
Allowable Subject Matter
Claims 3, 9, 13, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Macbeth (US 2014/0101600 A1) teaches contextual digital assistant suggested actions based on currently displayed content. (¶ 44, Fig. 2)
Gross (US 2016/0360336 A1) teaches providing query suggestions based on the content in an application that is currently displayed and based on previously displayed content. (¶ 1011-1018, 818-822, Figs. 11A-B)
Fung (US 10,459,745 B2) teaches suggested questions for a user of a design application interface that consider context, including previous workflows and other sections of the application. (Fig. 1B, claim 1, Col. 8:1-25)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAMI RAFAT OKASHA whose telephone number is (571)272-0675. The examiner can normally be reached M-F 10-6 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SCOTT BADERMAN can be reached at (571) 272-3644. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAMI R OKASHA/Primary Examiner, Art Unit 2118
Prosecution Insights