DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ganjam et al. (US 2017/0311053 A1), hereinafter referred to as Ganjam.
Regarding claim 1, Ganjam teaches a method implemented by one or more processors, the method comprising:
receiving a real-time image feed at a computing device from one or more cameras, the real-time image feed capturing one or more images of an object present in a field of view of the one or more cameras (Ganjam ¶¶0040: “the entity identifier 210 may continuously collect, receive, and/or record internal sensor data 135 and external sensor data 145 as a constant stream of sensor data. In other implementations, the sensor data may be received and/or collected in response to the user activating the personal digital assistant application 112 associated with the entity engine 120”; Ganjam Fig. 3);
displaying the real-time image feed at a display device (Ganjam Fig. 3);
causing one or more selectable elements to be rendered over the real-time image feed at the display device (Ganjam Fig. 3: see identified entities 305a-d and user-interface element 310; Ganjam ¶¶0091: “As shown in a user-interface element 310, the personal digital assistant application may have prompted the user to provide a query by asking ‘How can I help you?’ In addition, the entity identifier 210 may have identified the slice of pizza 305d as the entity of interest as indicated by the prompt ‘Are you interested in the pizza?’”);
receiving a selection of a selectable element, of the one or more selectable elements, that are rendered at the display device (Ganjam Fig. 4 & ¶¶0092: “the user has spoken ‘Yes’, which is interpreted by the personal digital assistant application as the user indicating that they are interested in the slice of pizza 305d”); and
in response to the selection of the selectable element that is rendered at the display device:
causing output to be rendered at one or more interfaces of the computing device, the output indicating object data corresponding to a conversation mode that is initiated based on receiving the selection of the selectable element, wherein the selectable element identifies the conversation mode in which the object data is generated (Ganjam Figs. 3-8 show various conversation prompts with the automated assistant and the user; Ganjam ¶¶0026: “the entity engine 120 may apply one or more of audio, face, text, and object recognition algorithms to the sensor data to identify the entities 167”; Ganjam ¶¶0135: “The action may include one or more of generating an alarm, generating a notification, performing an internet search, performing a text-to-speech translation, finding a location, and performing a financial transaction”).
Regarding claim 2, Ganjam teaches the method of claim 1, further comprising:
identifying, based on processing the one or more images, the object (Ganjam Abstract: “The sensor data is processed to identify entities such as persons, locations, and objects that may be of interest to the user”).
Regarding claim 3, Ganjam teaches the method of claim 2, wherein identifying, based on processing of the one or more images, the object comprises:
generating an identifier for the object (Ganjam ¶¶0033: “The information presented (in a card other format) may depend on the specific identity of an entity 167”; Ganjam ¶¶0103: “The plurality of entities 167 may be identified by the entity identifier 210 of the entity engine 120. The identified entities 167 may include people, objects, and locations that can be identified from the sensor data. Depending on the implementation, the entity identifier 210 may identify the entities 167 using one or more APIs directed to identifying different types of entities 167 (e.g., facial recognition, object recognition, OCR, etc.)”).
Regarding claim 4, Ganjam teaches the method of claim 1, wherein the object includes text and wherein the selectable element identifies the conversation mode, in which the object data is generated, as a translation mode (Ganjam Figs. 3-8 & ¶¶0103 discussed above; also see Ganjam ¶¶0043: “To identify entities that are books, an API that is directed to recognizing words or text may be used (i.e., OCR)”; Ganjam ¶¶0135: “performing a text-to-speech translation”).
Regarding claim 5, Ganjam teaches the method of claim 4, further comprising:
comparing a language of the text with a primary dialect setting of the computing device (Ganjam ¶¶0082: “Where the textual material is in a language other than the primary language spoken by the user, the entity engine 120 may translate the text using a variety of translation tools or APIs”; Ganjam ¶¶0135: “performing a text-to-speech translation”); and
in response to the language of the text being different from the primary dialect setting, providing a translated text at the display device, that is a translation of the text, at the display device (Ganjam ¶¶0082 & ¶¶0135 discussed above; Ganjam ¶¶0064: “The personal digital assistant application may receive the identified entities of interest, and may display indications of some or all of the entities of interest on the display of the client device 110”).
Regarding claim 6, Ganjam teaches the method of claim 5, further comprising:
receiving a selection of another selectable element, of the one or more selectable elements that are rendered at the display device, and in response to the selection of the other selectable element that is rendered at the display device (Ganjam ¶¶0090: “in response to the activation, the entity identifier 210 has identified several entities 167 in the video data. The identified entities 167 include a set of keys 305a, a can of soda 305b, a person 305c, and a slice of pizza 305d”):
causing additional output to be rendered at one or more of the interfaces of the computing device, the additional output indicating object data corresponding to another conversation mode that is initiated based on receiving the selection of the additional selectable element, wherein the additional selectable element identifies the other conversation mode (Ganjam ¶¶0090 discussed above; Ganjam ¶¶0092-¶¶0093: “the personal digital assistant application has displayed a user interface element 410 that includes attributes associated with the slice of pizza 305d … the user continues to interact with the personal digital assistant application and has provided the query of ‘How many calories are in this meal?’ as indicated in the user-interface element 510 … the query engine 220 may determine that the term ‘meal’ refers to food, and that in the context that includes the identified entities 167 there are two entities 167 that are food (i.e., the can of soda 305b and the slice of pizza 305d)”).
Regarding claim 7, Ganjam teaches the method of claim 6, wherein the object includes a food, and wherein the additional selectable element identifies the other conversation mode as a calorie mode (Ganjam Fig. 4: 305d, 410; Ganjam Figs. 5-6: 510, 610).
Regarding claim 8, Ganjam teaches the method of claim 1, wherein the object data is further tailored to the object identified from the one or more images in the real-time image feed (Ganjam Figs. 3-4 & ¶¶0092 discussed above teaches providing additional information regarding the identified object).
Regarding claim 9, Ganjam teaches the method of claim 1, wherein the selection of the selectable element is received responsive to touch interaction with a touch interface of the display device (Ganjam ¶¶0065: “the user may type the query using a touch interface provided by the client device 110”).
Regarding claim 10, Ganjam teaches the method of claim 1, wherein the output includes audible output that is rendered at the computing device while the real-time image feed is displayed at the display device (Ganjam ¶¶0075: “The generated notifications 169 may include any type of electronic notification including, but not limited to, text messages, email messages, and voicemail messages”; Ganjam ¶¶0089: “FIG. 3 is an illustration of an example client device 300. In the example, the client device 300 is a smart phone or a tablet computer that includes a touch screen 307”; Ganjam ¶¶0130: “Output device(s) 1216 such as a display, speakers, printer, etc. may also be included”; Ganjam ¶¶0135 discussed above teaches performing text-to-speech translation).
Regarding claim 11, Ganjam teaches the method of claim 1, wherein the display device is included in a client device that is different from the computing device (Ganjam ¶¶0089: “FIG. 3 is an illustration of an example client device 300. In the example, the client device 300 is a smart phone or a tablet computer that includes a touch screen 307”; Ganjam ¶¶0133: “The entity engine may be implemented on one or both of the at least one computing device or a cloud computing platform”).
Regarding claim 12, Ganjam teaches the method of claim 1, wherein the display device is included in the computing device (Ganjam ¶¶0133 discussed above).
Regarding claim 13, Ganjam teaches the method of claim 1, further comprising:
in response to a different object being present in the field of view of the camera, causing a different selectable element to be rendered over the real-time image feed at the display device (Ganjam Fig. 7: the automated assistant provides a selectable element regarding other detected object, e.g., a man in the window).
Regarding claim 14, Ganjam teaches a system comprising:
one or more storage devices storing instructions (Ganjam Fig. 12: 1204 & ¶¶0125: “Computer-executable instructions, such as program modules, being executed by a computer may be used … program modules and other data may be located in both local and remote computer storage media including memory storage devices”); and
one or more processors that are operable to execute the instructions to cause the one or more processors (Ganjam Fig. 12: 1202 & ¶¶0124: “Numerous other general purpose or special purpose computing devices environments or configurations may be used. Examples of well-known computing devices, environments, and/or configurations that may be suitable for use include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network personal computers (PCs), minicomputers, mainframe computers, embedded systems, distributed computing environments that include any of the above systems or devices, and the like”) to:
receive real-time image feed data from a computing device, the real-time image feed data indicating an object that is captured by a camera of the computing device or another computing device and that is present in a field of view of the camera (Ganjam ¶¶0040 & Fig. 3 discussed above);
identify selection of a selectable element that is rendered at the computing device and that corresponds to the real-time image feed data that is received from the computing device (Ganjam ¶¶0091 & Figs. 3-6 discussed above); and
in response to identifying the selection of the selectable element that is rendered at the computing device and receiving the real-time image feed data:
generating object data that corresponds to the object and a conversation mode that is initiated based on identifying the selection of the selectable element (Ganjam Figs. 3-6 & ¶¶0090-¶¶0093 discussed above), and
causing output to be rendered at one or more interfaces of the computing device, the output indicating the generated object data (Ganjam Figs. 6, 8).
Regarding claim 15, Ganjam teaches a wearable computing device (Ganjam ¶¶0021: “The client device 110, or a user associated with the client device 110 … sensors associated with a security system (e.g., cameras and motion detectors), sensors associated with a fitness tracker (e.g., heart rate monitor, scale, pedometer, blood pressure monitor, blood oxygen level monitor), sensors associated with a smart watch”) comprising:
a display interface (Ganjam ¶¶0064 discussed above; Ganjam ¶¶0130: “Output device(s) 1216 such as a display, speakers, printer, etc. may also be included.);
an audio interface (Ganjam ¶¶0130 discussed above);
memory storing instructions (Ganjam Fig. 12 & ¶¶0125 discussed above);
one or more processors operable to execute the instructions (Ganjam Fig. 12 & ¶¶0124 discussed above) to:
receive real-time image feed data from one or more cameras of the wearable computing device, the real-time image feed capturing one or more images of an object that is present in a field of view of the one or more cameras (Ganjam ¶¶0040 & Fig. 3 discussed above);
identify a selection, of a user of the wearable computing device and via one or more of the display interface or the audio interface, of a selectable element that corresponds to the real-time image feed that is received from the wearable computing device (Ganjam ¶¶0091 & Figs. 3-6 discussed above); and
in response to identifying the selection of the selectable element that is rendered at the wearable computing device:
cause output to be rendered at one or more of the display interface or the audio interface of the wearable computing device, the output indicating object data corresponding to a conversation mode that is initiated based on receiving the selection of the selectable element, wherein the object data is generated based on the selectable element corresponding to the conversation mode (Ganjam Figs. 3-8, ¶¶0026, & ¶¶0135 discussed above).
Regarding claim 16, Ganjam teaches the wearable computing device of claim 15, wherein he object data is filtered based on a context that is identified based on processing one or more previously received images (Ganjam ¶¶0072: “the query engine 220 may determine what is referred to as a context. The context for a query may be based on the attributes of the currently identified entities 167, and any recently received user input 137, for example. Any method for determining a query context may be used”; Ganjam ¶¶0093: “the query engine 220 may determine that the term ‘meal’ refers to food, and that in the context that includes the identified entities 167 there are two entities 167 that are food (i.e., the can of soda 305b and the slice of pizza 305d)” – the non-food entities 305a and 305c are excluded in the query-answer elements 510 & 610).
Regarding claim 17, Ganjam teaches the wearable computing device of claim 15, wherein the object data is filtered based on a context that is identified based on determining a location of the object, the wearable computing device, or the user (Ganjam ¶¶0072: “the query engine 220 may determine what is referred to as a context. The context for a query may be based on the attributes of the currently identified entities 167, and any recently received user input 137, for example”; Ganjam ¶¶0093: “the query engine 220 may determine that the term ‘meal’ refers to food, and that in the context that includes the identified entities 167 there are two entities 167 that are food (i.e., the can of soda 305b and the slice of pizza 305d)” – the non-food entities 305a and 305c are excluded in the query-answer elements 510 & 610).
Regarding claim 18, Ganjam teaches the wearable computing device of claim 15, wherein the output includes natural language content describing one or more facts corresponding to the object and a graphical representation of the object (Ganjam ¶¶0025: “A personal digital assistant application 112 may be an application of the client device 110 that receives one or more natural language queries from the user as part of a user input 137”; Ganjam Figs. 3-7 & ¶¶0089-¶¶0092 discussed above; also see Ganjam ¶¶0023: “Other types of entities 167 may include virtual objects (e.g., songs), characters in books, computer generated glyphs, icons and related UI widgets (cursor, documents, files, windows, title bars, etc.)”; Ganjam ¶¶0083: “The identified entities 167 may include characters, people, places, and objects described in the textual material or shown in pictures of the textual material”; Ganjam ¶¶0099: “To notify the user that the rule has been created, the personal digital assistant application has displayed a confirmation message of ‘I will alert you when Robert Smith is near you’ in a user-interface element 820”).
Regarding claim 19, Ganjam teaches the wearable computing device of claim 15, further comprising:
identifying, based on processing the one or more images, the object (Ganjam Abstract discussed above).
Regarding claim 20, Ganjam teaches the wearable computing device of claim 19, wherein identifying, based on processing the one or more images, the object, comprises:
generating an identifier for the object (Ganjam ¶¶0033 & ¶¶0103 discussed above).
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. US 12,175,747 B2 (S/N: 17/888,163) and claims 1-20 of U.S. Patent No. 11,417,092 (S/N: 16/806,505). Although the claims at issue are not identical, they are not patentably distinct from each other because they are directed to methods and devices for image-responsive automated assistants comprising receiving real-time image feeds from one or more cameras, identifying the objects in the field of view of the cameras, generating identifiers, recognizing texts, translating the texts, providing conversation modes corresponding to selectable elements, and providing natural language content and processing results on the user device, e.g., display, audio interface, etc.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOO J SHIN whose telephone number is (571)272-9753. The examiner can normally be reached M-F; 10-6.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Soo Shin/Primary Examiner, Art Unit 2667 571-272-9753
soo.shin@uspto.gov
