DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see page 8, filed 3/19/2026, with respect to the drawings have been fully considered and are persuasive. The objection of the drawings has been withdrawn.
Applicant's arguments filed 3/19/2026 have been fully considered but they are not persuasive. In particular, the specification objection is maintained since the current title is not descriptive of the claimed invention and is not sufficient to inform indexing, classifying and/or searching. The suggested title modification on page 9 is also not sufficient.
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on all references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. The arguments state that the previously applied reference does not disclose “contextual data determined from telecommunications network data and a user profile maintained by the telecommunications network, and an image data capture instruction identifying at least one of scanning a first area rather than an entire image or applying a predetermined character-string format during capture.” The newly added reference of Ramanujapuram cures the deficiencies of the primary reference and will be explained below.
Regarding the secondary reference of Ramanujapuram, the system discloses determining contextual data from the location of where the mobile device captures an image or the location of the mobile device. A user profile associated with the mobile device that captures an image is also considered contextual data. Both types of contextual data are taught in ¶ [15], [46]-[48] and [55]. The server uses a network, such as a LAN or WAN, or communicates using CDMA or GSM over the network, which is considered as telecommunications network device, or access point, a part of the telecommunications network. The server stores information, such as a profile on the user, that is used for comparison to other data and to gather further contextual information, which is taught in ¶ [50]-[56]. ¶ [15] and [55] discloses determining contextual data based on location and user profile data. Based on these features, the first contended feature is performed.
Regarding the second contended feature, the primary reference and the secondary reference perform this feature. The primary reference performs initially detecting location of a UE device in proximity to information that can be captured. In addition, the location of the device or the text within an image can be acquired as contextual information, which is taught in ¶ [48], [57], [86], [125], [231], [234] and [241]. Once the context information is received along with the proximity of the device to the page or paper including the text information is determined, the device can perform an OCR operation onto the text information to determine text of a certain font. This is taught in ¶ [241]-[245], [258], [264] and [544]-[554]. This performs the second contended feature of the claim. However, assuming arguendo, that the primary reference did not perform this aspect of the invention, this would be performed by the secondary reference as well.
Regarding the secondary reference, after the client device or server retrieves the image data and an image histogram to determine a match or not a match, the system causes the client device to perform capturing of data to identify an area within an entire image to be further scanned or recognized. The area can contain characters that reflects a western alphabet that is to be scanned for further comparison. This performs identifying a specific area within the image captured as well as a character font type that is recognized during the identification process. This is taught in ¶ [57] of the secondary reference. Therefore, based on the combination above, the features of the independent claims are disclosed.
Thus, based on the above, the features of the claims are disclosed below.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested:
CONTEXT AWARE DIGITAL VISION AND RECOGNITION COMPRISING RECEIVING LOCATION OF A USER EQUIPMENT, DETERMINING CONTEXTUAL DATA RELATED TO THE LOCATION, SELECTING AN IMAGE CAPTURING INSTRUCTION BASED ON THE LOCATION AND CONTEXTUAL DATA IN ORDER TO EXECUTE A CAPTURE INSTRUCTION.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-7, 9-15 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over King (US Pub 2014/0232889) in view of Ramanujapuram (US Pub 2009/0285492).
Re claim 1: King discloses a method for context aware digital vision in a network, the method comprising:
receiving a location of a user equipment (UE) (e.g. a location proximate to information of the capture device is detected, which is taught in ¶ [543].);
[0543] FIG. 4 is a flow diagram illustrating a routine 400 for identifying when a capture device is in proximity to or in the presence of information to be captured. In step 410, the system determines the capture device 300 is in the presence of information to be captured. The system may utilize the detection component 330 to determine the device is proximate to information. In some cases, the detection component 330 may facilitate detecting the presence of information via an imaging or other capture component 310 of the capture device. In some cases, the detection component 330 may facilitate detecting the presence of information via components that measure distances between the capture device and target objects (such as rendered documents, displays of information, and so on) and determine that the capture device is within a certain proximity to a target object associated with a user intention to capture information from or about that target object. In some cases, the detection component may facilitate detecting the presence of information via components that measure the orientation of the device relative to a target object and determine that the capture device is in a certain position or orientation, possibly for a certain duration of time, associated with a user intention to capture information from or about that target object. Further details and aspects regarding determining that a capture device is in the presence of information are described herein with respect to FIGS. 6A-6C.
determining contextual data related to the location of the UE, wherein the contextual data is determined from telecommunications network data associated with the UE and a user profile maintained (e.g. the proximity of the location of the capture device to a target object is determined, which is taught in ¶ [543] above. Moreover, contextual data can be considered as the geographic location of the user, which is considered as telecommunications network data associated with the UE, or the history of the capturing device or actions of the user. The capturing history or reading habits of the user are considered as user profile of a user, which is taught in ¶ [124]-[127].);
[0124] 4.2.2. Use of Context
[0125] Section 13 below describes a variety of different factors, which are external to the captured text itself, yet, which can be a significant aid in identifying a document. These include such things as the history of recent captures, the longer-term reading habits of a particular user, the geographic location of a user and the user's recent use of particular electronic documents. Such factors are referred to herein as "context".
[0126] Some of the context may be handled by the search engine itself, and be reflected in the search results. For example, the search engine may keep track of a user's capture history, and may also cross-reference this capture history to conventional keyboard-based queries. In such cases, the search engine maintains and uses more state information about each individual user than do most conventional search engines, and each interaction with a search engine may be considered to extend over several searches and a longer period of time than is typical today.
[0127] Some of the context may be transmitted to the search engine in the search query (Section 3.3), and may possibly be stored at the engine so as to play a part in future queries. Lastly, some of the context will best be handled elsewhere, and so becomes a filter or secondary search applied to the results from the search engine.
selecting an image data capture instruction based on the location of the UE and the contextual data related to the location of the UE (e.g. based on the proximity to the target object, detecting being proximate to information on the document and detecting if text is present on the page, the capturing device is changed to a selected capturing mode associated with capturing or recognizing text on a page, which is taught in ¶ [544]-[549], [551]-[554].); and
[0050] FIG. 1B is a data flow diagram that illustrates the flow of information in one example of a suitable system. A capture device 155 captures presented information such as text, audio, video, GPS coordinates, user gestures, barcodes, and so on, from information source 150 and other sources, such as sources in wireless communication with the device (not shown). At step 160, the Information Saver component collects and stores information captured by capture device 155. At step 165, the system passes the information collected from the capture device to a capture information-processing component. The capture information processing component 165 is configured to detect the presence of rendered documents, extract text regions from documents, and analyze the document information to recognize document and text features, such as absolute and relative layout information, paragraph, line and word shadows or profiles, glyph-related features, and character encodings. In some examples, the capture information processing component may be configured to process types of data other than text, such as audio, compass data, GPS, acceleration, history, temperature, humidity, body heat, etc. In some examples, the capture information processing unit will accumulate information over time and composite the accumulated information, for example, to form larger and/or higher resolution images of the information source as the capture device captures or sends more information. In some examples, the Capture Information Processing component may leverage the context (see sections 13 and 14), such as previous information captured by a user, to guide the capture information processing, e.g. by limiting or expanding the amount of processing performed and guiding the assumptions about what is being processed. For example, if the system has recently identified that the user has captured information from a particular source, less processing may be needed subsequently in order to attain a similar level of certainly about the newly captured information, because a search within a limited space of possibilities can quickly result in a match, which can then be further confirmed if desired. The Capture Information Processing component may verify the identified information, such as by automatically confirming or rejecting predictions in the information based on tentative conclusions, or by leveraging a Concierge Service 170 (See Section 19.8), or by requesting user feedback. In step 175, the system stores the captured and processed information as part of the system history and context.
[0051] At step 180, the system performs a search based on the processed information and context (see sections 4.2.2, 13 and 14). In some examples, search results may be accumulated and correlated over time, e.g. intersecting search results based on subsets of the information captured over time to resolve ambiguities (such as multiple portions of recorded audio, audio from multiple frequency bands, multiple images, etc.). In some examples, the search results can be further verified by the Capture Information Processing component, e.g. based on the principle that the Image Processing component may perform additional analysis on the search results (or document information retrieved by the Document Manager component 185) and the captured information. For example, if the search component generated 10 possible results, the Capture Information Processing component may determine that 6 of those are very unlikely to match the search results, such as the pattern of vertical strokes in the text. At step 185, if a document was identified, a Document Manager component of the system may retrieve a representation of the document. At step 190, a Markup component of the system may compute and/or retrieve dynamic and/or static markup related to the text output from the capture information-processing step and/or the identified document or the retrieved representation of the document. For more information on static and dynamic markup, see section 5. In some examples, the Markup component produces markup based on identified text, as soon as it is recognized, in parallel with document identification.
[0544] In step 420, the system automatically changes the mode of the operation of the capture device 300 in response to the determination of step 410 that the device is in the presence of information to be captured by the device. In some cases, the system changes, alters, or modifies a current mode of operation of the mobile device 300. For example, the capture device may currently be in a default mode (no applications or features running, displaying the home screen to a user), and, upon detecting the device is proximate to information to be captured (such as a rendered document), the system automatically changes or transitions the mode of operation to the document capture mode described herein. In some cases, the system does not change the mode of operation and instead launches an application (or modifies current functionality of a running application) within a current mode of operation, wherein the launched application enables the capture device 300 to capture the proximate information. For example, the capture device 300 is running an music application that plays music for a user, and upon detecting the device is proximate to information (such as poster advertising a rock band's new album), the system automatically changes the current screen of the running application (such as a screen that can subsequently present review information and options to listen to, download, or purchase songs from the advertised album).
[0545] In step 430, the system captures the present information. As described herein, the system may perform an optical capture of the information (i.e., take an image of the information using), may perform an audio capture of the information (i.e., record the information being read aloud), or may perform other techniques to capture the information, utilizing other components (i.e., components that read an RFID tag, bar code, or other non-repeating dot pattern, capture geo-location or environmental information, or time/date information; and so on).
[0546] In step 440, the system performs an action associated with the captured information. As described herein, the system may perform a number of actions associated with the captured information, including presenting content associated with the captured information, identifying documents associated with the captured information, ordering and purchasing products associated with the captured information, and so on. In some cases, the system performs the action via display components 320 of the capture device 300. In some cases, the system performs the action via components remote from the capture device 300, such as an associated computing device, an associated mobile device, associated media presentation devices (e.g., stereos, mp3 players, televisions, displays, projectors), and so on.
[0547] In some examples, the system, after capturing information proximate to the capture device 300, does not perform an action at the time of capture, and instead stores information associated with the capture for later use by a user of the capture device. The system may determine, based on certain input received from a user by the capture device 300 (or from a lack of input received from the user) that the user desires to capture the information for later use. The system may then store information about the capture, such as an indication of the capture, in a database associated with the user or the capture device 300. In some cases, the system may build a timeline of captured information for a user or a capture device, enabling the user to recall and interact with the information and content they witnessed during a day, among other benefits. Further details regarding the storing of the information and the building of timelines are discussed herein.
[0548] In some examples, the system may change operation of a device or launch an application only after the system determines that information is within proximity to a device (step 410) and the system determines that the proximate information is associated with electronic or additional content (step 430). That is, the system may require routine 400 to perform steps 410 and 430 before performing step 420 and, later, step 440. When determining that proximate information is associated with digital or additional content, the system first attempts to verify interactive physical information is present, before proceeding to step 420 and changing operation of the device. This may prevent the system, in some cases, from changing operational modes of a capture device when information proximate to the capture device is not associated with additional or alternative information, supplemental content, or performable actions, among other benefits.
[0549] Thus, the system enables a user of a capture device to automatically capture information that may be of interest to the user, among other benefits. In some cases, the system anticipates a user intention to capture information, modifying the operation of the user's device to an information capture mode, which may ease the effort involved in quickly and effectively capturing. That is, the system stages or readies a mobile device, such as a mobile phone that provides many different functions (e.g., voice communications, messaging, music playback, taking pictures, text captures, and so on), to capture information when information is available to be captured, among other benefits.
[0551] Referring to FIG. 5, a flow diagram illustrating a routine 500 for performing a capture of text from a rendered document using a document aware capture device is shown. In step 510, the system determines a capture device is in the presence of or proximate to a rendered document. In some cases, the system may detect a paper, printed, or painted document, a displayed document, an object having text printed or displayed on an outer surface (e.g., a real estate sign, a product for purchase, and so on), or other objects that present text visible to a user. The system may detect the presence of information using some or all of the detection components described herein.
[0552] As discussed with respect to step 410, the system may utilize the detection component 330 to determine the device is proximate to information. In some cases, the detection component 330 may facilitate detecting the presence of text or a rendered document via an imaging or other capture component 310 of the capture device. In some cases, the detection component 330 may facilitate detecting the presence of text or a rendered document via components that measure distances between the capture device and target objects (such as rendered documents, displays of information, and so on) and determine the capture device is within a certain proximity to a target object associated with a user intention to capture the text from or about that target object. In some cases, the detection component may facilitate detecting the presence of text or a rendered document via components that measure the orientation of the device relative to a target object and determine that the capture device is in a certain position or orientation, possibly for a certain duration of time, associated with a user intention to capture the text from or about that target object. Further details and aspects regarding determining that a capture device is in the presence of text or a rendered document are described herein with respect to FIGS. 6A-6C.
[0553] In step 520, the system automatically changes the mode of operation of the capture device in response to the determination of step 510 that the device is in the presence of text to be captured by the device. In some cases, the system changes, alters, or modifies a current mode of operation of the capture device 300. For example, the capture device may currently be in a default mode (e.g., no applications or features running, displaying the home screen to a user), and, upon detecting the device is proximate to a rendered document to be captured, the system automatically changes the mode of operation to the document capture mode described herein. In some cases, the system does not change the mode of operation and instead executes software (e.g., launches an application or modifies current functionality of a running application) within a current mode of operation, wherein the executed software enables the capture device 300 to capture the proximate text. As one example, a web browser may be running on the capture device, and upon detecting the device is proximate to a rendered document, the system automatically points the browser to a web address for a search engine, anticipating the user will subsequently desire to search for text presented by the rendered document.
[0554] In step 530, the system captures text from the rendered document. As described herein, the system may perform an optical capture of the text (i.e., take an image of the text and possibly perform OCR or other techniques to identify text within the captured image), may perform an audio capture of the information (i.e., record the text being read aloud and recognize the text using speech recognition techniques), and so on.
[0559] In some examples, the system may utilize techniques that record the behavior of a user or the capture device to predict that information is present or has been consumed, without actually finding or detecting the information. The system may analyze data from some or all components within a capture device to make a prediction that information is present. As an example, the system may measure the altitude, location and acceleration of a capture device, and determine that the capture device is at a certain location going in a certain direction, and capture any information associated with that combination of factors to make the prediction.
[0560] In some examples, the system utilizes the optical or imaging components of a capture device in order to detect a capture device is proximate to information to be captured. Referring to FIG. 6A, a flow diagram illustrating a routine 600 for automatically changing operation of a capture device upon determining information is proximate to the device is shown.
[0561] In step 601, the system, using a detection component 320 such as an imaging component, takes one or more images of the environment surrounding the device. The system may sample images within the view of an imaging component, or may act in response to receiving a trigger that indicates the capture device is or may be within the proximity of information to be captured. That is, in response to a trigger the system may activate an imaging component out of a sleep or off state to sample or take images.
[0562] Examples of triggers received alone or in various combinations include: [0563] Receiving information from components of a capture device used to detect proximity to information, including image, distance, proximity, and/or orientation information; [0564] Detecting movement of the device that may indicate a capture device is proximate to information; [0565] Detecting patterns of use of the device that may indicate a capture device is proximate to information; [0566] Detecting certain light patterns within view of an imaging component; [0567] Detecting certain features of target objects, including black to white transitions (indicating text), color transitions (indicating printed images), and so on; [0568] Receiving vocal or tactile commands from a user; [0569] Receiving commands from a user via a user interface of the capture device that indicate the user is attempting to capture information but has not been successful; [0570] Being proximate to documents and other objects having attached RFID tags; and so on.
[0571] In step 603, the system determines whether the capture device is in the presence of text. In some cases, the system may make a determination based on one image. In some cases, the system may make the determination based on two or more images. In some cases, the system may poll a number or range of images taken by an imaging component, and when the number or range of images that include text satisfy a certain threshold number associated with a positive determination, make a determination that the sample images include text. For example, orientation, lighting, or other factors may impair the imaging components ability to take clear or accurate images of a target object, and the system may therefore take one of every ten images for two seconds in order to make the determination.
[0572] When the system determines the capture device is in the presence of text, routine 600 proceeds to step 605, and the system automatically changes operation of the capture device to a mode of operation associated with capturing information, else routine 600 proceeds back to step 601 or ends. In step 605, the system automatically changes the operation of the captured device, as described with respect to step 420 of FIG. 4 or step 520 of FIG. 5.
causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture (e.g. the capture mode is used to capture the present information that is recognized text that can be in a predetermined format when applying the OCR function, which is taught in ¶ [241]-[245], [258], [264], [545] and [554]. The predetermined format can be a font that the OCR is programmed to recognize when performing the function.).
[0240] 9.2. Text Recognition
[0241] Based on the extracted location information, the system can attempt to recognize the text or features of the text within the captured image. For example, the system may send the text to an OCR component or generate a signature based on identified features of the text (e.g., patterns of ascenders and/or descenders within the text). Prior to performing text recognition, the system may normalize or canonicalize text by, for example, converting all italicized or bold text to a standard formatting.
[0242] The Text Recognition process may rely on several features to recognize characteristics of the text or generate a signature for a rendered document, such as glyph features (e.g., enclosed spaces, vertical and horizontal strokes, etc.), punctuation, capitalization, characters spaces, line features, paragraph features, column features, heading features, caption features, key/legend features, logo features, text-on-graphic features, etc. Additionally, word features may assist in the text recognition process, such as word spacing and densities. For example, the system may use information associated with spaces between words printed on a document, such as distances between spaces (horizontally, vertically, orthogonally, and so on), the width of the spaces, and so on. The system may further incorporate knowledge about line breaks into the analysis. For example, when line breaks are known, the system may rely on the vertical alignment of word positions whereas when line breaks are unknown, the system may rely on proximate sequences of relative word lengths. As another example, the system may use information associated with densities of characters, such as relative densities between characters (horizontally, vertically, orthogonally, and so on), relative densities between grouped pairs of characters, or absolute density information. Certain features may be invariant to font, font size, etc., such as point and line symmetries (e.g., auto-correlations within glyphs, around points and/or lines). The system may dynamically select which features to analyze within a captured image. For example, in the presence of optical and motion blur, the system may use less-detailed aspects of the text, such as relative word widths. In some examples, the system may leverage unique n-grams by determining whether unknown or infrequent n-grams are noise, or high-signal information (misspellings, email addresses, URLs, etc.) based on, for example, certainty of characters deviating from common n-grams, length of deviation, matching regular expressions, (e.g. for email addresses and URLs), and so on.
[0243] The system may use resources external to a rendered document to recognize text within the rendered document, such as knowledge pertaining to the approximate number of glyphs within a word, dictionaries (e.g., word frequency dictionaries), grammar and punctuation rules, probabilities of finding particular word-grams and character-grams within a corpus, regular expressions for matching various strings, such as email addresses, URL, and so on. Furthermore, the system may use resources such as DNS servers, address books, and phone books to verify recognized text, such as URLS, emails addresses, and telephone numbers. As another example, the system may use font matrices to assist in the recognition and verification of various glyphs. Unrecognized characters in a given font may be compared to recognized characters in the same font to assist in their recognition based on the relationship between the unrecognized and recognized characters reflected in a font matrix. By way of example, an unrecognized "d" may be recognized as a "d" based on a recognized "c" and "I" if a font matrix indicates that the representation of a "d" is similar to the combination of "c" and "I."
[0244] The system may use the recognized text or features to identify the document depicted in the captured image among the documents in a document corpus. The amount and type of information used to identify may vary based on any number of factors, such as the type of document, the size of the corpus, the document contents, etc. For example, a sequence of 5 or 6 words within a captured image or the relative position of spaces between words may uniquely identify a corresponding document within a relatively large corpus. In some examples, the system may employ a conversion table to determine the probability that information about certain features, or the combination of information pertaining to certain features, will uniquely identify a document. For example, the conversation table may indicate that a 5 word sequence of words has the same probability of uniquely identifying a document as two different 3 word sequences, the ascender/descender pattern of 2 consecutive lines, and so on. In some examples, the system may automatically accumulate or "stitch" together captured images to, for example, generate a composite image of a rendered document that is more likely to uniquely identify a corresponding document than the captured images individually.
[0245] In some examples, the Text Recognition process may influence the capture of information. For example, if the Text is recognized as out of focus or incomplete, the system can adjust the focus of the camera of the capture device or prompt the user to reposition or adjust the capture device. Various techniques that the system may employ to recognize text are described in further detail below.
[0258] Conventional OCR uses knowledge about fonts, letter structure and shape to attempt to determine characters in scanned text. Examples of the present invention are different; they employ a variety of methods that use the rendered text itself to assist in the recognition process. These use characters (or tokens) to "recognize each other." One way to refer to such self-recognition is "template matching," and is similar to "convolution." To perform such self-recognition, the system slides a copy of the text horizontally over itself and notes matching regions of the text images. Prior template matching and convolution techniques encompass a variety of related techniques. These techniques to tokenize and/or recognize characters/tokens will be collectively referred to herein as "autocorrelation," as the text is used to correlate with its own component parts when matching characters/tokens.
[0263] 9.2.6 Font/Character "Self-Recognition"
[0264] Conventional template-matching OCR compares scanned images to a library of character images. In essence, the alphabet is stored for each font and newly scanned images are compared to the stored images to find matching characters. The process generally has an initial delay until the correct font has been identified. After that, the OCR process is relatively quick because most documents use the same font throughout. Subsequent images can therefore be converted to text by comparison with the most recently identified font library.
[0545] In step 430, the system captures the present information. As described herein, the system may perform an optical capture of the information (i.e., take an image of the information using), may perform an audio capture of the information (i.e., record the information being read aloud), or may perform other techniques to capture the information, utilizing other components (i.e., components that read an RFID tag, bar code, or other non-repeating dot pattern, capture geo-location or environmental information, or time/date information; and so on).
[0554] In step 530, the system captures text from the rendered document. As described herein, the system may perform an optical capture of the text (i.e., take an image of the text and possibly perform OCR or other techniques to identify text within the captured image), may perform an audio capture of the information (i.e., record the text being read aloud and recognize the text using speech recognition techniques), and so on.
However, King fails to specifically teach the features of wherein the contextual data is determined from telecommunications network data associated with the UE and a user profile maintained by the telecommunications network; causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture.
However, an aspect of this is well known in the art as evidenced by Ramanujapuram. Similar to the primary reference, Ramanujapuram discloses receiving contextual correlation data to perform further operation (same field of endeavor or reasonably pertinent to the problem).
Ramanujapuram discloses wherein the contextual data is determined from telecommunications network data associated with the UE and a user profile maintained by the telecommunications network (e.g. the system discloses determining contextual data from the location of where the mobile device captures an image or the location of the mobile device, which is taught in ¶ [15]. A user profile associated with the mobile device that captures an image is also considered contextual data. Both types of contextual data are taught in ¶ [46]-[48] and [55]. The Server uses a network, such as a LAN or WAN, or communicates using CDMA or GSM over the network, which is taught in ¶ [23]-[26]. The server stores information, such as a profile on the user, that is used for comparison to other data and to gather further contextual information, which is taught in ¶ [50]-[56].);
[0015] Image data may comprise one or more images near a same location. The images may be taken by many different users with different cameras or other devices. The image(s) may include non-text information, such as logos, landmarks, or the like. In addition, or alternatively, the image(s) may include text information, such as character strings on a sign, a billboard, or the like. Contextual data may include the location where each image was taken, a user profile associated with a mobile device that took one or more of the images, or the like. In addition, or alternatively, contextual data may include information known about the location, such as merchants, buildings, street names, information about actions performed by one or more users near the location, or the like. An image may be analyzed to determine a histogram, identifying characteristics of the image. Some object recognition may be performed if image resolution permits. An image may also be analyzed to locate and recognize characters within the image. The image analyses may be evaluated relative to the analyses of other images by the same user and/or by other users to determine likely contents of an image. Similarly, contextual data may be evaluated to determine or revise the likely contents of the image and/or the likely information desired about an image. In one embodiment, determined, or recognized key words, categories, or other information may be submitted to a data search system to retrieve search results, web pages from specifically recognized uniform resource locators (URLs), phone directory information, advertisements, or other results. In addition, or alternatively, the results may be filtered, prioritized, categorized, or otherwise further processed.
[0023] Each client device within client devices 102-104 may include a browser application that is configured to send, receive, and display web pages, and the like. The browser application may be configured to receive and display graphics, text, multimedia, and the like, employing virtually any web based language, including, but not limited to Standard Generalized Markup Language (SMGL), such as HyperText Markup Language (HTML), extensible markup language (XML), a Handheld Device Markup Language (HDML), such as Wireless Markup Language (WML), WMLScript, JavaScript, and the like. Client devices 102-104 may further include a messaging application configured to send and/or receive a message to/from another computing device employing another mechanism, including, but not limited to instant messaging (IM), email, Short Message Service (SMS), Multimedia Message Service (MMS), internet relay chat (IRC), mIRC, Jabber, and the like.
[0024] Network 105 is configured to couple one computing device to another computing device to enable them to communicate. Network 105 is enabled to employ any form of computer readable media for communicating information from one electronic device to another. Also, network 105 may include a wireless interface, and/or a wired interface, such as the Internet, in addition to local area networks (LANs), wide area networks (WANs), direct connections, such as through a universal serial bus (USB) port, other forms of computer-readable media, or any combination thereof. On an interconnected set of LANs, including those based on differing architectures and protocols, a router acts as a link between LANs, enabling messages to be sent from one to another. Also, communication links within LANs typically include twisted wire pair or coaxial cable, while communication links between networks may utilize analog telephone lines, full or fractional dedicated digital lines including T1, T2, T3, and T4, Digital Signal level 3 (DS3), Optical Carrier 3 (OC3), OC12, OC48, Asynchronous Transfer Mode (ATM), Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines (DSLs), wireless links including satellite links, or other communications links known to those skilled in the art. Furthermore, remote computers and other related electronic devices could be remotely connected to either LANs or WANs via a modem and temporary telephone link. Network 105 is constructed for use with various communication protocols and technologies, including transmission control protocol/internet protocol (TCP/IP), user datagram protocol (UDP), a wireless application protocol (WAP), global system for mobile communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), general packet radio service (GPRS), ultra wide band (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMax), and the like. In essence, network 105 includes any communication method by which information may travel between client devices 102-104, and/or server 106.
[0025] The media used to transmit information in communication links as described above generally includes any media that can be accessed by a computing device. Computer-readable media may include computer storage media, wired and wireless communication media, or any combination thereof. Additionally, computer-readable media typically embodies computer-readable instructions, data structures, program modules, or other data. Such data can be communicated through communication media in a modulated data signal such as a carrier wave, data signal, or other transport mechanism and includes any information delivery media. The terms "modulated data signal," and "carrier-wave signal" includes a signal that has one or more of its characteristics set or changed in such a manner as to encode information, instructions, data, and the like, in the signal. By way of example, communication media includes wireless media such as fluids or space for acoustic, RF, infrared, and other wireless signals, and wired media such as twisted pair, coaxial cable, fiber optics, wave guides, and other wired media.
[0026] Server 106 may comprise multiple computing devices or a single computing device. Server 106 may provide image analysis services, such as determining histograms, performing OCR, comparing images to previously stored images, determining information about images, performing database operations, performing searches for additional information, storing information about images, tracking user behaviors, or the like. Server 106 may also provide content and/or other services such as web sites, online journals (e.g., blogs), photos, reviews, online services such as messaging, search, news, shopping, advertising, and/or the like. Server 106 may further provide administrative services, such as creation, modification, and management of relationships between network resources, such as web pages, or the like. Briefly, server 106 may include any computing device capable of connecting to network 105 and may manage services for a network user, such as a user of at least one of client devices 102-104. Devices that may operate as server 106 include dedicated server devices, personal computers, desktop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, and the like. Server 106 and/or any of clients 102-104 may be implemented on one or more computing devices, such as a client described with regard to FIG. 2.
Illustrative Logic
[0046] FIG. 4 illustrates an example flow diagram 400 for processing an information request based on image data. In one example embodiment, an information request may be a search request for additional information based on data determined from an image. At an operation 402, one or more data services establish one or more databases of contextual correlation data, such as location related information, user related information, time related data, or other data that indicates a context of images or used to correlate images. Examples of context correlation information may include geographic location data, advertising information, merchant information, communication node information, weather information, traffic information, or other information. Geographic location data may include GPS data, zip codes, street names, street addresses, building names, landmarks, or the like. Advertising information may include locations, content, and other information regarding billboards, painted wall signs, street-level signs, storefront signs, or the like. Advertising information may also include non-commercial signage. Merchant information may include merchant names, addresses, phone numbers, trademarks, logos, URLs, email addresses, products offered, inventory information, prices, or the like. Communication node information may include cellular tower locations, wifi hotspot locations, network address information, communication capability information, or the like. Real-time data may also be maintained through the database(s) of location related information. For example, data services may provide current weather conditions, traffic conditions, event activities, or the like.
[0047] Similarly, in this embodiment, the data service(s) track clients' online behaviors at an operation 404. Client user behaviors are generally associated with locations of the behaviors. With user permission, the data service(s) may track messaging, searches performed, URLs selected, purchases made, or the like. The data service(s) may also determine other parameters related to the online behaviors. For example, a data service may determine that a number of client users know each other based on message exchanges, may determine interests that a client user may have, or the like. A data service may also determine indirect relationships that comprise a user's social network.
[0048] At an operation 406, a client user captures image data with a mobile client device. The captured image data is generally associated with one or more context correlation data elements, such as location and/or other data discussed above. For example the user may take a photograph of a billboard with a cellular phone. The image and/or location data may be associated with a time stamp, a cell tower location, a wifi network node address, or other data. The billboard may include a merchant name, a logo, a phone number, a URL, or other content. The client device may perform further processing locally, or may communicate the image and/or location data to a server. For example, the client device may be capable of performing some image histogram analysis, image fingerprinting analysis, or the like. The client device may perform such analyses and communicate the results to the server. However, in many cases, client devices, such as cell phone, will have limited image processing capability. Such devices will generally send raw or compressed image data and location data to the server.
[0049] At an optional operation 408, the server may receive a client identifier for the client that captured the image, such as a phone number, a mobile identification number, user identifier, or the like. The server may use the client identifier to access previously stored information associated with the client identifier, such as prior images submitted, prior locations submitted, client device capabilities, user behaviors, aggregated information related to the client identifier, or the like. The server, or the client device, may also receive other information associated with the location of the image. As discussed above, such information may be pre-established location data or may include real-time data related to the location. For example, the server may access or receive merchant information that is associated with a location that is near the image capture location. This merchant information may identify merchants and their distance from the image capture location. Similarly, the server may access or receive other context data related to the image capture location, such as a street name, zip code, weather conditions, traffic conditions, or the like.
[0050] At an operation 410, the server or the client device analyzes the captured image to determine an image histogram. The image histogram generally identifies color parameters and other characteristics of the image, so that images can be readily compared. In an idealized situation, where everybody took pictures with the same camera and at the same location, then simple image comparison by comparing corresponding image pixels would give an exact measure of whether the images are taken of the same object. But this kind of simple measure generally is generally not sufficient.
[0051] In a simple case, two images can be compared for similarity by computing a histogram of the colors found in the image. The metric used for determining whether two images are similar is just a matter of comparing the fraction of pixels in each of (dozens) of different colors. This metric is useful because it generally works no matter how the cameras are rotated, and tends to be immune to scale and transformations. For example, it generally works well for determining which billboard is being seen, since the colors are unique between different companies. It may be less effective to determine which of several gray-stone bank buildings one is standing in front of in a captured image.
[0052] More sophisticated approaches, such as those based on salient points, are generally more robust. In this approach, an operator is run across the image that identifies points in the image that are especially salient, or that pop out no matter what the orientation. These are often image features such as corners. Once the salient points are identified they are characterized by any number of measures including color, local texture, and orientation. Two images are generally judged to be similar if a high percentage of salient points in each image can be matched, and they have the right alignment. Further details of example such techniques are described in "Object Recognition from Local Scale-Invariant Features," by David G. Lowe.
[0053] At a decision operation 412, the server may check whether a previously stored histogram (or salient points, etc.) matches a histogram (or salient points, etc.) of the newly received image within a predefined matching threshold. In this example embodiment, the server selects histogram data from prior received images, for comparison against the histogram results of the captured image. To narrow down the number of comparisons, the server may select prior image analysis data based on a context correlation element, such as a predefined radius from the location of the captured image. In one embodiment, the server may select a "best" prior image, or a top number of highest resolution images within a certain distance of the currently captured image. In addition, or alternatively, the server may use other criteria to filter, prioritize, or otherwise select prior histogram data. Other criteria may include histogram characteristics within certain ranges of the captured image, resolution of images in various databases, only prior image analysis data that are associated with a certain set of user identifiers or mobile device identifiers, prior image analysis data that are within a certain number of known blocks of the image capture location, or the like.
[0054] Once prior image analysis data is selected, the server compares the selected data against the image analysis results for the captured image. Another image may have a very similar image histogram, even though the other image may have different zoom, light, or other image parameters. For example, one image may capture a billboard from a certain distance and at a certain time of day. Another image may capture the same billboard from a different distance and at a different time of day. The image histogram of each image can be compared to determine whether they capture the same billboard. Alternatively, a part of an image may have a histogram that is very similar to the histogram of a part of another image. The similar parts may be detected and compared. Similar parts may correspond to logos or other non-character symbols. Each histogram can function as a digital fingerprint to identify an image. Histograms that are the same, or statistically within a predefined threshold, may be considered equivalent. These comparisons help identify the content of the image.
[0055] If user profile data is associated with the captured image and at least one of the prior images, prior user profile data may also be selected and compared with that of the currently captured image. This may help resolve uncertainty about resolve possible logos, provide prior merchant interaction information, or provide other additional context information regarding the captured image.
[0056] In general, if a match is found, the server may accesses any of the above, or other descriptive information, search terms, or other information related to the prior images and/or information related to context correlation data used to relate to the captured image. In one embodiment, the context correlation data is the location at which the current and prior images were captured. In some embodiments, the server may perform a search, perform other processing, and/or immediately return image content information, such as previously stored image-related information and/or previously stored location-related information to the client device, at an operation 414. The image content information may identify only contents of the image or may comprise information about the contents of the image. In some embodiments, the server may end its image processing operations at this point, or may continue with further processing, as shown, to obtain additional image-related information and/or location-related information.
causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture (e.g. after the client device or server retrieves the image data and an image histogram to determine a match or not a match, the system can cause the client device to perform capturing of data to identify an area within an entire image to be further scanned or recognized. The area can contain characters that reflects a western alphabet that is to be scanned for further comparison, which is taught in ¶ [57].).
[0057] Similar to the histogram evaluation, at an operation 416, the server or the client device may perform an optical character recognition (OCR) analysis to identify characters within the image. In this embodiment, the server optically recognizes characters that appear in a captured image of a scene. Such images are generally photographs rather than simple black and white pages, so more involved techniques, such as those used for video OCR are useful. One process consists of two steps: detection and recognition. Detection generally identifies likely regions of the image that include textures with the statistical behavior that indicates it could be text. For example, western alphabets have many horizontal and vertical lines in close proximity. Once likely regions are identified, pattern recognition methods, perhaps with a language model to constrain the possible words, are used to determine the most likely text. Details of example techniques for OCR is described in "Feature Extraction Approaches For Optical Character Recognition," by Roman Yampolskiy, including a chapter by Rainer Lienhart.
Therefore, in view of Ramanujapuram, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein the contextual data is determined from telecommunications network data associated with the UE and a user profile maintained by the telecommunications network; causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture, incorporated in the device of King, in order to allow for contextual data and the mobile device location information associated with a captured image to further recognize an area within an image using a mobile device , which improves system recognition of desired data related to the things a user sees (as stated in Ramanujapuram ¶ [02] and [03]).
Re claim 2: The method of claim 1, wherein the image data capture instruction is to capture data in the first area of an image (e.g. the present information is captured by the capturing device, which is taught in ¶ [545] above. An overlay can occur to appear on a portion of a document to be captured that may be associated with a markup portion, which is taught in ¶ [161].).
[0160] 5.7 Image Enhancements and Compensation
[0161] In some examples, the system provides an enhanced view of a document by overlaying a display showing the document with various display elements. The enhanced view may overlay a real-time image of a portion of the document within a capture device's field of view with various display elements associated with the document, or may present and overlay associated electronic versions or images of the document retrieved or generated by the system with various display elements associated with the document. In some examples, the system provides document interaction techniques that compensate for various hardware configurations of capture devices, such as the locations of cameras and other imaging components with respect to the display or a center point of a document, the size of a capture device and/or the display of the capture device. The system may provide document interaction techniques that enables user to navigate paper documents, identify markup associated with documents, zoom in or out of paper documents, and so on. For example, the system may respond to gestures made by a user of a capture device, such as gestures that move a capture device in various directions relative to a paper document. Thus, the system enables users to interact with paper documents, target objects, and other displays of information using multi-function mobile devices not necessarily manufactured only to interact with information or capture information from the environment around the device, among other benefits.
Re claim 3: King discloses the method of claim 1, wherein the image data capture instruction is to capture a specific character string in an image (e.g. when a document is detected with text, a specific OCR function can be used to detect specific characters, which is taught in ¶ [550]-[556].).
Document Awareness of a Capture Device
[0550] Although people interact with a variety of information sources, one large subset of these sources is rendered documents, such as printed documents, documents provided by a presentation layer of a computing device, a television, a radio, a media player, and so on. People consume an enormous amount of information by reading text from rendered documents such as books, magazines, newspapers, billboards, maps, signs, displayed web pages and blogs, movies, videos, TV shows, radio programs, receipts, bills, mail, chalkboards, whiteboards, presentations, and so on. The system, therefore, facilitates the detection of available information, such as text on a rendered document, in proximity to a capture device.
[0551] Referring to FIG. 5, a flow diagram illustrating a routine 500 for performing a capture of text from a rendered document using a document aware capture device is shown. In step 510, the system determines a capture device is in the presence of or proximate to a rendered document. In some cases, the system may detect a paper, printed, or painted document, a displayed document, an object having text printed or displayed on an outer surface (e.g., a real estate sign, a product for purchase, and so on), or other objects that present text visible to a user. The system may detect the presence of information using some or all of the detection components described herein.
[0552] As discussed with respect to step 410, the system may utilize the detection component 330 to determine the device is proximate to information. In some cases, the detection component 330 may facilitate detecting the presence of text or a rendered document via an imaging or other capture component 310 of the capture device. In some cases, the detection component 330 may facilitate detecting the presence of text or a rendered document via components that measure distances between the capture device and target objects (such as rendered documents, displays of information, and so on) and determine the capture device is within a certain proximity to a target object associated with a user intention to capture the text from or about that target object. In some cases, the detection component may facilitate detecting the presence of text or a rendered document via components that measure the orientation of the device relative to a target object and determine that the capture device is in a certain position or orientation, possibly for a certain duration of time, associated with a user intention to capture the text from or about that target object. Further details and aspects regarding determining that a capture device is in the presence of text or a rendered document are described herein with respect to FIGS. 6A-6C.
[0553] In step 520, the system automatically changes the mode of operation of the capture device in response to the determination of step 510 that the device is in the presence of text to be captured by the device. In some cases, the system changes, alters, or modifies a current mode of operation of the capture device 300. For example, the capture device may currently be in a default mode (e.g., no applications or features running, displaying the home screen to a user), and, upon detecting the device is proximate to a rendered document to be captured, the system automatically changes the mode of operation to the document capture mode described herein. In some cases, the system does not change the mode of operation and instead executes software (e.g., launches an application or modifies current functionality of a running application) within a current mode of operation, wherein the executed software enables the capture device 300 to capture the proximate text. As one example, a web browser may be running on the capture device, and upon detecting the device is proximate to a rendered document, the system automatically points the browser to a web address for a search engine, anticipating the user will subsequently desire to search for text presented by the rendered document.
[0554] In step 530, the system captures text from the rendered document. As described herein, the system may perform an optical capture of the text (i.e., take an image of the text and possibly perform OCR or other techniques to identify text within the captured image), may perform an audio capture of the information (i.e., record the text being read aloud and recognize the text using speech recognition techniques), and so on.
[0555] In step 540, the system identifies the rendered document from the captured text. As described herein, in many cases the rendered document has an electronic counterpart, and the system is able to identify the electronic counterpart of a rendered document based on the captured text. The system may utilize the identity of the rendered document to decide what actions to perform, to provide context for various actions to perform, to track reader usage and develop models of the reading habits of users, and so on.
[0556] As discussed with respect to step 440, the system, in step 550, performs an action associated with the captured text and/or the identified rendered document. As described herein, the system may perform a number of actions associated with the captured text and/or the rendered document, including presenting content associated with the captured text and/or rendered document, identifying other documents associated with the captured text and/or rendered document, purchasing products associated with the captured text and/or rendered documents, and so on. In some cases, the system performs the action via display components 320 of the capture device 300. In some cases, the system performs the action via components remote from the capture device 300, such as an associated computing device, an associated mobile device, associated media presentation devices (e.g., stereos, mp3 players, televisions, displays, projectors), and so on.
Re claim 4: King discloses the method of claim 1, wherein the contextual data includes a format of an anticipated image to be captured at the location of the UE (e.g. when capturing an image and further providing context that the image contains some type of text, the system can determine that a user desires to acquire the text from the image. This information is fed to the system to allow for the capturing device to perform an OCR operation or a search of the text on the page, which is taught in ¶ [550]-[556] above. A character is considered as a format of the data instead of a picture image.).
Re claim 5: King discloses the method of claim 1, wherein the contextual data includes a character string within an anticipated image to be captured at the location of the UE (e.g. when capturing an image, detecting a proximity to an image and further providing context that the image contains some type of text, the system can determine that a user desires to acquire the text from the image. This information is fed to the system to allow for the capturing device to perform an OCR operation or a search of the text on the page, which is taught in ¶ [550]-[556] above.).
Re claim 6: King discloses the method of claim 1, further comprising receiving an indication that an image capture device of the UE is accessed (e.g. when a mode of the device is from a default to an image capturing mode, this is considered as indication to the system that an image capturing function of the device is accessed, which is taught in ¶ [550]-[556] above.).
Re claim 7: King discloses the method of claim 1, further comprising updating a user profile based on UE activity after executing the image data capture instruction (e.g. a storage profile associated with the user is updated after capturing information in order to indicate information of a capture. This can build a timeline of the user’s actions for capture, which is taught in ¶ [547] and [548].).
[0547] In some examples, the system, after capturing information proximate to the capture device 300, does not perform an action at the time of capture, and instead stores information associated with the capture for later use by a user of the capture device. The system may determine, based on certain input received from a user by the capture device 300 (or from a lack of input received from the user) that the user desires to capture the information for later use. The system may then store information about the capture, such as an indication of the capture, in a database associated with the user or the capture device 300. In some cases, the system may build a timeline of captured information for a user or a capture device, enabling the user to recall and interact with the information and content they witnessed during a day, among other benefits. Further details regarding the storing of the information and the building of timelines are discussed herein.
[0548] In some examples, the system may change operation of a device or launch an application only after the system determines that information is within proximity to a device (step 410) and the system determines that the proximate information is associated with electronic or additional content (step 430). That is, the system may require routine 400 to perform steps 410 and 430 before performing step 420 and, later, step 440. When determining that proximate information is associated with digital or additional content, the system first attempts to verify interactive physical information is present, before proceeding to step 420 and changing operation of the device. This may prevent the system, in some cases, from changing operational modes of a capture device when information proximate to the capture device is not associated with additional or alternative information, supplemental content, or performable actions, among other benefits.
Re claim 9: King discloses a system for context aware digital vision in a network, the system comprising:
one or more processors; and one or more computer-readable media storing computer-usable instructions that, when executed by the one or more processors, cause the one or more processors to:
receive a location of a user equipment (UE) (e.g. a location proximate to information of the capture device is detected, which is taught in ¶ [543] above.);
determine contextual data related to the location of the UE (e.g. the proximity of the location of the capture device to a target object is determined, which is taught in ¶ [543] above.);
select an image data capture instruction based on the location of the UE and the contextual data related to the location of the UE (e.g. based on the proximity to the target object and detecting being proximate to information, the capturing device is changed to a selected capturing mode, which is taught in ¶ [544] above.); and
causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture (e.g. the capture mode is used to capture the present information that is recognized text that can be in a predetermined format when applying the OCR function, which is taught in ¶ [241]-[245], [258], [264], [545] and [554] above. The predetermined format can be a font that the OCR is programmed to recognize when performing the function.).
However, King fails to specifically teach the features of causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture.
However, an aspect of this is well known in the art as evidenced by Ramanujapuram. Similar to the primary reference, Ramanujapuram discloses receiving contextual correlation data to perform further operation (same field of endeavor or reasonably pertinent to the problem).
Ramanujapuram discloses causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture (e.g. after the client device or server retrieves the image data and an image histogram to determine a match or not a match, the system can cause the client device to perform capturing of data to identify an area within an entire image to be further scanned or recognized. The area can contain characters that reflects a western alphabet that is to be scanned for further comparison, which is taught in ¶ [57] above.).
Therefore, in view of Ramanujapuram, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture, incorporated in the device of King, in order to allow for contextual data and the mobile device location information associated with a captured image to further recognize an area within an image using a mobile device , which improves system recognition of desired data related to the things a user sees (as stated in Ramanujapuram ¶ [02] and [03]).
Re claim 10: King discloses the system of claim 9, wherein the image data capture instruction is to capture data in the first area of an image (e.g. the present information is captured by the capturing device, which is taught in ¶ [545] above. An overlay can occur to appear on a portion of a document to be captured that may be associated with a markup portion, which is taught in ¶ [161] above.).
Re claim 11: King discloses the system of claim 9, wherein the image data capture instruction is to capture a specific character string in an image (e.g. when a document is detected with text, a specific OCR function can be used to detect specific characters, which is taught in ¶ [550]-[556] above.).
Re claim 12: King discloses the system of claim 9, wherein the contextual data includes a format of an anticipated image to be captured at the location of the UE (e.g. when capturing an image and further providing context that the image contains some type of text, the system can determine that a user desires to acquire the text from the image. This information is fed to the system to allow for the capturing device to perform an OCR operation or a search of the text on the page, which is taught in ¶ [550]-[556] above. A character is considered as a format of the data instead of a picture image.).
Re claim 13: King discloses the system of claim 9, wherein the contextual data includes a character string within an anticipated image to be captured at the location of the UE (e.g. when capturing an image, detecting a proximity to an image and further providing context that the image contains some type of text, the system can determine that a user desires to acquire the text from the image. This information is fed to the system to allow for the capturing device to perform an OCR operation or a search of the text on the page, which is taught in ¶ [550]-[556] above.).
Re claim 14: King discloses the system of claim 9, wherein the one or more processors is further configured to receive an indication that an image capture device of the UE is accessed (e.g. when a mode of the device is from a default to an image capturing mode, this is considered as indication to the system that an image capturing function of the device is accessed, which is taught in ¶ [550]-[556] above.).
Re claim 15: King discloses the system of claim 9, wherein the one or more processors is further configured to update a user profile based on UE activity after executing the image data capture instruction (e.g. a storage profile associated with the user is updated after capturing information in order to indicate information of a capture. This can build a timeline of the user’s actions for capture, which is taught in ¶ [547] and [548] above.).
Re claim 17: King discloses a non-transitory computer storage media storing computer-usable instructions that, when used by one or more processors, cause the one or more processors to:
receive a location of a user equipment (UE) (e.g. a location proximate to information of the capture device is detected, which is taught in ¶ [543] above.);
determine contextual data related to the location of the UE (e.g. the proximity of the location of the capture device to a target object is determined, which is taught in ¶ [543] above.);
select an image data capture instruction based on the location of the UE and the contextual data related to the location of the UE (e.g. based on the proximity to the target object and detecting being proximate to information, the capturing device is changed to a selected capturing mode, which is taught in ¶ [544] above.); and
causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture (e.g. the capture mode is used to capture the present information that is recognized text that can be in a predetermined format when applying the OCR function, which is taught in ¶ [241]-[245], [258], [264], [545] and [554]. The predetermined format can be a font that the OCR is programmed to recognize when performing the function.).
However, King fails to specifically teach the features of causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture.
However, an aspect of this is well known in the art as evidenced by Ramanujapuram. Similar to the primary reference, Ramanujapuram discloses receiving contextual correlation data to perform further operation (same field of endeavor or reasonably pertinent to the problem).
Ramanujapuram discloses causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture (e.g. after the client device or server retrieves the image data and an image histogram to determine a match or not a match, the system can cause the client device to perform capturing of data to identify an area within an entire image to be further scanned or recognized. The area can contain characters that reflects a western alphabet that is to be scanned for further comparison, which is taught in ¶ [57] above.).
Therefore, in view of Ramanujapuram, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of causing the UE to execute the image data capture instruction, wherein the image data capture instruction identifies at least one of (i) a first area of an image to be scanned rather than an entire image or (ii) a predetermined character-string format to be applied during image data capture, incorporated in the device of King, in order to allow for contextual data and the mobile device location information associated with a captured image to further recognize an area within an image using a mobile device , which improves system recognition of desired data related to the things a user sees (as stated in Ramanujapuram ¶ [02] and [03]).
Re claim 18: King discloses the non-transitory computer storage media of claim 17, wherein the image data capture instruction is to capture a specific character string in a first area of an image (e.g. the present information is captured by the capturing device, which is taught in ¶ [545] above. An overlay can occur to appear on a portion of a document to be captured that may be associated with a markup portion, which is taught in ¶ [161] above.).
Re claim 19: King discloses the non-transitory computer storage media of claim 17, wherein the contextual data includes one or more of a format of an anticipated image to be captured at the location of the UE (e.g. when capturing an image and further providing context that the image contains some type of text, the system can determine that a user desires to acquire the text from the image. This information is fed to the system to allow for the capturing device to perform an OCR operation or a search of the text on the page, which is taught in ¶ [550]-[556] above. A character is considered as a format of the data instead of a picture image.), and
a character string within an anticipated image to be captured at the location of the UE (e.g. when capturing an image, detecting a proximity to an image and further providing context that the image contains some type of text, the system can determine that a user desires to acquire the text from the image. This information is fed to the system to allow for the capturing device to perform an OCR operation or a search of the text on the page, which is taught in ¶ [550]-[556] above.).
Re claim 20: King discloses the non-transitory computer storage media of claim 17, wherein the one or more processors is further configured to update a user profile based on UE activity after executing the image data capture instruction (e.g. a storage profile associated with the user is updated after capturing information in order to indicate information of a capture. This can build a timeline of the user’s actions for capture, which is taught in ¶ [547] and [548] above.).
Claim(s) 8 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over King, as modified by Ramanujapuram, as applied to claims 1 and 17 above, and further in view of Miller (US Pub 2024/0355065, Filing Date:12/7/2023).
Re claim 8: However, King fails to specifically teach the features of the method of claim 7, wherein the user profile is created by a machine learning module of the UE.
However, this is well known in the art as evidenced by Miller. Similar to the primary reference, Miller discloses AI that generates a user profile (same field of endeavor or reasonably pertinent to the problem).
Miller discloses wherein the user profile is created by a machine learning module of the UE (e.g. the system discloses an AI agent within an AI and ML system that is used to create a multimodal memory of information about a user, which is similar to a profile. This is taught in ¶ [82]-[86].).
[0082] An artificial intelligence and machine learning system 230 provides a variety of services to different subsystems within the interaction system 100. For example, the artificial intelligence and machine learning system 230 operates with the image processing system 202 and the camera system 204 to analyze images and extract information such as objects, text, or faces. This information can then be used by the image processing system 202 to enhance, filter, or manipulate images. The artificial intelligence and machine learning system 230 may be used by the augmentation system 206 to generate augmented content and augmented reality experiences, such as adding virtual objects or animations to real-world images. The communication system 208 and messaging system 210 may use the artificial intelligence and machine learning system 230 to analyze communication patterns and provide insights into how users interact with each other and provide intelligent message classification and tagging, such as categorizing messages based on sentiment or topic. The artificial intelligence and machine learning system 230 may also provide personalized AI agent system 232 functionality to message interactions 120 between user systems 102 and between a user system 102 and the interaction server system 110. The artificial intelligence and machine learning system 230 may also work with the audio communication system 216 to provide speech recognition and natural language processing capabilities, allowing users to interact with the interaction system 100 using voice commands.
[0083] In generative AI examples, the prediction/inference data that is output include trend assessment and predictions, translations, summaries, image or video recognition and categorization, natural language processing, face recognition, user sentiment assessments, advertisement targeting and optimization, voice recognition, or media content generation, recommendation, and personalization.
[0084] A personalized AI agent system 232 provides personalized features to a user of an interaction client 104 by analyzing user data and behavior to understand their preferences and interests. By utilizing machine learning algorithms and data analytics, the personalized AI agent system 232 can learn and adapt to inferences of the data, and then generatively suggest content relevant, specific, and custom tailored to the user. The personalized AI agent system 232 can analyze data from multiple sources, such as various user systems 102, messages, profile information, external data sources 316, image data captured in real-time by a camera of the user system 102, and/or any combination thereof to generate content items in real time and provide such content items to the user.
[0085] The personalized AI agent system 232 tracks interaction functions including user activity, such as the posts the users like, share, or comment on, the topics the users follow, the people the users connect with, and the time the users spend on the platform. Tracking performed by the personalized AI agent system 232 is only enabled if the user opts into the experience of receiving real-time generated content. The personalized AI agent system 232 can present to the user a full list of all activity and information that will be tracked and used to generate real time content recommendations. Only after receiving confirmation from the user that the user approves having such activity and information tracked does the personalized AI agent system 232 begin collecting such data and using such data to provide and generate the real time and on-the-fly content for presentation to the user.
[0086] The personalized AI agent system 232 can retrieve data from multiple data sources, such as activity on a user's mobile phone, an AR/VR device, a smart watch, a laptop, or other user device. Based on this information, the personalized AI agent system 232 identifies patterns and predicts user's interests to generate a multimodal memory for a particular user. The personalized AI agent system 232 analyzes the user's profile information, such as their age, gender, location, messages exchanged, and/or interactions performed on the user system 102 to provide personalized features. In some examples, the personalized AI agent system 232 suggests events and groups that are nearby, or recommend job opportunities that match the user's qualifications. The personalized AI agent system 232 generates real-time AR experiences and/or message content that is/are relevant to current circumstances and/or a real-world environment perceived by the user.
Therefore, in view of Miller, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the user profile is created by a machine learning module of the UE, incorporated in the device of King, in order to have a machine learning model create a user profile, which can increase user engagement with the application on the device (as stated in Miller ¶ [87]).
Re claim 16: However, King fails to specifically teach the features of the system of claim 15, wherein the user profile is created by a machine learning module of the UE.
However, this is well known in the art as evidenced by Miller. Similar to the primary reference, Miller discloses AI that generates a user profile (same field of endeavor or reasonably pertinent to the problem).
Miller discloses wherein the user profile is created by a machine learning module of the UE (e.g. the system discloses an AI agent within an AI and ML system that is used to create a multimodal memory of information about a user, which is similar to a profile. This is taught in ¶ [82]-[86] above.).
Therefore, in view of Miller, it would have been obvious to one of ordinary skill at the time the invention was made to have the feature of wherein the user profile is created by a machine learning module of the UE, incorporated in the device of King, as modified by Ramanujapuram, in order to have a machine learning model create a user profile, which can increase user engagement with the application on the device (as stated in Miller ¶ [87]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Shreve discloses smart image capturing.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abderrahim Merouan can be reached at 571-270-5254. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHAD DICKERSON/ Primary Examiner, Art Unit 2682
Prosecution Insights